[pgpool-general: 7831] Re: broken pipe in the logs

[Tatsuo Ishii]

> Hello Tatsuo.
> Thank you for your response.
> Here is the version: pgpool-II version 4.2.1 (chichiriboshi)
> Here is our config: https://pastebin.com/mHAdiZUU
> 
> Killing or restarting the process works but that is what we are trying to
> avoid since that requires manual intervention. I was wondering if we can
> identify the root cause and fix it so that human intervention is reduced.

To know the root cause, you need to do this:

>> I wonder if the corresponding backend process is still alive or
>> not. You can check it by using pcp_proc_info command. Once you
>> identify the backend process, you can check PostgreSQL log what
>> happend at that time.

If PostgreSQL backend process is not there, you should look for the
cause in PostgreSQL.  If the process is there, it is likely there was
a network issue between pgpool and PostgreSQL.

> On Thu, Oct 21, 2021 at 6:30 AM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
> 
>> Hi,
>>
>> > Hello.
>> >
>> > We are running a small self-managed pgpool and postgres setup in Google
>> > cloud. We have setup two Pgpools which connect to one primary and two
>> > replica postgres setup.
>> >
>> > On more than one occasion, one of the pgpools has printed a "Broken pipe"
>> > error and the backend went unreachable for that pgpool, while the other
>> > pgpool was able to reach the backend postgres just fine. The bad pgpool
>> > will only start working after a restart of the pgpool.
>> >
>> > I would really appreciate some help in understanding why this is
>> happening
>> > and what can be done to mitigate this issue.
>> >
>> > Log lines, are added below:
>> >
>> > Oct 19 13:05:31 pgpool-fdfv pgpool[935]: 2021-07-21 13:05:31: pid
>> > 10003: LOG:  received degenerate backend request for node_id: 1 from
>> > pid [10003]
>> > Oct 19 13:05:31 pgpool-fdfv pgpool[935]: 2021-07-21 13:05:31: pid 935:
>> > LOG:  Pgpool-II parent process has received failover request
>> > Oct 19 13:05:31 pgpool-fdfv pgpool[935]: 2021-07-21 13:05:31: pid
>> > 10003: WARNING:  write on backend 1 failed with error :"Broken pipe"
>> > Oct 19 13:05:31 pgpool-fdfv pgpool[935]: 2021-07-21 13:05:31: pid
>> > 10003: DETAIL:  while trying to write data from offset: 0 wlen: 5
>>
>> Can you please share the Pgpool-II version? Also sharing pgpool.conf
>> is helpful (you can remove parameters you don't want to put in
>> public).
>>
>> I wonder if the corresponding backend process is still alive or
>> not. You can check it by using pcp_proc_info command. Once you
>> identify the backend process, you can check PostgreSQL log what
>> happend at that time.
>>
>> In the mean time you can kill the pgpool process to let the pgpool
>> parent process to fork a new child process.
>>
>> kill -6 "pid" or kill -9 "pid"
>> ("pid" is 10003 in the case above)
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese:http://www.sraoss.co.jp
>>
>>
> 
> -- 
> *ADITYA PATAWARI*
> Devops Engineer
> ―
> *CareMessage | **caremessage.org <http://caremessage.org/>*
> M: +91-8310050990