[Pgpool-general] unexpected EOF on client connection

[Lonni J Friedman]

On Thu, Sep 15, 2011 at 8:57 AM, Johnny Tan <johnnydtan at gmail.com> wrote:
> On Wed, Sep 14, 2011 at 9:12 PM, Lonni J Friedman <netllama at gmail.com> wrote:
>> On Wed, Sep 14, 2011 at 6:00 PM, Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>>>> On Wed, Sep 14, 2011 at 4:22 PM, Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>>>>>>>> I'm pretty sure that's not the case as the messages stop whenever
>>>>>>>> pgpool isn't running, they were not present prior to using pgpool, and
>>>>>>>> pg_hba.conf is setup such that the database servers only accept
>>>>>>>> connections from each other, and the server running pgpool.  None of
>>>>>>>> these servers have normal users connected directly to them (such as
>>>>>>>> with ssh), nor are they running anything that would connect to the
>>>>>>>> database as a client.  Also, the volume of these messages are such
>>>>>>>> that something significant has to be causing them.  Last night, in the
>>>>>>>> span of 5 minutes, there were 117 of these messages.
>>>>>>>
>>>>>>> Ok. I would like to narraow down the reason why we have "unexpected
>>>>>>> EOF on client connection" message frequently. I think currently there
>>>>>>> are two possiblities:
>>>>>>>
>>>>>>> 1) pgpool child was killed by some unknown reason(we can omit
>>>>>>>   segfault case because you don't see it in the pgpool log)
>>>>>>>
>>>>>>> 2) pgpool child disconnects to PostgreSQL in ungraceful manner
>>>>>>>
>>>>>>> For 1) I would like to know if pgpool child process are fine since
>>>>>>> they are spawned. Are you seeing any pgpool child process disappeared
>>>>>>> since pgpool started?
>>>>>>
>>>>>> I assume this should be determined by num_init_children (which I've
>>>>>> set to 195 in pgpool.conf)?  If so, then I currently have 195
>>>>>> processes in either the "wait for connection request" state or
>>>>>> actively connected state.
>>>>>
>>>>> No. Pgpool parent process automatically respawns child process if it's
>>>>> dyning. So having num_init_children child process is not showing
>>>>> anything usefull. You record 195 process ids and compare current
>>>>> process ids. If some of them have been changed, we can assume that
>>>>> child process is dying.
>>>>
>>>> Ah, good point.  I just diffed the list of PIDs associated with pgpool
>>>> processes before and after another EOF message in the log, and there
>>>> were no differences.  So I think that rules out any processes dying?
>>>
>>> Right.
>>>
>>>> One other thing that I just noticed from comparing logs between all of
>>>> the database servers is that the time stamps for every one of the
>>>> 'unexpected EOF on client connection' instances are identical.  In
>>>> other words, they are happening at the same time on each server.  I
>>>> think this further suggests that pgpool has to be doing it?
>>>
>>> Yes, I think so unless you set connection_life_time to other than 0 or
>>> the network connection between PostgreSQL and pgpool is unstable.
>
> I've used both pgpool and pgbouncer. We also have recurring similar
> client-EOF messages in both pgpool and pgbouncer logs. On pgbouncer,
> the actual error is:
>
> LOG C-0x7db270: mydb/myuser at 10.0.0.160:43057 closing because: client
> unexpected eof (age=44005)
>
> On pgpool side, the error is exactly the same as Lonni's.
>
> So, from that, I had concluded that it was from our app side. But I
> have nothing further to substantiate that conclusion. Just adding it
> as another datapoint.
>
> Lonni, given how easy pgbouncer is to setup, it may be worth doing a
> quick proof-of-concept with it and see if you get similar EOF errors
> there. If so, I think that would eliminate the problem stemming from
> the middleware?

Actually I was using pgbouncer prior to switching to pgpool, and there
were never any errors related to unexpected EOF.  This definitely
started since i switched to pgpool.