[Pgpool-general] unexpected EOF on client connection

[Johnny Tan]

On Wed, Sep 14, 2011 at 9:12 PM, Lonni J Friedman <netllama at gmail.com> wrote:
> On Wed, Sep 14, 2011 at 6:00 PM, Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>>> On Wed, Sep 14, 2011 at 4:22 PM, Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>>>>>>> I'm pretty sure that's not the case as the messages stop whenever
>>>>>>> pgpool isn't running, they were not present prior to using pgpool, and
>>>>>>> pg_hba.conf is setup such that the database servers only accept
>>>>>>> connections from each other, and the server running pgpool.  None of
>>>>>>> these servers have normal users connected directly to them (such as
>>>>>>> with ssh), nor are they running anything that would connect to the
>>>>>>> database as a client.  Also, the volume of these messages are such
>>>>>>> that something significant has to be causing them.  Last night, in the
>>>>>>> span of 5 minutes, there were 117 of these messages.
>>>>>>
>>>>>> Ok. I would like to narraow down the reason why we have "unexpected
>>>>>> EOF on client connection" message frequently. I think currently there
>>>>>> are two possiblities:
>>>>>>
>>>>>> 1) pgpool child was killed by some unknown reason(we can omit
>>>>>>   segfault case because you don't see it in the pgpool log)
>>>>>>
>>>>>> 2) pgpool child disconnects to PostgreSQL in ungraceful manner
>>>>>>
>>>>>> For 1) I would like to know if pgpool child process are fine since
>>>>>> they are spawned. Are you seeing any pgpool child process disappeared
>>>>>> since pgpool started?
>>>>>
>>>>> I assume this should be determined by num_init_children (which I've
>>>>> set to 195 in pgpool.conf)?  If so, then I currently have 195
>>>>> processes in either the "wait for connection request" state or
>>>>> actively connected state.
>>>>
>>>> No. Pgpool parent process automatically respawns child process if it's
>>>> dyning. So having num_init_children child process is not showing
>>>> anything usefull. You record 195 process ids and compare current
>>>> process ids. If some of them have been changed, we can assume that
>>>> child process is dying.
>>>
>>> Ah, good point.  I just diffed the list of PIDs associated with pgpool
>>> processes before and after another EOF message in the log, and there
>>> were no differences.  So I think that rules out any processes dying?
>>
>> Right.
>>
>>> One other thing that I just noticed from comparing logs between all of
>>> the database servers is that the time stamps for every one of the
>>> 'unexpected EOF on client connection' instances are identical.  In
>>> other words, they are happening at the same time on each server.  I
>>> think this further suggests that pgpool has to be doing it?
>>
>> Yes, I think so unless you set connection_life_time to other than 0 or
>> the network connection between PostgreSQL and pgpool is unstable.

I've used both pgpool and pgbouncer. We also have recurring similar
client-EOF messages in both pgpool and pgbouncer logs. On pgbouncer,
the actual error is:

LOG C-0x7db270: mydb/myuser at 10.0.0.160:43057 closing because: client
unexpected eof (age=44005)

On pgpool side, the error is exactly the same as Lonni's.

So, from that, I had concluded that it was from our app side. But I
have nothing further to substantiate that conclusion. Just adding it
as another datapoint.

Lonni, given how easy pgbouncer is to setup, it may be worth doing a
quick proof-of-concept with it and see if you get similar EOF errors
there. If so, I think that would eliminate the problem stemming from
the middleware?

johnny