[pgpool-general: 3159] Re: 3.3.3 , Apparent hang

[Tatsuo Ishii]

> I have an application that has a pair of replicating Postgres 9.2 servers,
> that are accessed via a couple pgpool 3.3.3 boxes that behind an HA proxy
> box. The setup generally works well, but after some amount of time under
> high load, the pgpool boxes will sometimes stop responding. They seem
> healthy at a system level, but if you try to connect to a box in this
> state, it will open a connection but never present the expected login
> dialog. I've connected strace to an instance in this state, and it doesn't
> seem to be doing anything at all. It seems to be watching a couple of
> particular inodes for activity, but that's it. A functional but idle
> instance shows many more system calls happening. Connecting to child pids
> shows even less activity. They do not respond to HUPs or TERMs as far as I
> can tell, and the normal "pgpool stop" also does not work. The only way
> I've found to stop the processes once in this state is the good ol' kill -9.
> 
> The only interesting log traffic I've seen is this:
> 
> "ProcessFrontendResponse: failed to read kind from frontend. frontend
> abnormally exited"

We identified that this client side error (in your case HAProxy?)
triggeres hanging and was fixed in 3.3.4. Even if you upgrade and
still see hanging, please report it.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

> Is this a known issue? My searches have not turned up anything. If not,
> what is the best information I can provide to help track it down? Note that
> we've also seen the DISCARD problem that is apparently fixed in 3.3.4, but
> this seems to be a different problem. One of the apparent symptoms of this
> problem is a large number sockets stuck in SYN_RECV state, but I'm not
> certain it's related.
> 
> Thanks!
> 
> QH