[pgpool-general: 1242] PGPool Stops Accepting Connections Without Errors

[Nick Webb]

All,

Not sure this is a problem with PGPool or not, but every 3-4 days our
PGPool instance just stops accepting postgres connections, they all
time out.  When this situation occurs, the system is not particularly
busy, and CPU, memory, and number of current connections are all
normal.  What's more, connecting to the backends directly from the
PGPool host also works fine, and is quite snappy (via psql), but no
connections to PGPool work.  Restart pgpool and things are back up and
running in seconds.

Looking at the log, there are no errors or warnings, not sure where to
look next.  I checked all the usual Linux log suspects, and nothing
else was going on (like maxing out some system resource, etc.).

Here is the log immediately before and after the "hang up", which
happened approximately 19:42 UTC:
Dec  6 19:23:58 PGLB pgpool[3210]: statement: SELECT
Dec  6 19:23:58 PGLB pgpool[3210]: DB node id: 1 backend pid: 22221
statement: SELECT
Dec  6 19:23:58 PGLB pgpool[3210]: statement: SELECT
Dec  6 19:23:58 PGLB pgpool[3210]: DB node id: 1 backend pid: 22221
statement: SELECT
Dec  6 19:23:58 PGLB pgpool3210]: statement: SELECT
Dec  6 19:23:58 PGLB pgpool[3210]: DB node id: 1 backend pid: 22221
statement: SELECT
Dec  6 19:24:24 PGLB pgpool[26744]: statement:  DISCARD ALL
Dec  6 19:24:24 PGLB pgpool[26744]: DB node id: 0 backend pid: 27232
statement:  DISCARD ALL
Dec  6 19:24:24 PGLB pgpool[26744]: DB node id: 1 backend pid: 20837
statement:  DISCARD ALL
Dec  6 19:24:24 PGLB pgpool[26744]: connection received: host=3.myhost.com=43275
Dec  6 19:42:06 PGLB pgpool[26782]: statement:  DISCARD ALL
Dec  6 19:42:06 PGLB pgpool[26782]: DB node id: 0 backend pid: 28378
statement:  DISCARD ALL
Dec  6 19:42:06 PGLB pgpool[26782]: DB node id: 1 backend pid: 21991
statement:  DISCARD ALL
Dec  6 19:42:06 PGLB pgpool[26782]: connection received:
host=www.myhost.com port=57636

[ note the lack of entries after this point, until the restart,
despite many attempted connections ]

Dec  6 19:43:59 PGLB pgpool[26276]: received fast shutdown request
Dec  6 19:44:00 PGLB pgpool[13237]: connection received:
host=2.myhost.com port=5973
Dec  6 19:44:00 PGLB pgpool[13234]: pgpool-II successfully started.
version 3.1.5 (hatsuiboshi)

Next time I'll run a "show pool_pools", but it's hard for me to
believe that all the slots were full.

Thanks,

Nick
--
Nick Webb
Principal Systems Engineer
Red Wire Services, LLC
T: 206.829.8621 / F: 206.826.1223

1725 SW Roxbury St. STE 5
Seattle, WA 98106

http://www.redwireservices.com

http://linkedin.com/in/NickWebb