[pgpool-general: 418] pg-pool-3.0.4. -unable to create child processes

[Ludwig Adam]

Dear list,

we just encountered a production issue with pg-pool-3.0.4.: pg-pool killed all idle childprocesses which were waiting for a connection request and failed to create new ones.

I am unsure why the childs exited:


1)    As we have child_life_time = 300 I guess the childs exited normally after a 5 minute period.

2)    We did a Config reload on pg_pool, and that could mean also that waiting child processes are killed

but pg_pool did not create new ones. So we were left with ~110 active pg-pool processes (which still seemed to work) but we could make no new connections to pgppol (can't tell if the connections have been queued or not).
This is what the syslog says:

May  7 15:43:13 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 15:43:13 LOG:   pid 938: reload config files.
May  7 15:43:13 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 15:43:13 LOG:   pid 2707: reload config files.
May  7 16:09:29 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:17 ERROR: pid 938: fork() failed. reason: Resource temporarily unavailable
May  7 16:09:34 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:34 LOG:   pid 29194: connect_inet_domain_socket_by_port: exit request has been sent
May  7 16:09:34 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:34 ERROR: pid 29194: connection to <backendserver ha1 here> (5432) failed
May  7 16:09:34 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:34 ERROR: pid 29194: new_connection: create_cp() failed
May  7 16:09:34 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:34 LOG:   pid 29194: notice_backend_error: 0 fail over request from pid 29194
May  7 16:30:22 pgpool-PrepaidDB-HA1 heartbeat: [2425]: info: pgpool-prepaiddb-ha1 wants to go standby [all]

We also noticed some high load on the server which may be related to the issue.

Any idea what the the cause of this issue is? Do you recommend setting child_life_time = 0? I think there are some memory leak issues in the 3.0.4 versions.
3.0.4 is the latest official version in the Ubuntu repository and we would rather not go to self-compiled versions for reasons of maintenance and compliance.

Best regards,
Ludwig

petaFuel GmbH
Ludwig Adam
Geschäftsführer

Münchner Strasse 4
85354 Freising

Tel.: +49 (0) 8161 4060 - 202               Mobil: +49 (0) 179-22 911 02
Fax: +49 (0) 8161 4060 - 401               eMail: ludwig.adam at petafuel.de<mailto:ludwig.adam at petafuel.de>

Geschäftsführer: Ludwig Adam, Dr. Peter Schönweitz
Amtsgericht München, HRB 133773
www.petaFuel.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20120507/ff7824b8/attachment.html>