[pgpool-general: 426] Re: FW: Re: Re: pg-pool-3.0.4. -unable to create child processes

[Tatsuo Ishii]

That was *not* an estimation. That was from the output of "cat
/proc/process-id/status". Of course it might be a matter of way to
translate the number. I would like to hear opinions from Linux
experts...
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> Dear Tatsuo,
> 
> this seems to be a very high estimation. In our environment one child has max pool size of 1, i . e. It manages one connection to the backend. Memory usage of a idle process according to ps aux says 0.1 percent per child, which translates to roughly 1 meg if I am not mistaken...
> 
> Thanks,
> Ludwig
> 
> Mobil gesendet.
> 
> -----Original Message-----
> From: Tatsuo Ishii [ishii at postgresql.org]
> Received: Dienstag, 08 Mai 2012, 17:37
> To: Ludwig Adam [ludwig.adam at petafuel.de]
> CC: pgpool-general at pgpool.net [pgpool-general at pgpool.net]
> Subject: Re: [pgpool-general: 419] Re: pg-pool-3.0.4. -unable to create child processes
> 
> On my x86_64 linux machine, a pgpool process's VMSize is
> 56692+(68*num_init_children) (unit is kb). num_init_children = 1500
> case, it will be 158692 kb. So total VMSize is 158692 * 1500 =
> 238038000, which is about 227GB. I'm not sure how much of 227GB is
> actually required since some of memory is shared among multiple
> process, but I guess 1GB mem is too small.
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
> 
>> Dear list,
>>
>> just to provide you an update on the systems spec, as I have seen that the failed fork may be limited to some memory constraints / limits:
>>
>> We have set the num_init_childs to 1500 (from which ~500 are connected at low system load and around 900 at peak times currently).
>> The allocated RAM for the machine is 1 GB- what's your recommendation for the relation of allocated RAM and # of pgpool childs?
>>
>> I would suspect a memory leak on the connected, non-idle processes (as we are still running 3.04) but the kind of error message does not make really sense- why should just a fork not be possible and also we had no child processes at all running.
>>
>>
>> Ludwig
>>
>>
>> petaFuel GmbH
>> Ludwig Adam
>> Geschäftsführer
>>
>> Geschäftsführer: Ludwig Adam, Dr. Peter Schönweitz
>> Amtsgericht München, HRB 133773
>>
>> Tel.: +49 (0) 8161 4060 - 202               Mobil: +49 (0) 179-22 911 02
>>
>> Von: Ludwig Adam
>> Gesendet: Montag, 7. Mai 2012 17:15
>> An: pgpool-general at pgpool.net
>> Cc: EDV
>> Betreff: pg-pool-3.0.4. -unable to create child processes
>>
>> Dear list,
>>
>> we just encountered a production issue with pg-pool-3.0.4.: pg-pool killed all idle childprocesses which were waiting for a connection request and failed to create new ones.
>>
>> I am unsure why the childs exited:
>>
>>
>> 1)    As we have child_life_time = 300 I guess the childs exited normally after a 5 minute period.
>>
>> 2)    We did a Config reload on pg_pool, and that could mean also that waiting child processes are killed
>>
>> but pg_pool did not create new ones. So we were left with ~110 active pg-pool processes (which still seemed to work) but we could make no new connections to pgppol (can't tell if the connections have been queued or not).
>> This is what the syslog says:
>>
>> May  7 15:43:13 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 15:43:13 LOG:   pid 938: reload config files.
>> May  7 15:43:13 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 15:43:13 LOG:   pid 2707: reload config files.
>> May  7 16:09:29 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:17 ERROR: pid 938: fork() failed. reason: Resource temporarily unavailable
>> May  7 16:09:34 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:34 LOG:   pid 29194: connect_inet_domain_socket_by_port: exit request has been sent
>> May  7 16:09:34 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:34 ERROR: pid 29194: connection to <backendserver ha1 here> (5432) failed
>> May  7 16:09:34 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:34 ERROR: pid 29194: new_connection: create_cp() failed
>> May  7 16:09:34 pgpool-PrepaidDB-HA1 pgpool: 2012-05-07 16:09:34 LOG:   pid 29194: notice_backend_error: 0 fail over request from pid 29194
>> May  7 16:30:22 pgpool-PrepaidDB-HA1 heartbeat: [2425]: info: pgpool-prepaiddb-ha1 wants to go standby [all]
>>
>> We also noticed some high load on the server which may be related to the issue.
>>
>> Any idea what the the cause of this issue is? Do you recommend setting child_life_time = 0? I think there are some memory leak issues in the 3.0.4 versions.
>> 3.0.4 is the latest official version in the Ubuntu repository and we would rather not go to self-compiled versions for reasons of maintenance and compliance.
>>
>> Best regards,
>> Ludwig
>>
>> petaFuel GmbH
>> Ludwig Adam
>> Geschäftsführer
>>
>> Münchner Strasse 4
>> 85354 Freising
>>
>> Tel.: +49 (0) 8161 4060 - 202               Mobil: +49 (0) 179-22 911 02
>> Fax: +49 (0) 8161 4060 - 401               eMail: ludwig.adam at petafuel.de<mailto:ludwig.adam at petafuel.de>
>>
>> Geschäftsführer: Ludwig Adam, Dr. Peter Schönweitz
>> Amtsgericht München, HRB 133773
>> www.petaFuel.de<http://www.petaFuel.de>
>>