[pgpool-general: 7818] Re: Error: sorry, too many clients

[Tatsuo Ishii]

> That's great, I will watch the changelog of 4.2.x for a relevant change.
> 
> If I may suggest something, I think that the event of a child's crash (e.g
> child process signaled with SIGKILL by OOM killer) deserves a higher
> logging level than LOG (perhaps WARN). That would help in detecting such
> problematic situations.

Good suggestion. Thanks.

> Thanks for your support!

You are welcome!

> On Tue, Oct 19, 2021 at 2:06 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
> 
>> I personally think the fix should be included in an upcoming Pgpool
>> release. If other developers agree, I will commit/push the change to
>> all supported branches including 4.2.x.
>>
>> > Thanks Tatsuo
>> >
>> > Is there any plan for this fix to be included in an upcoming Pgpool
>> release
>> > (I am running 4.2.4 from Alpine repositories)  ?
>> > Or, do you consider this as too deployment-specific (and so I should
>> create
>> > a custom build from source of 4.2.4)?
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Oct 18, 2021 at 11:36 AM Tatsuo Ishii <ishii at sraoss.co.jp>
>> wrote:
>> >
>> >> >> In our environment:
>> >> >> reserved_connections=0
>> >> >> listen_backlog_multiplier=2
>> >> >>
>> >> >> So, if I understand correctly, we have 2 different problematic cases:
>> >> >>
>> >> >> (1) a client silently disappears (doesn't properly close the
>> >> connection):
>> >> >> In this case, setting `child_idle_limit` to a non-zero value (say
>> 15min)
>> >> >> could mitigate the problem, and eventually the child worker will
>> again
>> >> be
>> >> >> available to handle an incoming connection.
>> >> >>
>> >> >> (2) a server-side child process crashes: In this case, Pgpool still
>> >> keeps a
>> >> >> dangling reference to this process, and if some client is routed
>> there
>> >> will
>> >> >> get a "sorry, too many clients" error.
>> >> >
>> >> > Since the counter is shared by all process, any process could get a
>> >> > "sorry..." error.
>> >> >
>> >> >> As for now, this problem cannot be
>> >> >> mitigated unless the (parent) Pgpool process is restarted.
>> >> >
>> >> > Yes.
>> >> >
>> >> >> One more question:
>> >> >> What can make a child process crash? (well, I mean, what are the most
>> >> >> common reasons for that).
>> >> >
>> >> > Besides admin's mistake, probabbly the most case is OOM killer.
>> >> >
>> >> >> Does this crash leave any trace (that I should search for) in
>> Pgpool's
>> >> >> error log? Does parent Pgpool get notified on this (e.g. via waitpid
>> or
>> >> >> SIGCHLD)?
>> >> >
>> >> > Yes, Pgpool parent process should leave a log when catching SIGCHLD.
>> >> > If I kill a child process using kill -9, I got following log:
>> >> >
>> >> > 2021-10-18 14:44:48.951: main pid 997555: LOG:  child process with
>> pid:
>> >> 997582 exits with status 9 by signal 9
>> >> > 2021-10-18 14:44:48.952: main pid 997555: LOG:  fork a new child
>> process
>> >> with pid: 997609
>> >>
>> >> Actually if reserved_connections=0, then there's no need to manage the
>> >> connection counter. So I have created a patch to eliminate the
>> >> management part. You will never be troubled by the "sorry..."  error.
>> >>
>> >> If you like, please try attached patch.
>> >> --
>> >> Tatsuo Ishii
>> >> SRA OSS, Inc. Japan
>> >> English: http://www.sraoss.co.jp/index_en.php
>> >> Japanese:http://www.sraoss.co.jp
>> >>
>>