[pgpool-general: 2957] Re: Connections stuck in CLOSE_WAIT, again
Tatsuo Ishii
ishii at postgresql.org
Tue Jun 24 06:57:37 JST 2014
Thanks. I'll look into this.
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp
> This is the full output of gdb attached to one of the pgpool children
> in DISCARD state:
>
> #0 0x00007fa8da59fbe3 in select () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x0000000000416621 in pool_check_fd (cp=cp at entry=0x1cd86c0) at
> #pool_process_query.c:951
> #2 0x0000000000416826 in pool_check_fd (cp=cp at entry=0x1cd86c0) at
> #pool_process_query.c:971
> #3 0x000000000041d99c in pool_read (cp=0x1cd86c0,
> #buf=buf at entry=0x7fffb797e053, len=len at entry=1) at pool_stream.c:139
> #4 0x000000000041c4ce in read_kind_from_backend
> #(frontend=frontend at entry=0x1ccb7c0, backend=backend at entry=0x1cb5c28,
> decided_kind=decided_kind at entry=0x7fffb797e45f "") at
> pool_process_query.c:3771
> #5 0x000000000044e291 in ProcessBackendResponse
> #(frontend=frontend at entry=0x1ccb7c0, backend=backend at entry=0x1cb5c28,
> state=state at entry=0x7fffb797e4c8,
> num_fields=num_fields at entry=0x7fffb797e4c6) at
> pool_proto_modules.c:2742
> #6 0x0000000000419d85 in pool_process_query
> #(frontend=frontend at entry=0x1ccb7c0, backend=backend at entry=0x1cb5c28,
> reset_request=reset_request at entry=1) at pool_process_query.c:289
> #7 0x000000000040c68f in do_child (unix_fd=unix_fd at entry=4,
> #inet_fd=inet_fd at entry=5) at child.c:403
> #8 0x00000000004077ff in fork_a_child (unix_fd=4, inet_fd=5, id=42) at
> #main.c:1238
> #9 0x0000000000407e63 in reaper () at main.c:2457
> #10 reaper () at main.c:2369
> #11 0x00000000004087ed in pool_sleep (second=<optimized out>) at
> #main.c:2654
> #12 0x00000000004064c9 in main (argc=<optimized out>, argv=<optimized
> #out>) at main.c:836
>
> Many thanks,
>
> --
> Juanjo Pérez
>
> www.oteara.com
>
> El 20/06/14 01:19, Tatsuo Ishii escribió:
>> Can you please post stack trace with symbols? I need identify the
>> code path. There are too may similar code path like this.
>>
>> Best regards,
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese:http://www.sraoss.co.jp
>>
>>> I have the same problem with PuppetDB fortend.
>>> When I stop the service of PuppetDB, all connections from pgpool is
>>> CLOSE_WAIT and in DISCARD state, even with connection timeout set to 1
>>> minute.
>>> here is the summery from gdb:
>>>
>>> the hanged function is:
>>> __select_no_cancel() from /lib64/libc.so.6.
>>>
>>> stacktrace output:
>>>
>>> 0 __select_no_cancel() from /lib64/libc.so.6.
>>> 1 pool_check_fd
>>> 2 pool_read
>>> 3 read_kind_from_backend
>>> 4 ProcessBackendResponse
>>> 5 pool_process_query
>>> 6 fork_a_child()
>>> 7 reaper
>>> 8 pool_sleep
>>> 9 main
>>>
>>>
>>>
>>> On Thu, Jun 19, 2014 at 2:29 AM, Tatsuo Ishii <ishii at postgresql.org>
>>> wrote:
>>>
>>>> You said your developer found pgpool child process in DISCARD state.
>>>>
>>>> Please attach gdb to the process in DISCARD state and take
>>>> backtrace. Also I need actual netsta -anp outputs.
>>>>
>>>> Best regards,
>>>> --
>>>> Tatsuo Ishii
>>>> SRA OSS, Inc. Japan
>>>> English: http://www.sraoss.co.jp/index_en.php
>>>> Japanese:http://www.sraoss.co.jp
>>>>
>>>>> Hi Tatsuo,
>>>>>
>>>>> We are facing this same problem, and as I see it remains unsolved (or
>>>>> perhaps they didn't report the solution).
>>>>> We are using pgpool 3.3.3, with two postgres 9.1 in streaming
>>>>> replication. The OS is Debian 3.2.54-2 (64bits).
>>>>>
>>>>> The log shows messages like this:
>>>>>
>>>>> ProcessFrontendResponse: failed to read kind from frontend.
>>>> frontend
>>>>> abnormally exited
>>>>>
>>>>> I'm not sure but it looks like one for each connection left unclosed.
>>>>>
>>>>> Also, a ps -ef returns a lot of pgpool children in DISCARD state. And
>>>>> the netstat -anp, gives the correspondent TCP connection in
>>>> CLOSE_WAIT
>>>>> state.
>>>>>
>>>>> The developer has checked the client process is closing the
>>>>> connections and we try connecting directly to the postgres and it
>>>>> worked fine.
>>>>>
>>>>> When you say "to attach debugger", do you mean change log_statment =
>>>>> true or debug_level to a value other than 0 or any other procedure?
>>>>>
>>>>> Many thanks,
>>>>>
>>>>> --
>>>>> Juanjo Pérez
>>>>>
>>>>> www.oteara.com
>>>>>
>>>>> El 02/04/14 01:05, Tatsuo Ishii escribió:
>>>>>>> We have a client on 3.3 experiencing the the problem noted here:
>>>>>>>
>>>>>>>
>>>> http://www.sraoss.jp/pipermail/pgpool-general/2012-December/001283.html
>>>>>>> strace is showing the child processes at:
>>>>>>>
>>>>>>> clone(child_stack=0,
>>>>>>> flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD,
>>>>>>> child_tidptr=0x7fcf659f3a10) = 6258
>>>>>>>
>>>>>>> I didn't see any resolution of that issue; is there data we can gather
>>>>>>> to assist?
>>>>>> I followed the old 2012 posting with:
>>>>>>
>>>>>>> That means pgpool does not close the socket connected to by your
>>>>>>> applications. Is it possible to attach debugger to such that pgpool
>>>>>>> process to see what pgpool is doing?
>>>>>> But I got no response until now. Can you please do this?
>>>>>>
>>>>>> Best regards,
>>>>>> --
>>>>>> Tatsuo Ishii
>>>>>> SRA OSS, Inc. Japan
>>>>>> English: http://www.sraoss.co.jp/index_en.php
>>>>>> Japanese: http://www.sraoss.co.jp
>>>>>> _______________________________________________
>>>>>> pgpool-general mailing list
>>>>>> pgpool-general at pgpool.net
>>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>
>>>>> _______________________________________________
>>>>> pgpool-general mailing list
>>>>> pgpool-general at pgpool.net
>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>> _______________________________________________
>>>> pgpool-general mailing list
>>>> pgpool-general at pgpool.net
>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>
>
More information about the pgpool-general
mailing list