[pgpool-general: 2761] Re: All child processes hang at read_packets_and_process
Yugo Nagata
nagata at sraoss.co.jp
Mon Apr 14 17:48:10 JST 2014
Hi,
What version of pgpool do you use?
And, for analysys, I nead pgpool.conf and log messages of both pgpool & backends.
Could you please send me these?
On Mon, 14 Apr 2014 12:42:33 +0900
Junegunn Choi <junegunn.c at gmail.com> wrote:
> All child processes hang at read_packets_and_process
>
> Hi,
>
> We're having occasional problem where all child processes hang at
> `read_packets_and_process`, waiting for packets from backend when the
> backend
> connection is actually idle and not executing any query. If this happens,
> our only option is to restart pgpool.
>
> I'm aware of `child_idle_limit` parameter, but unfortunately we can't set
> it to
> a small value as we know that some of the queries takes a few minutes,
> so it doesn't help much in reducing the downtime.
>
> We have no idea when or why this happens, our pgpool process currently
> handles around 500 connections/second, and it happens maybe once in a
> couple of days.
>
> Any idea what could have gone wrong? Any help is greatly appreciated.
>
> # Callstack of each child process (all identical):
>
> (gdb) where
> #0 0x0000003d222cdaf3 in __select_nocancel () from /lib64/libc.so.6
> #1 0x000000000041addf in read_packets_and_process (frontend=0xac6cb60,
> backend=0xac5da40, reset_request=0, state=0x7fffb871aad0,
> num_fields=0x7fffb871aad6, cont=0x7fffb871aacc "\001")
> at pool_process_query.c:4859
> #2 0x000000000041b741 in pool_process_query (frontend=0xac6cb60,
> backend=0xac5da40, reset_request=0) at pool_process_query.c:260
> #3 0x000000000040ad7a in do_child (unix_fd=5, inet_fd=6) at child.c:355
> #4 0x000000000040455f in fork_a_child (unix_fd=5, inet_fd=6, id=91) at
> main.c:1258
> #5 0x0000000000404887 in reaper () at main.c:2482
> #6 0x0000000000404c15 in pool_sleep (second=<value optimized out>) at
> main.c:2679
> #7 0x00000000004079fa in main (argc=<value optimized out>, argv=<value
> optimized out>) at main.c:856
>
> # `ps` output when it happens:
>
> pgpool 12894 0.0 0.0 22452 1800 ? S Mar26 0:00
> \_ /db/pgpool/bin/pgpool -n -D -f /db/pgpool/conf/pg8/pgpool.conf
> pgpool 12918 0.0 0.0 22536 1356 ? S Mar26 0:23 |
> \_ pgpool: PCP: wait for connection request
> pgpool 12919 0.0 0.0 22452 1004 ? S Mar26 0:00 |
> \_ pgpool: worker process
> pgpool 26236 0.0 0.0 27240 2616 ? S 07:54 0:10 |
> \_ pgpool: cloud cloud 10.27.18.56(17208) idle
> pgpool 26936 0.0 0.0 27068 2468 ? S 07:59 0:06 |
> \_ pgpool: cloud cloud 10.30.25.96(24663) idle
> pgpool 27422 0.1 0.0 27308 2684 ? S 08:02 0:12 |
> \_ pgpool: cloud cloud 10.27.18.56(49400) idle
> pgpool 29357 0.0 0.0 27252 2644 ? S 08:13 0:11 |
> \_ pgpool: cloud cloud 10.27.33.88(17186) idle
> pgpool 2363 0.1 0.0 27244 2620 ? S 08:45 0:10 |
> \_ pgpool: cloud cloud 10.30.25.96(35038) idle
> pgpool 3672 0.1 0.0 27216 2612 ? S 08:54 0:10 |
> \_ pgpool: cloud cloud 10.27.18.56(49399) idle
> pgpool 4969 0.1 0.0 27168 2568 ? S 09:00 0:09 |
> \_ pgpool: cloud cloud 10.27.33.88(16792) idle
> pgpool 5981 0.1 0.0 27144 2540 ? S 09:06 0:08 |
> \_ pgpool: cloud cloud 10.27.33.88(16821) idle
> pgpool 10072 0.1 0.0 27308 2704 ? S 09:30 0:11 |
> \_ pgpool: cloud cloud 10.30.25.96(47224) idle
> pgpool 10825 0.0 0.0 27032 2404 ? S 09:35 0:05 |
> \_ pgpool: cloud cloud 10.27.33.88(22482) idle
> pgpool 12839 0.0 0.0 26960 2344 ? S 09:46 0:04 |
> \_ pgpool: cloud cloud 10.27.33.88(22429) idle
> pgpool 13630 0.0 0.0 26924 2304 ? S 09:51 0:03 |
> \_ pgpool: cloud cloud 10.27.18.56(45126) idle
> pgpool 13650 0.0 0.0 26932 2320 ? S 09:51 0:03 |
> \_ pgpool: cloud cloud 10.30.25.96(35105) idle
> pgpool 13869 0.0 0.0 26924 2300 ? S 09:52 0:03 |
> \_ pgpool: cloud cloud 10.30.25.96(34504) idle
> pgpool 15023 0.0 0.0 26916 2284 ? S 09:58 0:03 |
> \_ pgpool: cloud cloud 10.27.18.56(44532) idle
> pgpool 18577 0.2 0.0 27200 2580 ? S 10:16 0:07 |
> \_ pgpool: cloud cloud 10.27.33.88(29036) idle
> pgpool 27679 0.2 0.0 27080 2464 ? S 10:37 0:05 |
> \_ pgpool: cloud cloud 10.27.33.88(36875) idle
> pgpool 29064 0.1 0.0 26988 2364 ? S 10:39 0:03 |
> \_ pgpool: cloud cloud 10.30.25.96(46962) idle
> pgpool 30220 0.1 0.0 26936 2336 ? S 10:40 0:03 |
> \_ pgpool: cloud cloud 10.27.18.56(56669) idle
> pgpool 10215 0.0 0.0 26752 2148 ? S 10:55 0:00 |
> \_ pgpool: cloud cloud 10.30.25.96(46863) idle
> pgpool 26719 0.0 0.0 26752 1616 ? S 11:16 0:00 |
> \_ pgpool: cloud cloud 10.27.18.56(58985) idle
>
>
> Thanks,
> Junegunn Choi
--
Yugo Nagata <nagata at sraoss.co.jp>
More information about the pgpool-general
mailing list