[pgpool-general: 5225] Re: Questionable Failover
Muhammad Usama
m.usama at gmail.com
Wed Jan 11 20:32:09 JST 2017
Hi
It seems like there was a temporary network glitch/failure between the two
data centers around "Jan 05 00:04:29" time triggered the failover of
backend node0.
If we look at the logs of each node around this time we can see that the
Pgpool-II nodes in the first data center (dbserver1 and dbserver2) were not
able to connect to PostgreSQL on dbserver3 that is located in the second
data center, and at the very same time the Pgpool-II on dbserver3 was also
having the connection problem with PostgreSQL located on the first data
center on dbserver1. All this points towards the connectivity issue between
the two data centers.
Also, apparently the network issue seems to have resolved after a couple of
minutes, but till that time the PostgreSQL node failure was already
registered by Pgpool-II on dbserver3 and the failover was triggered.
Thanks
Best regards
Muhammad Usama
On Wed, Jan 11, 2017 at 12:04 AM, Yates, James C. -ND <
James.C.Yates.-ND at disney.com> wrote:
> PgPool initiated a failover last week, but the servers didn’t crash, but
> I do see a possible network issue connecting to DbServer3. I’m hoping for
> a little insight on what might have happened. The setup is 3 nodes, with
> one node (DbServer3) not allowed to failover and in another data center.
> I’m including the last few minutes of log files from each server. Did the
> failover occur because of the issues connecting to DbServer3? Even though
> Server1 and Server2 were fine?
>
>
> DBSERVER1- Node0:
> Jan 05 00:05:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:04: pid
> 9915: LOG: new connection received
> Jan 05 00:05:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:04: pid
> 9915: DETAIL: connecting host=10.153.199.194 port=53292
> Jan 05 00:05:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:04: pid
> 9915: LOCATION: child.c:2110
> Jan 05 00:05:05 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:05: pid
> 24127: LOG: child process with pid: 3178 exits with status 256
> Jan 05 00:05:05 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:05: pid
> 24127: LOCATION: pgpool_main.c:2195
> Jan 05 00:05:05 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:05: pid
> 24127: LOG: fork a new child process with pid: 13912
> Jan 05 00:05:05 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:05: pid
> 24127: LOCATION: pgpool_main.c:2281
> Jan 05 00:05:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:09: pid
> 13912: LOG: new connection received
> Jan 05 00:05:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:09: pid
> 13912: DETAIL: connecting host=10.153.197.151 port=48396
> Jan 05 00:05:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:09: pid
> 13912: LOCATION: child.c:2110
> Jan 05 00:05:13 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:13: pid
> 10608: LOG: new connection received
> Jan 05 00:05:13 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:13: pid
> 10608: DETAIL: connecting host=10.153.196.94 port=49742
> Jan 05 00:05:13 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:13: pid
> 10608: LOCATION: child.c:2110
> Jan 05 00:05:15 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:15: pid
> 29704: LOG: new connection received
> Jan 05 00:05:15 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:15: pid
> 29704: DETAIL: connecting host=10.153.197.151 port=48399
> Jan 05 00:05:15 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:15: pid
> 29704: LOCATION: child.c:2110
> Jan 05 00:05:23 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:23: pid
> 10608: LOG: trying connecting to PostgreSQL server on "dbserver3:5433"
> by INET socket
> Jan 05 00:05:23 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:23: pid
> 10608: DETAIL: timed out. retrying...
> Jan 05 00:05:23 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:23: pid
> 10608: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:33 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:33: pid
> 10608: LOG: trying connecting to PostgreSQL server on "dbserver3:5433"
> by INET socket
> Jan 05 00:05:33 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:33: pid
> 10608: DETAIL: timed out. retrying...
> Jan 05 00:05:33 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:33: pid
> 10608: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:43 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:43: pid
> 10608: LOG: trying connecting to PostgreSQL server on "dbserver3:5433"
> by INET socket
> Jan 05 00:05:43 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:43: pid
> 10608: DETAIL: timed out. retrying...
> Jan 05 00:05:43 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:43: pid
> 10608: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:47 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:47: pid
> 24127: LOG: child process with pid: 12008 exits with status 256
> Jan 05 00:05:47 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:47: pid
> 24127: LOCATION: pgpool_main.c:2195
> Jan 05 00:05:47 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:47: pid
> 24127: LOG: fork a new child process with pid: 14095
> Jan 05 00:05:47 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:47: pid
> 24127: LOCATION: pgpool_main.c:2281
> Jan 05 00:05:51 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:51: pid
> 24127: LOG: child process with pid: 7735 exits with status 256
> Jan 05 00:05:51 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:51: pid
> 24127: LOCATION: pgpool_main.c:2195
> Jan 05 00:05:51 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:51: pid
> 24127: LOG: fork a new child process with pid: 14096
> Jan 05 00:05:51 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:51: pid
> 24127: LOCATION: pgpool_main.c:2281
> Jan 05 00:06:00 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:00: pid
> 13090: LOG: new connection received
> Jan 05 00:06:00 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:00: pid
> 13090: DETAIL: connecting host=10.153.197.151 port=48443
> Jan 05 00:06:00 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:00: pid
> 13090: LOCATION: child.c:2110
> Jan 05 00:06:02 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:02: pid
> 14096: LOG: new connection received
> Jan 05 00:06:02 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:02: pid
> 14096: DETAIL: connecting host=10.153.198.80 port=41291
> Jan 05 00:06:02 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:02: pid
> 14096: LOCATION: child.c:2110
> Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid
> 13315: LOG: new connection received
> Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid
> 13315: DETAIL: connecting host=10.153.196.212 port=49587
> Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid
> 13315: LOCATION: child.c:2110
> Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid
> 12198: LOG: new connection received
> Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid
> 12198: DETAIL: connecting host=10.153.196.212 port=49588
> Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid
> 12198: LOCATION: child.c:2110
> Jan 05 00:06:06 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:06: pid
> 12898: LOG: new connection received
> Jan 05 00:06:06 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:06: pid
> 12898: DETAIL: connecting host=10.153.199.194 port=53421
> Jan 05 00:06:06 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:06: pid
> 12898: LOCATION: child.c:2110
> Jan 05 00:06:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:09: pid
> 24127: LOG: child process with pid: 6862 exits with status 256
> Jan 05 00:06:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:09: pid
> 24127: LOCATION: pgpool_main.c:2195
> Jan 05 00:06:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:09: pid
> 24127: LOG: fork a new child process with pid: 14164
> Jan 05 00:06:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:09: pid
> 24127: LOCATION: pgpool_main.c:2281
> Jan 05 00:06:28 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:28: pid
> 24127: LOG: child process with pid: 9661 exits with status 256
> Jan 05 00:06:28 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:28: pid
> 24127: LOCATION: pgpool_main.c:2195
> Jan 05 00:06:28 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:28: pid
> 24127: LOG: fork a new child process with pid: 14228
> Jan 05 00:06:28 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:28: pid
> 24127: LOCATION: pgpool_main.c:2281
> Jan 05 00:06:32 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:32: pid
> 24127: LOG: child process with pid: 10418 exits with status 256
> Jan 05 00:06:32 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:32: pid
> 24127: LOCATION: pgpool_main.c:2195
> Jan 05 00:06:32 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:32: pid
> 24127: LOG: fork a new child process with pid: 14229
> Jan 05 00:06:32 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:32: pid
> 24127: LOCATION: pgpool_main.c:2281
> Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid
> 24128: LOG: received degenerate backend request for node_id: 0 from pid
> [24128]
> Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid
> 24128: LOCATION: pgpool_main.c:1110
> Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid
> 24127: LOG: starting degeneration. shutdown host dbserver1(5433)
> Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid
> 24127: LOCATION: pgpool_main.c:1606
> Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid
> 24127: LOG: Restart all children
> Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid
> 24127: LOCATION: pgpool_main.c:1705
>
>
> DBSERVER2 - Node1:
> Jan 04 23:23:48 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:23:48: pid
> 13299: LOG: Replication of node:2 is behind 168 bytes from the primary
> server (node:0)
> Jan 04 23:23:48 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:23:48: pid
> 13299: CONTEXT: while checking replication time lag
> Jan 04 23:23:48 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:23:48: pid
> 13299: LOCATION: pool_worker_child.c:361
> Jan 04 23:28:52 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:28:52: pid
> 13299: LOG: Replication of node:1 is behind 112 bytes from the primary
> server (node:0)
> Jan 04 23:28:52 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:28:52: pid
> 13299: CONTEXT: while checking replication time lag
> Jan 04 23:28:52 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:28:52: pid
> 13299: LOCATION: pool_worker_child.c:361
> Jan 04 23:45:05 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:45:05: pid
> 13299: LOG: Replication of node:2 is behind 112 bytes from the primary
> server (node:0)
> Jan 04 23:45:05 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:45:05: pid
> 13299: CONTEXT: while checking replication time lag
> Jan 04 23:45:05 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:45:05: pid
> 13299: LOCATION: pool_worker_child.c:361
> Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid
> 13163: LOG: failed to connect to PostgreSQL server on "dbserver3:5433"
> using INET socket
> Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid
> 13163: DETAIL: select() system call failed with an error "Interrupted
> system call"
> Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid
> 13163: LOCATION: pool_connection_pool.c:700
> Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid
> 13163: ERROR: failed to make persistent db connection
> Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid
> 13163: DETAIL: connection to host:"dbserver3:5433" failed
> Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid
> 13163: LOCATION: child.c:1180
> Jan 05 00:03:19 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:03:19: pid
> 13299: LOG: Replication of node:2 is behind 11288 bytes from the primary
> server (node:0)
> Jan 05 00:03:19 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:03:19: pid
> 13299: CONTEXT: while checking replication time lag
> Jan 05 00:03:19 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:03:19: pid
> 13299: LOCATION: pool_worker_child.c:361
> Jan 05 00:04:29 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:29: pid
> 13299: LOG: trying connecting to PostgreSQL server on "dbserver3:5433"
> by INET socket
> Jan 05 00:04:29 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:29: pid
> 13299: DETAIL: timed out. retrying...
> Jan 05 00:04:29 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:29: pid
> 13299: LOCATION: pool_connection_pool.c:624
> Jan 05 00:04:35 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:35: pid
> 13299: LOG: Replication of node:2 is behind 6576 bytes from the primary
> server (node:0)
> Jan 05 00:04:35 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:35: pid
> 13299: CONTEXT: while checking replication time lag
> Jan 05 00:04:35 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:35: pid
> 13299: LOCATION: pool_worker_child.c:361
> Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid
> 13164: LOG: received degenerate backend request for node_id: 0 from pid
> [13164]
> Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid
> 13164: LOCATION: pgpool_main.c:1110
> Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid
> 13163: LOG: starting degeneration. shutdown host dbserver1(5433)
> Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid
> 13163: LOCATION: pgpool_main.c:1606
> Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid
> 13163: LOG: Restart all children
> Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid
> 13163: LOCATION: pgpool_main.c:1705
>
> DBSERVER3 - Node2:
> Jan 05 00:04:42 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:42: pid
> 18211: LOG: new connection received
> Jan 05 00:04:42 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:42: pid
> 18211: DETAIL: connecting host=10.177.198.144 port=56081
> Jan 05 00:04:42 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:42: pid
> 18211: LOCATION: child.c:2110
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 19465: LOG: new connection received
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 19465: DETAIL: connecting host=10.177.197.188 port=59880
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 19465: LOCATION: child.c:2110
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 2360: LOG: new connection received
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 2360: DETAIL: connecting host=10.177.197.188 port=59881
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 2360: LOCATION: child.c:2110
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 1125: LOG: new connection received
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 1125: DETAIL: connecting host=10.177.197.188 port=59882
> Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid
> 1125: LOCATION: child.c:2110
> Jan 05 00:04:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:48: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:04:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:48: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:04:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:48: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:04:50 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:50: pid
> 2758: LOG: new connection received
> Jan 05 00:04:50 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:50: pid
> 2758: DETAIL: connecting host=10.177.197.188 port=59949
> Jan 05 00:04:50 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:50: pid
> 2758: LOCATION: child.c:2110
> Jan 05 00:04:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:58: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:04:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:58: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:04:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:58: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:00 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:00: pid
> 15325: LOG: new connection received
> Jan 05 00:05:00 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:00: pid
> 15325: DETAIL: connecting host=10.177.196.145 port=39511
> Jan 05 00:05:00 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:00: pid
> 15325: LOCATION: child.c:2110
> Jan 05 00:05:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:08: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:05:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:08: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:05:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:08: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:18: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:05:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:18: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:05:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:18: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:28: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:05:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:28: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:05:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:28: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:38: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:05:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:38: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:05:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:38: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:48: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:05:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:48: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:05:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:48: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:05:55 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:55: pid
> 20634: LOG: new connection received
> Jan 05 00:05:55 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:55: pid
> 20634: DETAIL: connecting host=10.177.198.220 port=40178
> Jan 05 00:05:55 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:55: pid
> 20634: LOCATION: child.c:2110
> Jan 05 00:05:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:58: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:05:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:58: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:05:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:58: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:06:05 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:05: pid
> 4790: LOG: child process with pid: 17758 exits with status 256
> Jan 05 00:06:05 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:05: pid
> 4790: LOCATION: pgpool_main.c:2195
> Jan 05 00:06:05 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:05: pid
> 4790: LOG: fork a new child process with pid: 21581
> Jan 05 00:06:05 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:05: pid
> 4790: LOCATION: pgpool_main.c:2281
> Jan 05 00:06:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:08: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:06:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:08: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:06:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:08: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:06:15 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:15: pid
> 4790: LOG: child process with pid: 18279 exits with status 256
> Jan 05 00:06:15 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:15: pid
> 4790: LOCATION: pgpool_main.c:2195
> Jan 05 00:06:15 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:15: pid
> 4790: LOG: fork a new child process with pid: 21639
> Jan 05 00:06:15 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:15: pid
> 4790: LOCATION: pgpool_main.c:2281
> Jan 05 00:06:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:18: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:06:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:18: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:06:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:18: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:06:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:28: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:06:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:28: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:06:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:28: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:06:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:38: pid
> 12826: LOG: trying connecting to PostgreSQL server on "dbserver1:5433"
> by INET socket
> Jan 05 00:06:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:38: pid
> 12826: DETAIL: timed out. retrying...
> Jan 05 00:06:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:38: pid
> 12826: LOCATION: pool_connection_pool.c:624
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 12826: LOG: failed to connect to PostgreSQL server on "dbserver1:5433",
> getsockopt() detected error "Connection timed out"
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 12826: LOCATION: pool_connection_pool.c:661
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 12826: LOG: received degenerate backend request for node_id: 0 from pid
> [12826]
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 12826: LOCATION: pgpool_main.c:1110
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 4791: LOG: new IPC connection received
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 4791: LOCATION: watchdog.c:2441
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 12826: FATAL: failed to create a backend connection
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 12826: DETAIL: executing failover on backend
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 12826: LOCATION: pool_connection_pool.c:853
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 4790: LOG: starting degeneration. shutdown host dbserver1(5433)
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 4790: LOCATION: pgpool_main.c:1606
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 4790: LOG: Restart all children
> Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid
> 4790: LOCATION: pgpool_main.c:1705
>
>
>
> _______________________________________________
> pgpool-general mailing list
> pgpool-general at pgpool.net
> http://www.pgpool.net/mailman/listinfo/pgpool-general
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20170111/906962a2/attachment-0001.html>
More information about the pgpool-general
mailing list