[pgpool-general: 5223] Questionable Failover

Yates, James C. -ND James.C.Yates.-ND at disney.com
Wed Jan 11 04:04:46 JST 2017


PgPool initiated a failover last week, but the servers didn’t crash, but I do see a possible network issue connecting to DbServer3.  I’m hoping for a little insight on what might have happened.   The setup is 3 nodes, with one node (DbServer3) not allowed to failover and in another data center.  I’m including the last few minutes of log files from each server.  Did the failover occur because of the issues connecting to DbServer3?  Even though Server1 and Server2 were fine?


DBSERVER1- Node0:
Jan 05 00:05:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:04: pid 9915: LOG:  new connection received
Jan 05 00:05:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:04: pid 9915: DETAIL:  connecting host=10.153.199.194 port=53292
Jan 05 00:05:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:04: pid 9915: LOCATION:  child.c:2110
Jan 05 00:05:05 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:05: pid 24127: LOG:  child process with pid: 3178 exits with status 256
Jan 05 00:05:05 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:05: pid 24127: LOCATION:  pgpool_main.c:2195
Jan 05 00:05:05 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:05: pid 24127: LOG:  fork a new child process with pid: 13912
Jan 05 00:05:05 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:05: pid 24127: LOCATION:  pgpool_main.c:2281
Jan 05 00:05:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:09: pid 13912: LOG:  new connection received
Jan 05 00:05:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:09: pid 13912: DETAIL:  connecting host=10.153.197.151 port=48396
Jan 05 00:05:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:09: pid 13912: LOCATION:  child.c:2110
Jan 05 00:05:13 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:13: pid 10608: LOG:  new connection received
Jan 05 00:05:13 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:13: pid 10608: DETAIL:  connecting host=10.153.196.94 port=49742
Jan 05 00:05:13 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:13: pid 10608: LOCATION:  child.c:2110
Jan 05 00:05:15 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:15: pid 29704: LOG:  new connection received
Jan 05 00:05:15 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:15: pid 29704: DETAIL:  connecting host=10.153.197.151 port=48399
Jan 05 00:05:15 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:15: pid 29704: LOCATION:  child.c:2110
Jan 05 00:05:23 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:23: pid 10608: LOG:  trying connecting to PostgreSQL server on "dbserver3:5433" by INET socket
Jan 05 00:05:23 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:23: pid 10608: DETAIL:  timed out. retrying...
Jan 05 00:05:23 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:23: pid 10608: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:33 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:33: pid 10608: LOG:  trying connecting to PostgreSQL server on "dbserver3:5433" by INET socket
Jan 05 00:05:33 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:33: pid 10608: DETAIL:  timed out. retrying...
Jan 05 00:05:33 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:33: pid 10608: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:43 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:43: pid 10608: LOG:  trying connecting to PostgreSQL server on "dbserver3:5433" by INET socket
Jan 05 00:05:43 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:43: pid 10608: DETAIL:  timed out. retrying...
Jan 05 00:05:43 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:43: pid 10608: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:47 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:47: pid 24127: LOG:  child process with pid: 12008 exits with status 256
Jan 05 00:05:47 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:47: pid 24127: LOCATION:  pgpool_main.c:2195
Jan 05 00:05:47 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:47: pid 24127: LOG:  fork a new child process with pid: 14095
Jan 05 00:05:47 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:47: pid 24127: LOCATION:  pgpool_main.c:2281
Jan 05 00:05:51 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:51: pid 24127: LOG:  child process with pid: 7735 exits with status 256
Jan 05 00:05:51 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:51: pid 24127: LOCATION:  pgpool_main.c:2195
Jan 05 00:05:51 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:51: pid 24127: LOG:  fork a new child process with pid: 14096
Jan 05 00:05:51 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:05:51: pid 24127: LOCATION:  pgpool_main.c:2281
Jan 05 00:06:00 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:00: pid 13090: LOG:  new connection received
Jan 05 00:06:00 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:00: pid 13090: DETAIL:  connecting host=10.153.197.151 port=48443
Jan 05 00:06:00 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:00: pid 13090: LOCATION:  child.c:2110
Jan 05 00:06:02 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:02: pid 14096: LOG:  new connection received
Jan 05 00:06:02 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:02: pid 14096: DETAIL:  connecting host=10.153.198.80 port=41291
Jan 05 00:06:02 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:02: pid 14096: LOCATION:  child.c:2110
Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid 13315: LOG:  new connection received
Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid 13315: DETAIL:  connecting host=10.153.196.212 port=49587
Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid 13315: LOCATION:  child.c:2110
Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid 12198: LOG:  new connection received
Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid 12198: DETAIL:  connecting host=10.153.196.212 port=49588
Jan 05 00:06:04 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:04: pid 12198: LOCATION:  child.c:2110
Jan 05 00:06:06 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:06: pid 12898: LOG:  new connection received
Jan 05 00:06:06 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:06: pid 12898: DETAIL:  connecting host=10.153.199.194 port=53421
Jan 05 00:06:06 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:06: pid 12898: LOCATION:  child.c:2110
Jan 05 00:06:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:09: pid 24127: LOG:  child process with pid: 6862 exits with status 256
Jan 05 00:06:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:09: pid 24127: LOCATION:  pgpool_main.c:2195
Jan 05 00:06:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:09: pid 24127: LOG:  fork a new child process with pid: 14164
Jan 05 00:06:09 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:09: pid 24127: LOCATION:  pgpool_main.c:2281
Jan 05 00:06:28 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:28: pid 24127: LOG:  child process with pid: 9661 exits with status 256
Jan 05 00:06:28 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:28: pid 24127: LOCATION:  pgpool_main.c:2195
Jan 05 00:06:28 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:28: pid 24127: LOG:  fork a new child process with pid: 14228
Jan 05 00:06:28 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:28: pid 24127: LOCATION:  pgpool_main.c:2281
Jan 05 00:06:32 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:32: pid 24127: LOG:  child process with pid: 10418 exits with status 256
Jan 05 00:06:32 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:32: pid 24127: LOCATION:  pgpool_main.c:2195
Jan 05 00:06:32 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:32: pid 24127: LOG:  fork a new child process with pid: 14229
Jan 05 00:06:32 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:32: pid 24127: LOCATION:  pgpool_main.c:2281
Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid 24128: LOG:  received degenerate backend request for node_id: 0 from pid [24128]
Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid 24128: LOCATION:  pgpool_main.c:1110
Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid 24127: LOG:  starting degeneration. shutdown host dbserver1(5433)
Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid 24127: LOCATION:  pgpool_main.c:1606
Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid 24127: LOG:  Restart all children
Jan 05 00:06:45 ip-10-153-96-210 pgpool[24127]: 2017-01-05 00:06:45: pid 24127: LOCATION:  pgpool_main.c:1705


DBSERVER2 - Node1:
Jan 04 23:23:48 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:23:48: pid 13299: LOG:  Replication of node:2 is behind 168 bytes from the primary server (node:0)
Jan 04 23:23:48 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:23:48: pid 13299: CONTEXT:  while checking replication time lag
Jan 04 23:23:48 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:23:48: pid 13299: LOCATION:  pool_worker_child.c:361
Jan 04 23:28:52 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:28:52: pid 13299: LOG:  Replication of node:1 is behind 112 bytes from the primary server (node:0)
Jan 04 23:28:52 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:28:52: pid 13299: CONTEXT:  while checking replication time lag
Jan 04 23:28:52 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:28:52: pid 13299: LOCATION:  pool_worker_child.c:361
Jan 04 23:45:05 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:45:05: pid 13299: LOG:  Replication of node:2 is behind 112 bytes from the primary server (node:0)
Jan 04 23:45:05 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:45:05: pid 13299: CONTEXT:  while checking replication time lag
Jan 04 23:45:05 ip-10-153-96-140 pgpool[13163]: 2017-01-04 23:45:05: pid 13299: LOCATION:  pool_worker_child.c:361
Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid 13163: LOG:  failed to connect to PostgreSQL server on "dbserver3:5433" using INET socket
Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid 13163: DETAIL:  select() system call failed with an error "Interrupted system call"
Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid 13163: LOCATION:  pool_connection_pool.c:700
Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid 13163: ERROR:  failed to make persistent db connection
Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid 13163: DETAIL:  connection to host:"dbserver3:5433" failed
Jan 05 00:00:24 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:00:24: pid 13163: LOCATION:  child.c:1180
Jan 05 00:03:19 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:03:19: pid 13299: LOG:  Replication of node:2 is behind 11288 bytes from the primary server (node:0)
Jan 05 00:03:19 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:03:19: pid 13299: CONTEXT:  while checking replication time lag
Jan 05 00:03:19 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:03:19: pid 13299: LOCATION:  pool_worker_child.c:361
Jan 05 00:04:29 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:29: pid 13299: LOG:  trying connecting to PostgreSQL server on "dbserver3:5433" by INET socket
Jan 05 00:04:29 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:29: pid 13299: DETAIL:  timed out. retrying...
Jan 05 00:04:29 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:29: pid 13299: LOCATION:  pool_connection_pool.c:624
Jan 05 00:04:35 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:35: pid 13299: LOG:  Replication of node:2 is behind 6576 bytes from the primary server (node:0)
Jan 05 00:04:35 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:35: pid 13299: CONTEXT:  while checking replication time lag
Jan 05 00:04:35 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:04:35: pid 13299: LOCATION:  pool_worker_child.c:361
Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid 13164: LOG:  received degenerate backend request for node_id: 0 from pid [13164]
Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid 13164: LOCATION:  pgpool_main.c:1110
Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid 13163: LOG:  starting degeneration. shutdown host dbserver1(5433)
Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid 13163: LOCATION:  pgpool_main.c:1606
Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid 13163: LOG:  Restart all children
Jan 05 00:06:45 ip-10-153-96-140 pgpool[13163]: 2017-01-05 00:06:45: pid 13163: LOCATION:  pgpool_main.c:1705

DBSERVER3 - Node2:
Jan 05 00:04:42 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:42: pid 18211: LOG:  new connection received
Jan 05 00:04:42 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:42: pid 18211: DETAIL:  connecting host=10.177.198.144 port=56081
Jan 05 00:04:42 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:42: pid 18211: LOCATION:  child.c:2110
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 19465: LOG:  new connection received
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 19465: DETAIL:  connecting host=10.177.197.188 port=59880
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 19465: LOCATION:  child.c:2110
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 2360: LOG:  new connection received
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 2360: DETAIL:  connecting host=10.177.197.188 port=59881
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 2360: LOCATION:  child.c:2110
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 1125: LOG:  new connection received
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 1125: DETAIL:  connecting host=10.177.197.188 port=59882
Jan 05 00:04:46 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:46: pid 1125: LOCATION:  child.c:2110
Jan 05 00:04:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:48: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:04:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:48: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:04:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:48: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:04:50 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:50: pid 2758: LOG:  new connection received
Jan 05 00:04:50 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:50: pid 2758: DETAIL:  connecting host=10.177.197.188 port=59949
Jan 05 00:04:50 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:50: pid 2758: LOCATION:  child.c:2110
Jan 05 00:04:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:58: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:04:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:58: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:04:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:04:58: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:00 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:00: pid 15325: LOG:  new connection received
Jan 05 00:05:00 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:00: pid 15325: DETAIL:  connecting host=10.177.196.145 port=39511
Jan 05 00:05:00 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:00: pid 15325: LOCATION:  child.c:2110
Jan 05 00:05:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:08: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:05:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:08: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:05:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:08: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:18: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:05:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:18: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:05:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:18: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:28: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:05:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:28: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:05:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:28: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:38: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:05:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:38: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:05:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:38: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:48: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:05:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:48: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:05:48 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:48: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:05:55 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:55: pid 20634: LOG:  new connection received
Jan 05 00:05:55 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:55: pid 20634: DETAIL:  connecting host=10.177.198.220 port=40178
Jan 05 00:05:55 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:55: pid 20634: LOCATION:  child.c:2110
Jan 05 00:05:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:58: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:05:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:58: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:05:58 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:05:58: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:06:05 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:05: pid 4790: LOG:  child process with pid: 17758 exits with status 256
Jan 05 00:06:05 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:05: pid 4790: LOCATION:  pgpool_main.c:2195
Jan 05 00:06:05 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:05: pid 4790: LOG:  fork a new child process with pid: 21581
Jan 05 00:06:05 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:05: pid 4790: LOCATION:  pgpool_main.c:2281
Jan 05 00:06:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:08: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:06:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:08: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:06:08 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:08: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:06:15 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:15: pid 4790: LOG:  child process with pid: 18279 exits with status 256
Jan 05 00:06:15 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:15: pid 4790: LOCATION:  pgpool_main.c:2195
Jan 05 00:06:15 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:15: pid 4790: LOG:  fork a new child process with pid: 21639
Jan 05 00:06:15 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:15: pid 4790: LOCATION:  pgpool_main.c:2281
Jan 05 00:06:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:18: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:06:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:18: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:06:18 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:18: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:06:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:28: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:06:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:28: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:06:28 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:28: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:06:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:38: pid 12826: LOG:  trying connecting to PostgreSQL server on "dbserver1:5433" by INET socket
Jan 05 00:06:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:38: pid 12826: DETAIL:  timed out. retrying...
Jan 05 00:06:38 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:38: pid 12826: LOCATION:  pool_connection_pool.c:624
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 12826: LOG:  failed to connect to PostgreSQL server on "dbserver1:5433", getsockopt() detected error "Connection timed out"
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 12826: LOCATION:  pool_connection_pool.c:661
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 12826: LOG:  received degenerate backend request for node_id: 0 from pid [12826]
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 12826: LOCATION:  pgpool_main.c:1110
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 4791: LOG:  new IPC connection received
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 4791: LOCATION:  watchdog.c:2441
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 12826: FATAL:  failed to create a backend connection
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 12826: DETAIL:  executing failover on backend
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 12826: LOCATION:  pool_connection_pool.c:853
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 4790: LOG:  starting degeneration. shutdown host dbserver1(5433)
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 4790: LOCATION:  pgpool_main.c:1606
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 4790: LOG:  Restart all children
Jan 05 00:06:45 ip-10-177-96-155 pgpool[4790]: 2017-01-05 00:06:45: pid 4790: LOCATION:  pgpool_main.c:1705


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20170110/211943f8/attachment-0001.html>


More information about the pgpool-general mailing list