View Issue Details

IDProjectCategoryView StatusLast Update
0000177Pgpool-IIBugpublic2019-02-15 00:20
ReporterswAssigned ToMuhammad Usama 
PrioritynormalSeveritymajorReproducibilityalways
Status assignedResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0000177: Pgpool stops accepting connections when Postgresql slave goes down
DescriptionUbuntu 14.04 TLS (Trusty)
pgpool2 3.3.7
postgresql 9.3.5

Postgresql master/slave with streaming replication. Pgpool stops accepting connects when Postgresql slave goes down and never recovers until it is restarted.
Steps To Reproduce# service pgpool2 start

#############################################################

Mar 19 11:50:07 hostname pgpool: 2016-03-19 10:50:07 LOG: pid 9152: pgpool-II successfully started. version 3.3.7 (tokakiboshi)
Mar 19 11:50:07 hostname pgpool: 2016-03-19 10:50:07 LOG: pid 9152: find_primary_node: primary node id is 0

#############################################################

# pcp show
0: 192.168.181.230 5432 2 0.000000
1: 192.168.181.57 5432 2 1.000000
            
# Stopped Postgresql slave (node 1), IP: 192.168.181.57

#############################################################

Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 LOG: pid 9162: connection closed. retry to create new connection pool.
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9162: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9162: connection to 192.168.181.57(5432) failed
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9162: new_connection: create_cp() failed
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 LOG: pid 9162: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 LOG: pid 9165: connection closed. retry to create new connection pool.
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9165: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9165: connection to 192.168.181.57(5432) failed
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9165: new_connection: create_cp() failed
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 LOG: pid 9165: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 LOG: pid 9161: connection closed. retry to create new connection pool.
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9161: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9161: connection to 192.168.181.57(5432) failed
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 ERROR: pid 9161: new_connection: create_cp() failed
Mar 19 11:51:19 hostname pgpool: 2016-03-19 10:51:19 LOG: pid 9161: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9169: connection closed. retry to create new connection pool.
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9169: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9169: connection to 192.168.181.57(5432) failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9169: new_connection: create_cp() failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9169: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9168: connection closed. retry to create new connection pool.
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9168: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9168: connection to 192.168.181.57(5432) failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9168: new_connection: create_cp() failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9168: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9163: connection closed. retry to create new connection pool.
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9163: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9163: connection to 192.168.181.57(5432) failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9163: new_connection: create_cp() failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9163: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9160: connection closed. retry to create new connection pool.
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9160: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9160: connection to 192.168.181.57(5432) failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9160: new_connection: create_cp() failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9160: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9164: connection closed. retry to create new connection pool.
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9164: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9164: connection to 192.168.181.57(5432) failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9164: new_connection: create_cp() failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9164: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9166: connection closed. retry to create new connection pool.
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9166: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9166: connection to 192.168.181.57(5432) failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9166: new_connection: create_cp() failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9166: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9167: connection closed. retry to create new connection pool.
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9167: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9167: connection to 192.168.181.57(5432) failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 ERROR: pid 9167: new_connection: create_cp() failed
Mar 19 11:51:20 hostname pgpool: 2016-03-19 10:51:20 LOG: pid 9167: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10589: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10589: connection to 192.168.181.57(5432) failed
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10589: new_connection: create_cp() failed
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 LOG: pid 10589: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10590: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10590: connection to 192.168.181.57(5432) failed
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10590: new_connection: create_cp() failed
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 LOG: pid 10590: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10630: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10630: connection to 192.168.181.57(5432) failed
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10630: new_connection: create_cp() failed
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 LOG: pid 10630: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10646: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10646: connection to 192.168.181.57(5432) failed
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 ERROR: pid 10646: new_connection: create_cp() failed
Mar 19 11:51:21 hostname pgpool: 2016-03-19 10:51:21 LOG: pid 10646: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10650: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10650: connection to 192.168.181.57(5432) failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10650: new_connection: create_cp() failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 LOG: pid 10650: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10707: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10652: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10652: connection to 192.168.181.57(5432) failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10652: new_connection: create_cp() failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 LOG: pid 10652: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10707: connection to 192.168.181.57(5432) failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10707: new_connection: create_cp() failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 LOG: pid 10707: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10710: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10710: connection to 192.168.181.57(5432) failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10710: new_connection: create_cp() failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 LOG: pid 10710: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10711: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10711: connection to 192.168.181.57(5432) failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10711: new_connection: create_cp() failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 LOG: pid 10711: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10761: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10761: connection to 192.168.181.57(5432) failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10761: new_connection: create_cp() failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 LOG: pid 10761: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10906: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10906: connection to 192.168.181.57(5432) failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 ERROR: pid 10906: new_connection: create_cp() failed
Mar 19 11:51:22 hostname pgpool: 2016-03-19 10:51:22 LOG: pid 10906: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10939: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10939: connection to 192.168.181.57(5432) failed
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10939: new_connection: create_cp() failed
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 LOG: pid 10939: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10651: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10651: connection to 192.168.181.57(5432) failed
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10651: new_connection: create_cp() failed
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 LOG: pid 10651: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10944: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10944: connection to 192.168.181.57(5432) failed
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 ERROR: pid 10944: new_connection: create_cp() failed
Mar 19 11:51:23 hostname pgpool: 2016-03-19 10:51:23 LOG: pid 10944: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10946: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10946: connection to 192.168.181.57(5432) failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10946: new_connection: create_cp() failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 LOG: pid 10946: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10947: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10947: connection to 192.168.181.57(5432) failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10945: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10947: new_connection: create_cp() failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 LOG: pid 10947: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10945: connection to 192.168.181.57(5432) failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10945: new_connection: create_cp() failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 LOG: pid 10945: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10872: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10872: connection to 192.168.181.57(5432) failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10872: new_connection: create_cp() failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 LOG: pid 10872: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10960: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10960: connection to 192.168.181.57(5432) failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10960: new_connection: create_cp() failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 LOG: pid 10960: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10954: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10954: connection to 192.168.181.57(5432) failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10954: new_connection: create_cp() failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10966: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 LOG: pid 10954: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10966: connection to 192.168.181.57(5432) failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 ERROR: pid 10966: new_connection: create_cp() failed
Mar 19 11:51:24 hostname pgpool: 2016-03-19 10:51:24 LOG: pid 10966: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10971: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10971: connection to 192.168.181.57(5432) failed
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10971: new_connection: create_cp() failed
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 LOG: pid 10971: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10973: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10973: connection to 192.168.181.57(5432) failed
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10973: new_connection: create_cp() failed
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 LOG: pid 10973: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10974: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10974: connection to 192.168.181.57(5432) failed
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 ERROR: pid 10974: new_connection: create_cp() failed
Mar 19 11:51:25 hostname pgpool: 2016-03-19 10:51:25 LOG: pid 10974: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10955: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10955: connection to 192.168.181.57(5432) failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10955: new_connection: create_cp() failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 LOG: pid 10955: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10961: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10961: connection to 192.168.181.57(5432) failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10961: new_connection: create_cp() failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 LOG: pid 10961: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10980: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10953: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10953: connection to 192.168.181.57(5432) failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10953: new_connection: create_cp() failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 LOG: pid 10953: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10980: connection to 192.168.181.57(5432) failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10980: new_connection: create_cp() failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 LOG: pid 10980: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10981: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10981: connection to 192.168.181.57(5432) failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10981: new_connection: create_cp() failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 LOG: pid 10981: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10982: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10982: connection to 192.168.181.57(5432) failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10982: new_connection: create_cp() failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 LOG: pid 10982: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10989: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10989: connection to 192.168.181.57(5432) failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 ERROR: pid 10989: new_connection: create_cp() failed
Mar 19 11:51:26 hostname pgpool: 2016-03-19 10:51:26 LOG: pid 10989: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 9152: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 9152: make_persistent_db_connection: connection to 192.168.181.57(5432) failed
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 9152: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 9152: make_persistent_db_connection: connection to 192.168.181.57(5432) failed
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 9152: health check failed. 1 th host 192.168.181.57 at port 5432 is down
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 LOG: pid 9152: health check retry sleep time: 5 second(s)
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 10990: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 10990: connection to 192.168.181.57(5432) failed
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 10990: new_connection: create_cp() failed
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 LOG: pid 10990: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 10992: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 10992: connection to 192.168.181.57(5432) failed
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 10992: new_connection: create_cp() failed
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 LOG: pid 10992: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 9171: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 9171: make_persistent_db_connection: connection to 192.168.181.57(5432) failed
Mar 19 11:51:27 hostname pgpool: 2016-03-19 10:51:27 ERROR: pid 9171: check_replication_time_lag: could not connect to DB node 1, check sr_check_user and sr_check_password
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11000: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11000: connection to 192.168.181.57(5432) failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11000: new_connection: create_cp() failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 LOG: pid 11000: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11001: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11001: connection to 192.168.181.57(5432) failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11001: new_connection: create_cp() failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 LOG: pid 11001: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11003: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11003: connection to 192.168.181.57(5432) failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11003: new_connection: create_cp() failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 LOG: pid 11003: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11005: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11005: connection to 192.168.181.57(5432) failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11005: new_connection: create_cp() failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 LOG: pid 11005: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11006: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11006: connection to 192.168.181.57(5432) failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11006: new_connection: create_cp() failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 LOG: pid 11006: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11007: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11007: connection to 192.168.181.57(5432) failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11007: new_connection: create_cp() failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 LOG: pid 11007: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11017: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11017: connection to 192.168.181.57(5432) failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11017: new_connection: create_cp() failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 LOG: pid 11017: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11015: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11015: connection to 192.168.181.57(5432) failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 ERROR: pid 11015: new_connection: create_cp() failed
Mar 19 11:51:28 hostname pgpool: 2016-03-19 10:51:28 LOG: pid 11015: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:29 hostname pgpool: 2016-03-19 10:51:29 ERROR: pid 10972: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:29 hostname pgpool: 2016-03-19 10:51:29 ERROR: pid 10972: connection to 192.168.181.57(5432) failed
Mar 19 11:51:29 hostname pgpool: 2016-03-19 10:51:29 ERROR: pid 10972: new_connection: create_cp() failed
Mar 19 11:51:29 hostname pgpool: 2016-03-19 10:51:29 LOG: pid 10972: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:29 hostname pgpool: 2016-03-19 10:51:29 ERROR: pid 11023: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:29 hostname pgpool: 2016-03-19 10:51:29 ERROR: pid 11023: connection to 192.168.181.57(5432) failed
Mar 19 11:51:29 hostname pgpool: 2016-03-19 10:51:29 ERROR: pid 11023: new_connection: create_cp() failed
Mar 19 11:51:29 hostname pgpool: 2016-03-19 10:51:29 LOG: pid 11023: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11024: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11024: connection to 192.168.181.57(5432) failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11024: new_connection: create_cp() failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 LOG: pid 11024: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11025: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11025: connection to 192.168.181.57(5432) failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11025: new_connection: create_cp() failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 LOG: pid 11025: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11016: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11016: connection to 192.168.181.57(5432) failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11016: new_connection: create_cp() failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 LOG: pid 11016: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11002: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11002: connection to 192.168.181.57(5432) failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11002: new_connection: create_cp() failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 LOG: pid 11002: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11080: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11080: connection to 192.168.181.57(5432) failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 ERROR: pid 11080: new_connection: create_cp() failed
Mar 19 11:51:30 hostname pgpool: 2016-03-19 10:51:30 LOG: pid 11080: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11079: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11079: connection to 192.168.181.57(5432) failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11079: new_connection: create_cp() failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 LOG: pid 11079: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11091: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11091: connection to 192.168.181.57(5432) failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11091: new_connection: create_cp() failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 LOG: pid 11091: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11097: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11097: connection to 192.168.181.57(5432) failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11097: new_connection: create_cp() failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 LOG: pid 11097: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11026: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11026: connection to 192.168.181.57(5432) failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11026: new_connection: create_cp() failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 LOG: pid 11026: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11123: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11123: connection to 192.168.181.57(5432) failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 ERROR: pid 11123: new_connection: create_cp() failed
Mar 19 11:51:31 hostname pgpool: 2016-03-19 10:51:31 LOG: pid 11123: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11124: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11124: connection to 192.168.181.57(5432) failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11124: new_connection: create_cp() failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 LOG: pid 11124: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 9152: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 9152: make_persistent_db_connection: connection to 192.168.181.57(5432) failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 9152: health check failed. 1 th host 192.168.181.57 at port 5432 is down
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 LOG: pid 9152: health check retry sleep time: 5 second(s)
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11125: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11125: connection to 192.168.181.57(5432) failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11125: new_connection: create_cp() failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 LOG: pid 11125: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11081: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11081: connection to 192.168.181.57(5432) failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11081: new_connection: create_cp() failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 LOG: pid 11081: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11093: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11093: connection to 192.168.181.57(5432) failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11093: new_connection: create_cp() failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 LOG: pid 11093: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11130: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11130: connection to 192.168.181.57(5432) failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11130: new_connection: create_cp() failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 LOG: pid 11130: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11131: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11131: connection to 192.168.181.57(5432) failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 ERROR: pid 11131: new_connection: create_cp() failed
Mar 19 11:51:32 hostname pgpool: 2016-03-19 10:51:32 LOG: pid 11131: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11132: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11132: connection to 192.168.181.57(5432) failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11132: new_connection: create_cp() failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 LOG: pid 11132: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11136: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11136: connection to 192.168.181.57(5432) failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11136: new_connection: create_cp() failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 LOG: pid 11136: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11137: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11137: connection to 192.168.181.57(5432) failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11137: new_connection: create_cp() failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 LOG: pid 11137: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11139: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11139: connection to 192.168.181.57(5432) failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11139: new_connection: create_cp() failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 LOG: pid 11139: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11126: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11126: connection to 192.168.181.57(5432) failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 ERROR: pid 11126: new_connection: create_cp() failed
Mar 19 11:51:33 hostname pgpool: 2016-03-19 10:51:33 LOG: pid 11126: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11147: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11147: connection to 192.168.181.57(5432) failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11147: new_connection: create_cp() failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 LOG: pid 11147: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11150: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11150: connection to 192.168.181.57(5432) failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11150: new_connection: create_cp() failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 LOG: pid 11150: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11148: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11148: connection to 192.168.181.57(5432) failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11148: new_connection: create_cp() failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 LOG: pid 11148: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11159: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11159: connection to 192.168.181.57(5432) failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11159: new_connection: create_cp() failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 LOG: pid 11159: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11165: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11165: connection to 192.168.181.57(5432) failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 ERROR: pid 11165: new_connection: create_cp() failed
Mar 19 11:51:34 hostname pgpool: 2016-03-19 10:51:34 LOG: pid 11165: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11166: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11166: connection to 192.168.181.57(5432) failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11166: new_connection: create_cp() failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 LOG: pid 11166: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11174: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11149: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11149: connection to 192.168.181.57(5432) failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11149: new_connection: create_cp() failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 LOG: pid 11149: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11174: connection to 192.168.181.57(5432) failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11174: new_connection: create_cp() failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 LOG: pid 11174: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11164: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11164: connection to 192.168.181.57(5432) failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11164: new_connection: create_cp() failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 LOG: pid 11164: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11179: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11179: connection to 192.168.181.57(5432) failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 ERROR: pid 11179: new_connection: create_cp() failed
Mar 19 11:51:35 hostname pgpool: 2016-03-19 10:51:35 LOG: pid 11179: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11181: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11181: connection to 192.168.181.57(5432) failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11181: new_connection: create_cp() failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 LOG: pid 11181: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11182: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11182: connection to 192.168.181.57(5432) failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11182: new_connection: create_cp() failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 LOG: pid 11182: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11175: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11175: connection to 192.168.181.57(5432) failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11175: new_connection: create_cp() failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 LOG: pid 11175: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11187: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11187: connection to 192.168.181.57(5432) failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11187: new_connection: create_cp() failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 LOG: pid 11187: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11188: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11188: connection to 192.168.181.57(5432) failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 ERROR: pid 11188: new_connection: create_cp() failed
Mar 19 11:51:36 hostname pgpool: 2016-03-19 10:51:36 LOG: pid 11188: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 9152: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 9152: make_persistent_db_connection: connection to 192.168.181.57(5432) failed
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 9152: health check failed. 1 th host 192.168.181.57 at port 5432 is down
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 LOG: pid 9152: health check retry sleep time: 5 second(s)
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11180: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11180: connection to 192.168.181.57(5432) failed
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11180: new_connection: create_cp() failed
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 LOG: pid 11180: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11192: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11192: connection to 192.168.181.57(5432) failed
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11192: new_connection: create_cp() failed
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 LOG: pid 11192: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11193: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11193: connection to 192.168.181.57(5432) failed
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 11193: new_connection: create_cp() failed
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 LOG: pid 11193: new_connection: do not failover because fail_over_on_backend_error is off

#############################################################

# psql -h localhost -p 9999 mydb myuser
psql: SSL SYSCALL error: EOF detected

#############################################################

Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 9171: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 9171: make_persistent_db_connection: connection to 192.168.181.57(5432) failed
Mar 19 11:51:37 hostname pgpool: 2016-03-19 10:51:37 ERROR: pid 9171: check_replication_time_lag: could not connect to DB node 1, check sr_check_user and sr_check_password
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11198: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11198: connection to 192.168.181.57(5432) failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11198: new_connection: create_cp() failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 LOG: pid 11198: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11173: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11173: connection to 192.168.181.57(5432) failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11173: new_connection: create_cp() failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 LOG: pid 11173: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11202: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11202: connection to 192.168.181.57(5432) failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11202: new_connection: create_cp() failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 LOG: pid 11202: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11186: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11186: connection to 192.168.181.57(5432) failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11186: new_connection: create_cp() failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 LOG: pid 11186: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11203: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11203: connection to 192.168.181.57(5432) failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11203: new_connection: create_cp() failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 LOG: pid 11203: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11205: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11206: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11206: connection to 192.168.181.57(5432) failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11205: connection to 192.168.181.57(5432) failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11206: new_connection: create_cp() failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 LOG: pid 11206: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 ERROR: pid 11205: new_connection: create_cp() failed
Mar 19 11:51:38 hostname pgpool: 2016-03-19 10:51:38 LOG: pid 11205: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11266: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11266: connection to 192.168.181.57(5432) failed
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11266: new_connection: create_cp() failed
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 LOG: pid 11266: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11208: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11208: connection to 192.168.181.57(5432) failed
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11208: new_connection: create_cp() failed
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 LOG: pid 11208: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11270: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11270: connection to 192.168.181.57(5432) failed
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 ERROR: pid 11270: new_connection: create_cp() failed
Mar 19 11:51:39 hostname pgpool: 2016-03-19 10:51:39 LOG: pid 11270: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11204: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11204: connection to 192.168.181.57(5432) failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11204: new_connection: create_cp() failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 LOG: pid 11204: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11274: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11274: connection to 192.168.181.57(5432) failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11274: new_connection: create_cp() failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 LOG: pid 11274: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11207: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11207: connection to 192.168.181.57(5432) failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11207: new_connection: create_cp() failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 LOG: pid 11207: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11276: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11276: connection to 192.168.181.57(5432) failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11276: new_connection: create_cp() failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 LOG: pid 11276: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11277: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11277: connection to 192.168.181.57(5432) failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11277: new_connection: create_cp() failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 LOG: pid 11277: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11267: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11267: connection to 192.168.181.57(5432) failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11267: new_connection: create_cp() failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 LOG: pid 11267: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11281: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11281: connection to 192.168.181.57(5432) failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 ERROR: pid 11281: new_connection: create_cp() failed
Mar 19 11:51:40 hostname pgpool: 2016-03-19 10:51:40 LOG: pid 11281: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11282: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11282: connection to 192.168.181.57(5432) failed
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11282: new_connection: create_cp() failed
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 LOG: pid 11282: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11275: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11275: connection to 192.168.181.57(5432) failed
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11275: new_connection: create_cp() failed
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 LOG: pid 11275: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11292: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11292: connection to 192.168.181.57(5432) failed
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 ERROR: pid 11292: new_connection: create_cp() failed
Mar 19 11:51:41 hostname pgpool: 2016-03-19 10:51:41 LOG: pid 11292: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11302: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11302: connection to 192.168.181.57(5432) failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11302: new_connection: create_cp() failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 LOG: pid 11302: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11293: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11293: connection to 192.168.181.57(5432) failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11293: new_connection: create_cp() failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 LOG: pid 11293: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 9152: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 9152: make_persistent_db_connection: connection to 192.168.181.57(5432) failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 9152: health check failed. 1 th host 192.168.181.57 at port 5432 is down
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 LOG: pid 9152: health check retry sleep time: 5 second(s)
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11303: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11303: connection to 192.168.181.57(5432) failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11303: new_connection: create_cp() failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 LOG: pid 11303: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11305: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11305: connection to 192.168.181.57(5432) failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11305: new_connection: create_cp() failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 LOG: pid 11305: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11283: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11283: connection to 192.168.181.57(5432) failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11283: new_connection: create_cp() failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 LOG: pid 11283: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11309: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11309: connection to 192.168.181.57(5432) failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11309: new_connection: create_cp() failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 LOG: pid 11309: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11314: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11314: connection to 192.168.181.57(5432) failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 ERROR: pid 11314: new_connection: create_cp() failed
Mar 19 11:51:42 hostname pgpool: 2016-03-19 10:51:42 LOG: pid 11314: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11315: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11315: connection to 192.168.181.57(5432) failed
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11315: new_connection: create_cp() failed
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 LOG: pid 11315: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11304: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11304: connection to 192.168.181.57(5432) failed
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11304: new_connection: create_cp() failed
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 LOG: pid 11304: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11320: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11320: connection to 192.168.181.57(5432) failed
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 ERROR: pid 11320: new_connection: create_cp() failed
Mar 19 11:51:43 hostname pgpool: 2016-03-19 10:51:43 LOG: pid 11320: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11325: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11325: connection to 192.168.181.57(5432) failed
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11325: new_connection: create_cp() failed
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 LOG: pid 11325: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11313: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11313: connection to 192.168.181.57(5432) failed
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11313: new_connection: create_cp() failed
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 LOG: pid 11313: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11300: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11300: connection to 192.168.181.57(5432) failed
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11300: new_connection: create_cp() failed
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 LOG: pid 11300: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11326: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11326: connection to 192.168.181.57(5432) failed
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 ERROR: pid 11326: new_connection: create_cp() failed
Mar 19 11:51:44 hostname pgpool: 2016-03-19 10:51:44 LOG: pid 11326: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11327: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11327: connection to 192.168.181.57(5432) failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11327: new_connection: create_cp() failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 LOG: pid 11327: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11316: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11316: connection to 192.168.181.57(5432) failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11316: new_connection: create_cp() failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 LOG: pid 11316: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11324: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11324: connection to 192.168.181.57(5432) failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11324: new_connection: create_cp() failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 LOG: pid 11324: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11332: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11332: connection to 192.168.181.57(5432) failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11332: new_connection: create_cp() failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 LOG: pid 11332: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11333: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11338: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11338: connection to 192.168.181.57(5432) failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11338: new_connection: create_cp() failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 LOG: pid 11338: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11333: connection to 192.168.181.57(5432) failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 ERROR: pid 11333: new_connection: create_cp() failed
Mar 19 11:51:45 hostname pgpool: 2016-03-19 10:51:45 LOG: pid 11333: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11339: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11339: connection to 192.168.181.57(5432) failed
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11339: new_connection: create_cp() failed
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 LOG: pid 11339: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11343: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11343: connection to 192.168.181.57(5432) failed
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11343: new_connection: create_cp() failed
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 LOG: pid 11343: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11344: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11344: connection to 192.168.181.57(5432) failed
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 ERROR: pid 11344: new_connection: create_cp() failed
Mar 19 11:51:46 hostname pgpool: 2016-03-19 10:51:46 LOG: pid 11344: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 ERROR: pid 11346: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 ERROR: pid 11346: connection to 192.168.181.57(5432) failed
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 ERROR: pid 11346: new_connection: create_cp() failed
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 LOG: pid 11346: new_connection: do not failover because fail_over_on_backend_error is off
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 ERROR: pid 11334: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 ERROR: pid 9152: connect_inet_domain_socket: getsockopt() detected error: Connection refused
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 ERROR: pid 9152: make_persistent_db_connection: connection to 192.168.181.57(5432) failed
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 ERROR: pid 9152: health check failed. 1 th host 192.168.181.57 at port 5432 is down
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 LOG: pid 9152: set 1 th backend down status
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 LOG: pid 9152: starting degeneration. shutdown host 192.168.181.57(5432)
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 LOG: pid 9152: Restart all children
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 LOG: pid 9152: failover: set new primary node: 0
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 LOG: pid 9152: failover: set new master node: 0
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 ERROR: pid 11334: connection to 192.168.181.57(5432) failed
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 LOG: pid 9171: worker process received restart request
Mar 19 11:51:47 hostname pgpool: 2016-03-19 10:51:47 LOG: pid 9152: failover done. shutdown host 192.168.181.57(5432)
Mar 19 11:51:48 hostname pgpool: 2016-03-19 10:51:48 LOG: pid 9170: pcp child process received restart request
Mar 19 11:51:48 hostname pgpool: 2016-03-19 10:51:48 LOG: pid 9152: PCP child 9170 exits with status 256 in failover()
Mar 19 11:51:48 hostname pgpool: 2016-03-19 10:51:48 LOG: pid 9152: fork a new PCP child pid 11383 in failover()
Mar 19 11:51:48 hostname pgpool: 2016-03-19 10:51:48 LOG: pid 9152: worker child 9171 exits with status 256
Mar 19 11:51:48 hostname pgpool: 2016-03-19 10:51:48 LOG: pid 9152: fork a new worker child pid 11384

#############################################################

# pcp show
0: 192.168.181.230 5432 2 0.000000
1: 192.168.181.57 5432 3 1.000000

# psql -h localhost -p 9999 mydb myuser
psql: could not connect to server: Connection timed out
    Is the server running on host "localhost" (127.0.0.1) and accepting
    TCP/IP connections on port 9999?


# Started Postgresql slave (node 1)
# No action in pgpool

# pcp attach 1 (attach node 1 to pgpool)

#############################################################

Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 11383: send_failback_request: fail back 1 th node request from pid 11383
Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 9152: starting fail back. reconnect host 192.168.181.57(5432)
Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 9152: Do not restart children because we are failbacking node id 1 host192.168.181.57 port:5432 and we are in streaming replication mode
Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 9152: find_primary_node_repeatedly: waiting for finding a primary node
Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 9152: find_primary_node: primary node id is 0
Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 9152: failover: set new primary node: 0
Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 9152: failover: set new master node: 0
Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 9152: failback done. reconnect host 192.168.181.57(5432)
Mar 19 11:53:24 hostname pgpool: 2016-03-19 10:53:24 LOG: pid 11384: worker process received restart request
Mar 19 11:53:25 hostname pgpool: 2016-03-19 10:53:25 LOG: pid 11383: pcp child process received restart request
Mar 19 11:53:25 hostname pgpool: 2016-03-19 10:53:25 LOG: pid 9152: PCP child 11383 exits with status 256 in failover()
Mar 19 11:53:25 hostname pgpool: 2016-03-19 10:53:25 LOG: pid 9152: fork a new PCP child pid 13073 in failover()
Mar 19 11:53:25 hostname pgpool: 2016-03-19 10:53:25 LOG: pid 9152: worker child 11384 exits with status 256
Mar 19 11:53:25 hostname pgpool: 2016-03-19 10:53:25 LOG: pid 9152: fork a new worker child pid 13074

#############################################################

# pcp show
0: 192.168.181.230 5432 2 0.000000
1: 192.168.181.57 5432 1 1.000000

# psql -h localhost -p 9999 mydb myuser
psql: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

# Node 1 never reaches state 2.

# sudo service pgpool2 restart

#############################################################

Mar 19 11:56:32 hostname pgpool: 2016-03-19 10:56:32 LOG: pid 9152: received fast shutdown request
Mar 19 11:56:32 hostname pgpool: 2016-03-19 10:56:32 LOG: pid 9152: pgpool main: close listen socket
Mar 19 11:56:33 hostname pgpool: 2016-03-19 10:56:33 LOG: pid 16547: pgpool-II successfully started. version 3.3.7 (tokakiboshi)
Mar 19 11:56:33 hostname pgpool: 2016-03-19 10:56:33 LOG: pid 16547: find_primary_node: primary node id is 0

#############################################################

# pcp show
0: 192.168.181.230 5432 2 0.000000
1: 192.168.181.57 5432 2 1.000000

# psql -h localhost -p 9999 mydb myuser
Password for user myuser:

# All good after restart of pgpool
Additional Information# ----------------------------
# pgPool-II configuration file
# ----------------------------
#
# This file consists of lines of the form:
#
# name = value
#
# Whitespace may be used. Comments are introduced with "#" anywhere on a line.
# The complete list of parameter names and allowed values can be found in the
# pgPool-II documentation.
#
# This file is read on server startup and when the server receives a SIGHUP
# signal. If you edit the file on a running system, you have to SIGHUP the
# server for the changes to take effect, or use "pgpool reload". Some
# parameters, which are marked below, require a server shutdown and restart to
# take effect.
#


#------------------------------------------------------------------------------
# CONNECTIONS
#------------------------------------------------------------------------------

# - pgpool Connection Settings -

listen_addresses = '*'
                                   # Host name or IP address to listen on:
                                   # '*' for all, '' for no TCP/IP connections
                                   # (change requires restart)
port = 9999 # Port number
                                   # (change requires restart)
socket_dir = '/var/run/postgresql'
                                   # Unix domain socket path
                                   # The Debian package defaults to
                                   # /var/run/postgresql
                                   # (change requires restart)


# - pgpool Communication Manager Connection Settings -

pcp_port = 9898 # Port number for pcp
                                   # (change requires restart)
pcp_socket_dir = '/var/run/postgresql'
                                   # Unix domain socket path for pcp
                                   # The Debian package defaults to
                                   # /var/run/postgresql
                                   # (change requires restart)

# - Backend Connection Settings -


backend_hostname0 = '192.168.181.230'
backend_port0 = 5432
backend_weight0 = 0
#backend_data_directory0 = '/data/postgresql/9.3/main'
backend_hostname1 = '192.168.181.57'
backend_port1 = 5432
backend_weight1 = 5
#backend_data_directory1 = '/data/postgresql/9.3/main'


# - Authentication -

enable_pool_hba = on
                                   # Use pool_hba.conf for client authentication
pool_passwd = ''
                                   # File name of pool_passwd for md5 authentication.
                                   # "" disables pool_passwd.
                                   # (change requires restart)
authentication_timeout = 10
                                   # Delay in seconds to complete client authentication
                                   # 0 means no timeout.

# - SSL Connections -


ssl = on

                                   # Enable SSL support
                                   # (change requires restart)
ssl_key = '/etc/pgpool2/server.key'
                                   # Path to the SSL private key file
                                   # (change requires restart)
ssl_cert = '/etc/pgpool2/server.crt'
                                   # Path to the SSL public certificate file
                                   # (change requires restart)
#ssl_ca_cert = ''
                                   # Path to a single PEM format file
                                   # containing CA root certificate(s)
                                   # (change requires restart)
#ssl_ca_cert_dir = ''
                                   # Directory containing CA root certificate(s)
                                   # (change requires restart)


#------------------------------------------------------------------------------
# POOLS
#------------------------------------------------------------------------------

# - Pool size -

num_init_children = 10 # Number of pools
                                   # (change requires restart)
max_pool = 1
                                   # Number of connections per pool
                                   # (change requires restart)

# - Life time -


child_life_time = 300 # If a child is _IDLE_ for that many seconds _AFTER_ the first connect, kill it

child_max_connections = 1000
                                   # Pool exits after receiving that many connections
                                   # 0 means no exit
connection_life_time = 0
                                   # Connection to backend closes after being idle for this many seconds
                                   # 0 means no close
client_idle_limit = 600 # Client is disconnected after being idle for that many seconds
                                   # (even inside an explicit transactions!)
                                   # 0 means no disconnection


#------------------------------------------------------------------------------
# LOGS
#------------------------------------------------------------------------------

# - Where to log -

log_destination = 'stderr'
                                   # Where to log
                                   # Valid values are combinations of stderr,
                                   # and syslog. Default to stderr.

# - What to log -

print_timestamp = on
                                   # Print timestamp on each line
                                   # (change requires restart)

log_connections = off
                                   # Log connections
log_hostname = off
                                   # Hostname will be shown in ps status
                                   # and in logs if connections are logged
log_statement = off
                                   # Log all statements
log_per_node_statement = off
                                   # Log all statements
                                   # with node and backend informations
log_standby_delay = 'if_over_threshold'
                                   # Log standby delay
                                   # Valid values are combinations of always,
                                   # if_over_threshold, none

# - Syslog specific -

syslog_facility = 'LOCAL0'
                                   # Syslog local facility. Default to LOCAL0
syslog_ident = 'pgpool'
                                   # Syslog program identification string
                                   # Default to 'pgpool'

# - Debug -

debug_level = 0
                                   # Debug message verbosity level
                                   # 0 means no message, 1 or more mean verbose


#------------------------------------------------------------------------------
# FILE LOCATIONS
#------------------------------------------------------------------------------

pid_file_name = '/var/run/postgresql/pgpool2.pid'
                                   # PID file name
                                   # (change requires restart)
logdir = '/var/log/postgresql'
                                   # Directory of pgPool status file
                                   # (change requires restart)


#------------------------------------------------------------------------------
# CONNECTION POOLING
#------------------------------------------------------------------------------

connection_cache = on
                                   # Activate connection pools
                                   # (change requires restart)

                                   # Semicolon separated list of queries
                                   # to be issued at the end of a session
                                   # The default is for 8.3 and later
reset_query_list = 'ABORT; DISCARD ALL'
# Exploding DISCARD ALL to its single commands according to http://www.postgresql.org/docs/9.3/static/sql-discard.html
# Goal is to see at which command DISCARD ALL hangs
#reset_query_list = 'ABORT; SET SESSION AUTHORIZATION DEFAULT; RESET ALL; DEALLOCATE ALL; CLOSE ALL; UNLISTEN *; SELECT pg_advisory_unlock_all(); DISCARD PLANS; DISCARD TEMP;'
                                   # The following one is for 8.2 and before
#reset_query_list = 'ABORT; RESET ALL; SET SESSION AUTHORIZATION DEFAULT'


#------------------------------------------------------------------------------
# REPLICATION MODE
#------------------------------------------------------------------------------

replication_mode = off
                                   # Activate replication mode
                                   # (change requires restart)
replicate_select = off
                                   # Replicate SELECT statements
                                   # when in replication or parallel mode
                                   # replicate_select is higher priority than
                                   # load_balance_mode.

insert_lock = on
                                   # Automatically locks a dummy row or a table
                                   # with INSERT statements to keep SERIAL data
                                   # consistency
                                   # Without SERIAL, no lock will be issued
lobj_lock_table = ''
                                   # When rewriting lo_creat command in
                                   # replication mode, specify table name to
                                   # lock

# - Degenerate handling -

replication_stop_on_mismatch = off
                                   # On disagreement with the packet kind
                                   # sent from backend, degenerate the node
                                   # which is most likely "minority"
                                   # If off, just force to exit this session

failover_if_affected_tuples_mismatch = off
                                   # On disagreement with the number of affected
                                   # tuples in UPDATE/DELETE queries, then
                                   # degenerate the node which is most likely
                                   # "minority".
                                   # If off, just abort the transaction to
                                   # keep the consistency


#------------------------------------------------------------------------------
# LOAD BALANCING MODE
#------------------------------------------------------------------------------

load_balance_mode = on
                                   # Activate load balancing mode
                                   # (change requires restart)
ignore_leading_white_space = on
                                   # Ignore leading white spaces of each query
white_function_list = ''
                                   # Comma separated list of function names
                                   # that don't write to database
                                   # Regexp are accepted
black_function_list = 'nextval,setval'
                                   # Comma separated list of function names
                                   # that write to database
                                   # Regexp are accepted


#------------------------------------------------------------------------------
# MASTER/SLAVE MODE
#------------------------------------------------------------------------------

master_slave_mode = on
                                   # Activate master/slave mode
                                   # (change requires restart)
master_slave_sub_mode = 'stream'
                                   # Master/slave sub mode
                                   # Valid values are combinations slony or
                                   # stream. Default is slony.
                                   # (change requires restart)

# - Streaming -

sr_check_period = 10
                                   # Streaming replication check period
                                   # Disabled (0) by default
sr_check_user = 'healthcheck'
                                   # Streaming replication check user
                                   # This is necessary even if you disable
                                   # streaming replication delay check with
                                   # sr_check_period = 0
sr_check_password = 'Noo9oph0oova'
                                   # Password for streaming replication check user
delay_threshold = 32000000
                                   # Threshold before not dispatching query to standby node
                                   # Unit is in bytes
                                   # Disabled (0) by default

# - Special commands -

follow_master_command = ''
                                   # Executes this command after master failover
                                   # Special values:
                                   # %d = node id
                                   # %h = host name
                                   # %p = port number
                                   # %D = database cluster path
                                   # %m = new master node id
                                   # %H = hostname of the new master node
                                   # %M = old master node id
                                   # %P = old primary node id
                                   # %r = new master port number
                                   # %R = new master database cluster path
                                   # %% = '%' character


#------------------------------------------------------------------------------
# PARALLEL MODE
#------------------------------------------------------------------------------

parallel_mode = off
                                   # Activates parallel query mode
                                   # (change requires restart)
pgpool2_hostname = ''
                                   # Set pgpool2 hostname
                                   # (change requires restart)

# - System DB info -

#system_db_hostname = 'localhost'
                                   # (change requires restart)
#system_db_port = 5432
                                   # (change requires restart)
#system_db_dbname = 'pgpool'
                                   # (change requires restart)
#system_db_schema = 'pgpool_catalog'
                                   # (change requires restart)
#system_db_user = 'pgpool'
                                   # (change requires restart)
#system_db_password = ''
                                   # (change requires restart)


#------------------------------------------------------------------------------
# HEALTH CHECK
#------------------------------------------------------------------------------

health_check_period = 40
                                   # Health check period
                                   # Disabled (0) by default
health_check_timeout = 10
                                   # Health check timeout
                                   # 0 means no timeout
health_check_user = 'healthcheck'
                                   # Health check user
health_check_password = 'Noo9oph0oova'
                                   # Password for health check user
health_check_max_retries = 4
                                   # Maximum number of times to retry a failed health check before giving up.
health_check_retry_delay = 5
                                   # Amount of time to wait (in seconds) between retries.


#------------------------------------------------------------------------------
# FAILOVER AND FAILBACK
#------------------------------------------------------------------------------

failover_command = ''
                                   # Executes this command at failover
                                   # Special values:
                                   # %d = node id
                                   # %h = host name
                                   # %p = port number
                                   # %D = database cluster path
                                   # %m = new master node id
                                   # %H = hostname of the new master node
                                   # %M = old master node id
                                   # %P = old primary node id
                                   # %r = new master port number
                                   # %R = new master database cluster path
                                   # %% = '%' character
failback_command = ''
                                   # Executes this command at failback.
                                   # Special values:
                                   # %d = node id
                                   # %h = host name
                                   # %p = port number
                                   # %D = database cluster path
                                   # %m = new master node id
                                   # %H = hostname of the new master node
                                   # %M = old master node id
                                   # %P = old primary node id
                                   # %r = new master port number
                                   # %R = new master database cluster path
                                   # %% = '%' character

fail_over_on_backend_error = off
                                   # Initiates failover when reading/writing to the
                                   # backend communication socket fails
                                   # If set to off, pgpool will report an
                                   # error and disconnect the session.

search_primary_node_timeout = 120
                                   # Timeout in seconds to search for the
                                   # primary node when a failover occurs.
                                   # 0 means no timeout, keep searching
                                   # for a primary node forever.

#------------------------------------------------------------------------------
# ONLINE RECOVERY
#------------------------------------------------------------------------------

#recovery_user = 'nobody'
                                   # Online recovery user
#recovery_password = ''
                                   # Online recovery password
#recovery_1st_stage_command = ''
                                   # Executes a command in first stage
#recovery_2nd_stage_command = ''
                                   # Executes a command in second stage
#recovery_timeout = 90
                                   # Timeout in seconds to wait for the
                                   # recovering node's postmaster to start up
                                   # 0 means no wait
client_idle_limit_in_recovery = 0
                                   # Client is disconnected after being idle
                                   # for that many seconds in the second stage
                                   # of online recovery
                                   # 0 means no disconnection
                                   # -1 means immediate disconnection


#------------------------------------------------------------------------------
# WATCHDOG
#------------------------------------------------------------------------------

# - Enabling -

use_watchdog = off # Activates watchdog
                                    # (change requires restart)

# -Connection to up stream servers -

trusted_servers = ''
                                    # trusted server list which are used
                                    # to confirm network connection
                                    # (hostA,hostB,hostC,...)
                                    # (change requires restart)
ping_path = '/bin'
                                    # ping command path
                                    # (change requires restart)

# - Watchdog communication Settings -

wd_hostname = '192.168.181.98'
                                    # Host name or IP address of this watchdog
                                    # (change requires restart)
wd_port = 9000 # port number for watchdog service
                                    # (change requires restart)
wd_authkey = ''
                                    # Authentication key for watchdog communication
                                    # (change requires restart)

# - Virtual IP control Setting -

delegate_IP = '127.0.0.1'
                                    # delegate IP address
                                    # If this is empty, virtual IP never bring up.
                                    # (change requires restart)
ifconfig_path = '/etc/pgpool2'
                                    # ifconfig command path
                                    # (change requires restart)
if_up_cmd = 'pgpool_ifconfig eth1:pgpool2 inet $_IP_$ netmask 255.255.0.0'
                                    # startup delegate IP command
                                    # (change requires restart)
if_down_cmd = 'pgpool_ifconfig eth1:pgpool2 down'
                                    # shutdown delegate IP command
                                    # (change requires restart)

arping_path = '/usr/bin' # arping command path
                                    # (change requires restart)

arping_cmd = 'arping -U -I eth1 -c 10 $_IP_$'
                                    # arping command
                                    # (change requires restart)

# - Behaivor on escalation Setting -

clear_memqcache_on_escalation = on
                                    # Clear all the query cache on shared memory
                                    # when standby pgpool escalate to active pgpool
                                    # (= virtual IP holder).
                                    # This should be off if client connects to pgpool
                                    # not using virtual IP.
                                    # (change requires restart)
wd_escalation_command = ''
                                    # Executes this command at escalation on new active pgpool.
                                    # (change requires restart)

# - Lifecheck Setting -

# -- common --

wd_lifecheck_method = 'heartbeat'
                                    # Method of watchdog lifecheck ('heartbeat' or 'query')
                                    # (change requires restart)
wd_interval = 10
                                    # lifecheck interval (sec) > 0
                                    # (change requires restart)

# -- heartbeat mode --

wd_heartbeat_port = 9694 # Port number for receiving heartbeat signal
                                    # (change requires restart)
wd_heartbeat_keepalive = 2
                                    # Interval time of sending heartbeat signal (sec)
                                    # (change requires restart)
wd_heartbeat_deadtime = 20
                                    # Deadtime interval for heartbeat signal (sec)
                                    # (change requires restart)



#heartbeat_destination0 = ''
                                    # Host name or IP address of destination 0
                                    # for sending heartbeat signal.
                                    # (change requires restart)
#heartbeat_destination_port0 = 9694 # Port number of destination 0 for sending
                                    # heartbeat signal. Usually this is the
                                    # same as wd_heartbeat_port.
                                    # (change requires restart)
#heartbeat_device0 = 'eth1'
                                    # Name of NIC device (such like 'eth0')
                                    # used for sending/receiving heartbeat
                                    # signal to/from destination 0.
                                    # This works only when this is not empty
                                    # and pgpool has root privilege.
                                    # (change requires restart)

#heartbeat_destination1 = 'node1'
#heartbeat_destination_port1 = 9694#heartbeat_device1 = 'eth1'

# -- query mode --

wd_life_point = 6
                                    # lifecheck retry times
                                    # (change requires restart)
wd_lifecheck_query = 'SELECT 1'
                                    # lifecheck query to pgpool from watchdog
                                    # (change requires restart)
wd_lifecheck_dbname = 'postgres'
                                    # Database name connected for lifecheck
                                    # (change requires restart)
wd_lifecheck_user = 'healthcheck'
                                    # watchdog user monitoring pgpools in lifecheck
                                    # (change requires restart)
wd_lifecheck_password = 'Noo9oph0oova'
                                    # Password for watchdog user in lifecheck
                                    # (change requires restart)

# - Other pgpool Connection Settings -




#other_pgpool_hostname0 = ''
                                    # Host name or IP address to connect to for other pgpool 0
                                    # (change requires restart)
#other_pgpool_port0 = 5432
                                    # Port number for othet pgpool 0
                                    # (change requires restart)
#other_wd_port0 = 9000 # Port number for othet watchdog 0
                                    # (change requires restart)
#other_pgpool_hostname1 = 'host1'
#other_pgpool_port1 = 5432
#other_wd_port1 = 9000

#------------------------------------------------------------------------------
# OTHERS
#------------------------------------------------------------------------------
relcache_expire = 60
                                   # Life time of relation cache in seconds.
                                   # 0 means no cache expiration(the default).
                                   # The relation cache is used for cache the
                                   # query result against PostgreSQL system
                                   # catalog to obtain various information
                                   # including table structures or if it's a
                                   # temporary table or not. The cache is
                                   # maintained in a pgpool child local memory
                                   # and being kept as long as it survives.
                                   # If someone modify the table by using
                                   # ALTER TABLE or some such, the relcache is
                                   # not consistent anymore.
                                   # For this purpose, cache_expiration
                                   # controls the life time of the cache.

relcache_size = 512
                                   # Number of relation cache
                                   # entry. If you see frequently:
                                   # "pool_search_relcache: cache replacement happend"
                                   # in the pgpool log, you might want to increate this number.

check_temp_table = off
                                   # If on, enable temporary table check in SELECT statements.
                                   # This initiates queries against system catalog of primary/master
                                   # thus increases load of master.
                                   # If you are absolutely sure that your system never uses temporary tables
                                   # and you want to save access to primary/master, you could turn this off.
                                   # Default is on.


#------------------------------------------------------------------------------
# ON MEMORY QUERY MEMORY CACHE
#------------------------------------------------------------------------------
memory_cache_enabled = off
                                   # If on, use the memory cache functionality, off by default
memqcache_method = 'shmem'
                                   # Cache storage method. either 'shmem'(shared memory) or
                                   # 'memcached'. 'shmem' by default
                                   # (change requires restart)
memqcache_memcached_host = 'localhost'
                                   # Memcached host name or IP address. Mandatory if
                                   # memqcache_method = 'memcached'.
                                   # Defaults to localhost.
                                   # (change requires restart)
memqcache_memcached_port = 11211
                                   # Memcached port number. Mondatory if memqcache_method = 'memcached'.
                                   # Defaults to 11211.
                                   # (change requires restart)
memqcache_total_size = 67108864
                                   # Total memory size in bytes for storing memory cache.
                                   # Mandatory if memqcache_method = 'shmem'.
                                   # Defaults to 64MB.
                                   # (change requires restart)
memqcache_max_num_cache = 1000000
                                   # Total number of cache entries. Mandatory
                                   # if memqcache_method = 'shmem'.
                                   # Each cache entry consumes 48 bytes on shared memory.
                                   # Defaults to 1,000,000(45.8MB).
                                   # (change requires restart)
memqcache_expire = 0
                                   # Memory cache entry life time specified in seconds.
                                   # 0 means infinite life time. 0 by default.
                                   # (change requires restart)
memqcache_auto_cache_invalidation = on
                                   # If on, invalidation of query cache is triggered by corresponding
                                   # DDL/DML/DCL(and memqcache_expire). If off, it is only triggered
                                   # by memqcache_expire. on by default.
                                   # (change requires restart)
memqcache_maxcache = 409600
                                   # Maximum SELECT result size in bytes.
                                   # Must be smaller than memqcache_cache_block_size. Defaults to 400KB.
                                   # (change requires restart)
memqcache_cache_block_size = 1048576
                                   # Cache block size in bytes. Mandatory if memqcache_method = 'shmem'.
                                   # Defaults to 1MB.
                                   # (change requires restart)
memqcache_oiddir = '/var/log/pgpool/oiddir'
                                   # Temporary work directory to record table oids
                                   # (change requires restart)
white_memqcache_table_list = ''
                                   # Comma separated list of table names to memcache
                                   # that don't write to database
                                   # Regexp are accepted
black_memqcache_table_list = ''
                                   # Comma separated list of table names not to memcache
                                   # that don't write to database
                                   # Regexp are accepted
TagsNo tags attached.

Activities

sw

2016-03-19 22:53

reporter   ~0000696

Just tested pgpool 3.4.3. Same issue.

sw

2016-03-19 23:17

reporter   ~0000697

Also tested with postgresql 9.4.5. Whenever I stop the slave, pgpool just stops working.

sw

2016-03-19 23:21

reporter   ~0000698

Maybe worth mentioning: When I start slave again before pgpool sets node status to 3, pgpool is able to recover and it accepts connections again. When node is in state 3, it is impossible to attach the node again and connection to pgpool times out.

sw

2016-03-20 22:40

reporter   ~0000699

Had some more time to look into this issue today. Look like I was a bit impatient while testing yesterday and noted some false positives.

I can reproduce this issue (that it stops working until manual restart of service) with pgpool2 3.3.7 (plus semaphore_accept.patch from another issue) and postgresql 9.4.5 only. Since this is a patched version of pgpool, it is maybe not worth to investigate further.

Using pgpool 3.4.3 (without extra patches) and postgresql 9.4.5, pgpool stops accepting new connections when Postgresql slave is stopped until its node state is DOWN (3). This takes around 1 minute. After that minute, pgpool restarts internally and uses the existing connect to master only as expected. Maybe this is intented behavior but I never noticed a minute long downtime of pgpool when stopping a slave before.

t-ishii

2016-05-06 17:27

developer   ~0000792

I have fixed this from 3.5 to 3.2 stable trees so that pgpool accepts connection from clients while pgpool is doing health checking. Please try if you like.

caudatus

2019-02-15 00:19

reporter   ~0002385

I have the same problem for me. with pgpool-II 3.7.7 or pgpool-II-4.0.2 + postgres9.5.
Setup:
hosts: 3 separate docker containers: single pgpool + pgdb1 (master) + pgdb2 (slave)
- master_slave_mode = on
- master_slave_sub_mode = stream
- load_balance_mode = off
network: same subnet/24
reduced communication:
- health_check_period=0 disabled
- sr_check_period=0 disabled

incident: drop tcp packets between pgpool and pgdb2
event:
- pgpool cannot accept new psql connections (hangs),
- but: pgpool serves already opened psql connections

Issue History

Date Modified Username Field Change
2016-03-19 21:48 sw New Issue
2016-03-19 22:53 sw Note Added: 0000696
2016-03-19 23:17 sw Note Added: 0000697
2016-03-19 23:21 sw Note Added: 0000698
2016-03-20 22:40 sw Note Added: 0000699
2016-04-01 13:07 t-ishii Assigned To => Muhammad Usama
2016-04-01 13:07 t-ishii Status new => assigned
2016-05-06 17:27 t-ishii Note Added: 0000792
2019-02-15 00:19 caudatus Note Added: 0002385