[pgpool-general: 6606] Re: After shutdown or poweroff commands pgpool on slave machine failed to start
Bo Peng
pengbo at sraoss.co.jp
Sun Jun 23 21:35:24 JST 2019
Hello,
On Thu, 20 Jun 2019 13:58:17 +0000
Avi Weinberg <AviW at gilat.com> wrote:
> Hi experts
>
> We are using pgpool version 3.6.9 in master slave steaming replication setup (postgres 9.6.7). When we issue a poweroff or shutdown command on the machine with pgpool and database was master, in many cases pgpool status command from the slave machine will show "connection to host:"172.18.255.11:5432" failed" where 172.18.255.11 is the old master. My search_primary_node_timeout is 10, but even after 3 minutes it still tries the old master IP.
"search_primary_node_timeout" parameter is the maximum amount of
time to search for the primary node when failover occurs.
But the log below shows that failover was not triggered.
Therefore, even though you set "search_primary_node_timeout",
it did not work.
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 11207: LOG: trying connecting to PostgreSQL server on "172.18.255.11:5432" by INET socket
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 11207: DETAIL: timed out. retrying...
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 9520: LOG: failed to connect to PostgreSQL server on "172.18.255.11:5432", timed out
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 9520: ERROR: failed to make persistent db connection
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 9520: DETAIL: connection to host:"172.18.255.11:5432" failed
When you shutdown the server, "connect_timeout" doesn't wrok and
the connection to backend server will retry until TCP/IP timeout.
So, if you want to detach the down backend server and trigger failover,
you need to enable "failover_on_backend_error" parameter or enable health check.
> Your help is most welcome
> Avi
>
>
> service pgpool status
> Redirecting to /bin/systemctl status pgpool.service
> * pgpool.service - Pgpool-II
> Loaded: loaded (/usr/lib/systemd/system/pgpool.service; disabled; vendor preset: disabled)
> Active: active (running) since Tue 2019-05-28 10:14:23 GMT; 26s ago
> Main PID: 9520 (pgpool)
> CGroup: /system.slice/pgpool.service
> |- 9520 /usr/bin/pgpool -f /etc/pgpool-II/pgpool.conf -n
> |- 9537 pgpool: watchdog
> |-11110 pgpool: lifecheck
> |-11111 pgpool: wait for connection request
> |-11113 pgpool: wait for connection request
> |-11114 pgpool: wait for connection request
> |-11116 pgpool: wait for connection request
> |-11118 pgpool: wait for connection request
> |-11119 pgpool: wait for connection request
> |-11120 pgpool: wait for connection request
> |-11121 pgpool: wait for connection request
> |-11122 pgpool: wait for connection request
> |-11123 pgpool: wait for connection request
> |-11124 pgpool: wait for connection request
> |-11126 pgpool: wait for connection request
> |-11127 pgpool: wait for connection request
> |-11128 pgpool: wait for connection request
> |-11129 pgpool: wait for connection request
> |-11131 pgpool: wait for connection request
> |-11132 pgpool: wait for connection request
> |-11133 pgpool: wait for connection request
> |-11134 pgpool: wait for connection request
> |-11135 pgpool: wait for connection request
> |-11138 pgpool: heartbeat receiver
> |-11139 pgpool: heartbeat sender
> |-11140 pgpool: wait for connection request
> |-11141 pgpool: wait for connection request
> |-11142 pgpool: wait for connection request
> |-11143 pgpool: wait for connection request
> |-11144 pgpool: wait for connection request
> |-11145 pgpool: wait for connection request
> |-11146 pgpool: wait for connection request
> |-11148 pgpool: wait for connection request
> |-11149 pgpool: wait for connection request
> |-11150 pgpool: wait for connection request
> |-11151 pgpool: wait for connection request
> |-11152 pgpool: wait for connection request
> |-11153 pgpool: wait for connection request
> |-11155 pgpool: wait for connection request
> |-11156 pgpool: wait for connection request
> |-11157 pgpool: wait for connection request
> |-11158 pgpool: wait for connection request
> |-11159 pgpool: wait for connection request
> |-11160 pgpool: wait for connection request
> |-11161 pgpool: accept connection
> |-11163 pgpool: wait for connection request
> |-11164 pgpool: wait for connection request
> |-11165 pgpool: wait for connection request
> |-11166 pgpool: wait for connection request
> |-11168 pgpool: wait for connection request
> |-11169 pgpool: wait for connection request
> |-11170 pgpool: wait for connection request
> |-11171 pgpool: wait for connection request
> |-11173 pgpool: wait for connection request
> |-11174 pgpool: wait for connection request
> |-11177 pgpool: wait for connection request
> |-11178 pgpool: wait for connection request
> |-11179 pgpool: wait for connection request
> |-11181 pgpool: wait for connection request
> |-11182 pgpool: wait for connection request
> |-11183 pgpool: wait for connection request
> |-11185 pgpool: wait for connection request
> |-11186 pgpool: wait for connection request
> |-11187 pgpool: wait for connection request
> |-11189 pgpool: wait for connection request
> |-11192 pgpool: wait for connection request
> |-11193 pgpool: wait for connection request
> |-11194 pgpool: wait for connection request
> |-11195 pgpool: wait for connection request
> |-11196 pgpool: wait for connection request
> |-11197 pgpool: accept connection
> |-11198 pgpool: wait for connection request
> |-11199 pgpool: wait for connection request
> |-11200 pgpool: wait for connection request
> |-11201 pgpool: wait for connection request
> |-11205 pgpool: PCP: wait for connection request
> `-11207 pgpool: worker process
>
> May 28 10:14:38 h1-nms pgpool[9520]: 2019-05-28 10:14:38: pid 11138: LOG: creating watchdog heartbeat receive socket.
> May 28 10:14:38 h1-nms pgpool[9520]: 2019-05-28 10:14:38: pid 11138: DETAIL: set SO_REUSEPORT
> May 28 10:14:38 h1-nms pgpool[9520]: 2019-05-28 10:14:38: pid 11139: LOG: set SO_REUSEPORT option to the socket
> May 28 10:14:38 h1-nms pgpool[9520]: 2019-05-28 10:14:38: pid 11139: LOG: creating socket for sending heartbeat
> May 28 10:14:38 h1-nms pgpool[9520]: 2019-05-28 10:14:38: pid 11139: DETAIL: set SO_REUSEPORT
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 11207: LOG: trying connecting to PostgreSQL server on "172.18.255.11:5432" by INET socket
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 11207: DETAIL: timed out. retrying...
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 9520: LOG: failed to connect to PostgreSQL server on "172.18.255.11:5432", timed out
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 9520: ERROR: failed to make persistent db connection
> May 28 10:14:47 h1-nms pgpool[9520]: 2019-05-28 10:14:47: pid 9520: DETAIL: connection to host:"172.18.255.11:5432" failed
>
>
> IMPORTANT - This email and any attachments is intended for the above named addressee(s), and may contain information which is confidential or privileged. If you are not the intended recipient, please inform the sender immediately and delete this email: you should not copy or use this e-mail for any purpose nor disclose its contents to any person.
--
Bo Peng <pengbo at sraoss.co.jp>
SRA OSS, Inc. Japan
More information about the pgpool-general
mailing list