[pgpool-general: 7736] Re: Recovery after loosing connection to all backends

[Bo Peng]

Hello,

> Hi all,
> 
> During testing of some of the more intense failure scenario's we sometimes
> end up with a cluster where all backends are down according to pgpool. We
> run in a cluster with 3 nodes, all running a pgpool and a postgresql
> instance. In some situations when the pgpool instances lose connection to
> the postgresql backends, they are all marked as down and the cluster never
> recovers. To make things worse, this state is shared between the pgpool
> nodes, so even restarting the pgpool instances one by one does not help.
> Once you enter this state, the only way out is to stop all pgpool instances
> and then start them again, to clear the cluster and have pgpool start with
> a fresh state. Is there any way to trigger pgpool to re-evaluate the status
> of a backend, even when the cluster think it's down?

If pgpool detected the failure of PostgreSQL, pgpool will detach the PostgreSQL from cluster
and mark the status as "DOWN".
After you recover the PostgreSQL, you need to use "pcp_attach_node" command to 
attach the PostgreSQL to cluster.

For example:

  pcp_attach_node -h localhost -U postgres -p 9898 -n 1

[pcp_attach_node]
https://www.pgpool.net/docs/latest/en/html/pcp-attach-node.html

Alternatively, you can turn on "auto_failback" parameter to 
automaticly reattach a "healthy" PostgreSQL.

[auto_failback]
https://www.pgpool.net/docs/latest/en/html/runtime-config-failover.html#GUC-AUTO-FAILBACK


> Best regards,
> Emond


-- 
Bo Peng <pengbo at sraoss.co.jp>
SRA OSS, Inc. Japan
http://www.sraoss.co.jp/