[pgpool-hackers: 4023] invalid degenerate backend request on slave failure

[anirudh.goel at cern.ch]

Hello

I have a setup with 3 postgres nodes running behind one pgpool docker container.

The setup works fine when the primary node fails/is shutdown. Failover goes fine in that case.

However, if the standby goes down, the health check fails but instead of performing a failover, pgpool throws this error message-

2021-09-23 08:37:54: pid 94: LOG: invalid degenerate backend request, node id : 2 status: [2] is not valid for failover

What’s a bit strange to me is that this only happens when I am running the pgpool container through Nomad.

If I run it directly without Nomad, it still works as expected. Even though this is the only difference that I see between the working and non-working setup, I believe this isn’t the root cause.

If you can point me towards when a degenerate request is considered invalid, it might help.

Below are the logs for working and non-working setups.



Non-working Setup Logs:
2021-09-23 08:37:16: pid 414: DETAIL: postmaster on DB node 2 was shutdown by administrative command
2021-09-23 08:37:24: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:24: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:24: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:24: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:24: pid 94: LOG: health check retrying on DB node: 2 (round:1)
2021-09-23 08:37:26: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:26: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:26: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:26: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:26: pid 94: LOG: health check retrying on DB node: 2 (round:2)
2021-09-23 08:37:28: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:28: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:28: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:28: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:28: pid 94: LOG: health check retrying on DB node: 2 (round:3)
2021-09-23 08:37:30: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:30: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:30: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:30: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:30: pid 94: LOG: health check retrying on DB node: 2 (round:4)
2021-09-23 08:37:32: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:32: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:32: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:32: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:32: pid 94: LOG: health check retrying on DB node: 2 (round:5)
2021-09-23 08:37:33: pid 342: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:33: pid 342: DETAIL: Operation now in progress
2021-09-23 08:37:33: pid 342: ERROR: failed to make persistent db connection
2021-09-23 08:37:33: pid 342: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:34: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:34: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:34: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:34: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:34: pid 94: LOG: health check failed on node 2 (timeout:0)
2021-09-23 08:37:34: pid 94: LOG: invalid degenerate backend request, node id : 2 status: [2] is not valid for failover
2021-09-23 08:37:44: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:44: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:44: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:44: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:44: pid 94: LOG: health check retrying on DB node: 2 (round:1)
2021-09-23 08:37:45: pid 414: LOG: Query message from frontend.
2021-09-23 08:37:45: pid 414: DETAIL: query: "show pool_nodes;"
2021-09-23 08:37:46: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:46: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:46: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:46: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:46: pid 94: LOG: health check retrying on DB node: 2 (round:2)
2021-09-23 08:37:48: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:48: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:48: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:48: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:48: pid 94: LOG: health check retrying on DB node: 2 (round:3)
2021-09-23 08:37:50: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:50: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:50: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:50: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:50: pid 94: LOG: health check retrying on DB node: 2 (round:4)
2021-09-23 08:37:52: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:52: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:52: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:52: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:52: pid 94: LOG: health check retrying on DB node: 2 (round:5)
2021-09-23 08:37:53: pid 342: ERROR: Failed to check replication time lag
2021-09-23 08:37:53: pid 342: DETAIL: No persistent db connection for the node 2
2021-09-23 08:37:53: pid 342: HINT: check sr_check_user and sr_check_password
2021-09-23 08:37:53: pid 342: CONTEXT: while checking replication time lag
2021-09-23 08:37:53: pid 342: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:53: pid 342: DETAIL: Operation now in progress
2021-09-23 08:37:53: pid 342: ERROR: failed to make persistent db connection
2021-09-23 08:37:53: pid 342: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:54: pid 94: LOG: failed to connect to PostgreSQL server on "dbod-ag-test3.ch:6603", getsockopt() failed
2021-09-23 08:37:54: pid 94: DETAIL: Operation now in progress
2021-09-23 08:37:54: pid 94: ERROR: failed to make persistent db connection
2021-09-23 08:37:54: pid 94: DETAIL: connection to host:"dbod-ag-test3.ch:6603" failed
2021-09-23 08:37:54: pid 94: LOG: health check failed on node 2 (timeout:0)
2021-09-23 08:37:54: pid 94: LOG: invalid degenerate backend request, node id : 2 status: [2] is not valid for failover


Working Setup Logs:
2021-09-23 09:45:21: pid 78: DETAIL: postmaster on DB node 2 was shutdown by administrative command
2021-09-23 09:45:21: pid 78: LOG: received degenerate backend request for node_id: 2 from pid [78]
2021-09-23 09:45:21: pid 10: LOG: Pgpool-II parent process has received failover request
2021-09-23 09:45:21: pid 10: LOG: starting degeneration. shutdown host dbod-ag-test3.ch(6603)

Thanks!

Anirudh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-hackers/attachments/20210923/d82ff779/attachment.htm>