[pgpool-general: 9070] Re: Segmentation after switchover

[Emond Papegaaij]

Op wo 3 apr 2024 om 13:36 schreef Tatsuo Ishii <ishii at sraoss.co.jp>:

> > Is there any way I can help you track down the cause of this issue? In
> our
> > test setup, it is fairly easy to reproduce, so I can add additional
> logging
> > if needed. I'm currently rerunning the tests with the fix for the
> > uninitialized value and report back with the results (this does take a
> > couple of hours).
>
> If you could provide a self contained test case, especially using
> pgpool_setup or watchdog_setup, it would be very helpful to me.
>

I was afraid this was going to be your answer :) Unfortunately, I really
don't know what sequence of events and actions triggers the error. All I
know is that our reboot test triggers it. Also, I don't know how to setup
such a test case. The main sequence of events is as follows:

* Startup the cluster using the following steps
  - Start the database on node 1 and make sure it is the primary database
  - Start pgpool on node 1 and wait a few seconds
  - Start pgpool on node 2 and 3 and wait a few seconds
  - Initialize the databases on node 2 and 3, setting them up as standby
  - The result now is: node 1 is primary db and leader, node 2 and 3 are
standby db and standby pgpool watchdog
* Shtudown and reboot node 2
* Wait for node 2 to come back up
* Shutdown and reboot node 3
* Wait for node 3 to come back up
* Restart pgpool on node 1 to force it to drop its leader status and wait a
few seconds
* Shutdown and detach the database on node 1 to trigger a failover and wait
for a new primary to be selected
* Wait 10 more seconds
* Rewind the database on node 1 and instruct it to follow the new primary
* (CRASH ON NODE 2) Wait for the now standby database on node 1 to be fully
synchronized with the new primary
* (CRASH ON NODE 1) Shutdown and reboot node 1
* Wait for node 1 to come back up
* Verify the integrity of the cluster (all databases and pgpool instances
must be up)

With Valgrind, I've been able to determine that the SIGSEGV on node 1 has
the same backtrace as the SIGSEGV on node 3 (the one in
free_persistent_db_connection_memory). I'm not sure about the exact moment
for the crash on node 3, as it happens less frequently and I don't have
reliable logs of a crash at the moment. The crash on node 2
(get_query_result in pool_worker_child.c) is by far the most frequent,
happening about 2 out of 3 times. It seems to happen at the moment when the
database on node 1 is started up again, which triggers a failback. When the
database on node 1 reports "database system is ready to accept read only
connections", only a few ms later, the SIGSEGV happens on node 2.

I dove into the code, and I think I've found the cause of the error. Just
prior to crashing, it reports "find_primary_node:
make_persistent_db_connection_noerror failed on node 0". This must come
from pgpool_main.c:2782. This means that slots[0] is NULL. Then, at
pgpool_main.c:2791 it enters verify_backend_node_status with this slots
array. At lines 2569-2579 it loops over these slots,
calling get_server_version for every slot, including slots[0], which is
NULL. This crashes when get_server_version calls get_query_result, which
tries to dereference slots[0]->con. At pgpool_main.c:2456 there is an
explicit check for NULL, this is missing in the other for loop, but it is
also missing at line 2609.

I've attached another log from node 2, with the latest patch applied. I
still see some uninitialized values. I've enabled origin tracking in
valgrind to get detailed information on the origin of the uninitialized
values.

Best regards,
Emond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20240403/5d77493f/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tkh-pgpool-node2.log
Type: text/x-log
Size: 260976 bytes
Desc: not available
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20240403/5d77493f/attachment-0001.bin>