[pgpool-general: 3466] Master restart puts pgpool into state of sending writes to slave(s)

[Keith Fiske]

Working on a pgpool setup for a client last Friday and were at a point
where we had to restart the master database to change some postgresql.conf
settings. We'd done this before without incident, but not with their
application actually using pgpool and writing to the DB. This time it was,
but wasn't doing anything of importance that a failed write would matter
for the duration of a master restart.

However, when the master came back up, suddenly the pgool nodes (which were
running on the application servers) were sending writes to the slave. These
were just plain INSERT & UPDATE statements, not inside of a function or
CTE. Restarting pgpool on some of the app nodes seemed to fix it, but two
of the nodes that were much busier would not seem to reset even after a
restart. I tried just a normal "-m fast" restart of the slave, but that
didn't fix it. I actually had to shut the slave down for 10-15 seconds and
suddenly the pgpool nodes realized those connections were no longer valid
and reconnected to the master.

We tried to reproduced this again with another master restart, but it
didn't happen. So, seems to be some sort of edge case with a busy pgpool
node and restarting the master system. Wasn't able to retry again on this
system since they had to get it working. I'll see if I can get some test
systems up and running to try and reproduce it, but I figured I'd send an
email in to at least see if someone else has run into this or can look at
the parts of the code that handle this stuff. If you need anymore info, let
me know.

Thanks,
--
Keith Fiske
Database Administrator
OmniTI Computer Consulting, Inc.
http://www.keithf4.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20150209/fa1a1bd0/attachment.html>