[pgpool-general: 7199] Blocked queries upon pgpool smart shutdown

[Michal Jemala]

Hi guys,

I deployed Bitnami's pgPool
<https://github.com/bitnami/bitnami-docker-pgpool> as a Kubernetes
container sidecar to proxy database queries from our API server to a
Postgres DB. The DB is run and managed by Google
<https://cloud.google.com/sql/docs/postgres>and has a master and 1 replica.
The primary aim of this setup is to load-balance SQL statements between the
Postgres master and its replica and the config pretty much follows pgpool's
Aurora guide
<https://www.pgpool.net/docs/latest/en/html/example-aurora.html>.

Upon updating the API server deployment, the pod gets terminated by
Kubernetes by sending a SIGTERM signal. Thus our API server and pgpool
processes running in separate containers within a single pod, will receive
SIGTERM at the same time. API server handles SIGTERM and starts
a graceful shutdown and so does pgpool. The pgpool's smart shutdown ensures
child processes end gracefully. i.e. it waits for all clients to finish up
and disconnect. However, I still see errors in my API server logs
indicating the connection was closed by the pgpool.

Upon investigation it seems pgpool was not able to gracefully shutdown
within a defined period (30s) and it was SIGKILLed by Kubernetes. API
server logs show few hanging (> 20s) requests doing coupole of DB read
queries. Usually these requests finish within a few ms. pgpool logs show me
nothing (using default log config). Do you have any suggestion on what
could be going on and how to best troubleshoot this? Could I get detailed
logs from pgpool to follow the smart shutdown sequence? Or is there maybe
pgpool misconfiguration which prevents the correct and quick shutdown.

Many thanks,
-Michal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200813/62f388a3/attachment.html>