[pgpool-general: 2835] Issue with pgpool health check timeout

[Shyalika Benthotage]

Hi,

We have configured pgpool 3.3 for our PostgreSQL db in aws. And using the healthcheck parameters as below.

health_check_period = 40
health_check_timeout = 10
health_check_max_retries = 2
health_check_retry_delay = 2

I am trying to test this by using an iptables rule in the active server. When I add a drop rule to the PostgreSQL db port it does trigger a timeout but I think it timeout too soon. I.e. according to my understanding it should wait for the time period specified in "health_check_timeout " in this case, but actually times out in less than a second.(Please see the log below). Shouldn't it wait for 10 secs before time out or is my understanding incorrect. Please note that I checked my DROP rule with telnet and it takes 60 seconds to time out.
I have restarted pgpool every time I tested this (I know just reloading would do)

I have enabled debug mode just to check this.

2014-05-13 12:14:43 DEBUG: pid 15822: health_check: 0 th DB node status: 1
2014-05-13 12:14:44 LOG: pid 15857: connect_inet_domain_socket: select() timed out. retrying...
2014-05-13 12:14:44 ERROR: pid 15822: connect_inet_domain_socket: select() timed out
2014-05-13 12:14:44 ERROR: pid 15822: make_persistent_db_connection: connection to 10.0.0.5(5432) failed
2014-05-13 12:14:44 DEBUG: pid 15822: health_check: 0 th DB node status: 1
2014-05-13 12:14:45 LOG: pid 15857: connect_inet_domain_socket: select() timed out. retrying...
2014-05-13 12:14:45 ERROR: pid 15822: connect_inet_domain_socket: select() timed out
2014-05-13 12:14:45 ERROR: pid 15822: make_persistent_db_connection: connection to 10.0.0.5(5432) failed
2014-05-13 12:14:45 ERROR: pid 15822: health check failed. 0 th host 10.0.0.5 at port 5432 is down
2014-05-13 12:14:45 DEBUG: pid 15822: health check: clearing alarm
2014-05-13 12:14:45 DEBUG: pid 15822: health check: clearing alarm
2014-05-13 12:14:45 LOG: pid 15822: health check retry sleep time: 2 second(s)
2014-05-13 12:14:46 LOG: pid 15857: connect_inet_domain_socket: select() timed out. retrying...
2014-05-13 12:14:47 LOG: pid 15857: connect_inet_domain_socket: select() timed out. retrying...
2014-05-13 12:14:47 DEBUG: pid 15822: retrying 1 th health checking
2014-05-13 12:14:47 DEBUG: pid 15822: health check: clearing alarm
2014-05-13 12:14:47 DEBUG: pid 15822: health_check: 0 th DB node status: 1
2014-05-13 12:14:48 LOG: pid 15857: connect_inet_domain_socket: select() timed out. retrying...
2014-05-13 12:14:48 ERROR: pid 15822: connect_inet_domain_socket: select() timed out
2014-05-13 12:14:48 ERROR: pid 15822: make_persistent_db_connection: connection to 10.0.0.5(5432) failed
2014-05-13 12:14:48 ERROR: pid 15822: health check failed. 0 th host 10.0.0.5 at port 5432 is down
2014-05-13 12:14:48 DEBUG: pid 15822: health check: clearing alarm
2014-05-13 12:14:48 DEBUG: pid 15822: health check: clearing alarm
2014-05-13 12:14:48 LOG: pid 15822: health check retry sleep time: 2 second(s)
2014-05-13 12:14:49 LOG: pid 15857: connect_inet_domain_socket: select() timed out. retrying...
2014-05-13 12:14:50 LOG: pid 15857: connect_inet_domain_socket: select() timed out. retrying...
2014-05-13 12:14:50 DEBUG: pid 15822: retrying 2 th health checking
2014-05-13 12:14:50 DEBUG: pid 15822: health check: clearing alarm
2014-05-13 12:14:50 DEBUG: pid 15822: health_check: 0 th DB node status: 1
2014-05-13 12:14:51 LOG: pid 15857: connect_inet_domain_socket: select() timed out. retrying...
2014-05-13 12:14:51 ERROR: pid 15822: connect_inet_domain_socket: select() timed out
2014-05-13 12:14:51 ERROR: pid 15822: make_persistent_db_connection: connection to 10.0.0.5(5432) failed
2014-05-13 12:14:51 ERROR: pid 15822: health check failed. 0 th host 10.0.0.5 at port 5432 is down
2014-05-13 12:14:51 DEBUG: pid 15822: health check: clearing alarm
2014-05-13 12:14:51 DEBUG: pid 15822: health check: clearing alarm
2014-05-13 12:14:51 LOG: pid 15822: set 0 th backend down status
2014-05-13 12:14:51 DEBUG: pid 15822: failover_handler called
2014-05-13 12:14:51 DEBUG: pid 15822: failover_handler: starting to select new master node

I even tried setting health_check_timeout to 0 to see whether it defaults to TCP/IP timeout. No difference.

Regards,
Shyalika

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20140515/0b46a724/attachment-0001.html>