[Pgpool-general] Why did my pgpool-II fail out my PostgreSQL database? and: phantom connections

[Aleksey Tsalolikhin]

Hi.  I am running pgpool-II on a CentOS 5.3 Linux box, talking over
TCP/IP to a PostgreSQL database server 8.3.7 also on CentOS 5.3

I had a failover event, where the pgpool failed out the Postgres DB,
and no further queries through pgpool would work.

I only have one pgpool configured.

I have health check disabled:

    # Health check period.  0 means no health check.
    health_check_period = 0


I would like to know why pgpool failed out the database, because the
database is OK...  How can I tell what happened, please?

I do have failover_command configured in pgpool.conf as :

failover_command = 'echo %h %p %D %d >> /var/log/pgpool/failover; echo
"pgpool failover event:  %h %p %D %d " | mail -s "pgpool failover
event:  %h %p %D %d " root'

but that does not tell me what caused the failover.


Actually, here is the full sequence of events, as it may be relevant:

1. we changed the pg_hba.conf on the postgres database to disallow
'postgres' user to connect from pgpool server

1b. pgpool server was configured with health_check_user='postgres' but
health check period was set to 0.

2. a day later, pgpool failed out the database.  (I would love to know why.)

3. I shut down pgpool, and then tried to shut down the database,
   but that failed because the database said pgpool was connected:


[root at database-server ~]#  netstat -anp|grep 5432|grep ESTAB
tcp        0      0 10.10.1.112:5432           10.10.1.114:42676
   ESTABLISHED 3145/postgres: keym
tcp        0      0 10.10.1.112:5432           10.10.1.114:42683
   ESTABLISHED 3152/postgres: keym
tcp        0      0 10.10.1.112:5432           10.10.1.114:42706
   ESTABLISHED 3226/postgres: nagi
[root at database-server ~]#



[root at pgpool-server ~]# netstat -anp | grep 42676
[root at pgpool-server ~]# lsof -i :42676
[root at pgpool-server ~]#


Could anybody enlighten me why did postgres think pgpool was connected to it?

(I waited a couple of minutes but the sessions stayed there.)

And I would especially like to know why pgpool failed out the database.


Best,
Aleksey