[Pgpool-general] pgpool-II failed out my database - why?

[Aleksey Tsalolikhin]

Any suggestions for me?   We're about to take this system live, but I
am uncomfortable
leaving pgpool-II in if it fails out the database with no apparent
reason.  Are we using pgpool-II wrong or did we run into a bug or ?

I've upgraded to v2.2.5 but would like some idea of what went wrong.
This happened twice in the last 2 months, that pgpool-II failed out
the single DB back end for no apparent reason.

How can I get insight into this?

I tried turning on debug mode but that generated too many messages and
filled up the disk, so I had to turn it off.

Thanks very much,
Aleksey


On Wed, Oct 14, 2009 at 7:02 PM, Aleksey Tsalolikhin
<atsaloli.tech at gmail.com> wrote:
> Hi.  pgpool-II v2.2.4 has failed out my PostgreSQL 8.3.7 database again.
>
> I found the following in pgpool log:
>
> 2009-10-14 13:16:29 ERROR: pid 2877: pool_read: read failed
> (Connection reset by peer)
> 2009-10-14 13:16:29 LOG:   pid 2877: ProcessFrontendResponse: failed
> to read kind from frontend. frontend abnormally exited
> 2009-10-14 13:30:47 LOG:   pid 2889: ProcessFrontendResponse: failed
> to read kind from frontend. frontend abnormally exited
> 2009-10-14 18:44:36 LOG:   pid 2845: ProcessFrontendResponse: failed
> to read kind from frontend. frontend abnormally exited
> 2009-10-14 18:44:36 LOG:   pid 2891: ProcessFrontendResponse: failed
> to read kind from frontend. frontend abnormally exited
> 2009-10-14 18:44:36 LOG:   pid 2913: ProcessFrontendResponse: failed
> to read kind from frontend. frontend abnormally exited
> 2009-10-14 18:44:36 ERROR: pid 2913: pool_read: read failed
> (Connection reset by peer)
> 2009-10-14 18:44:36 LOG:   pid 2892: ProcessFrontendResponse: failed
> to read kind from frontend. frontend abnormally exited
> 2009-10-14 18:44:36 LOG:   pid 2913: notice_backend_error: 0 fail over
> request from pid 2913
> 2009-10-14 18:44:36 ERROR: pid 2913: pool_flush_it: write failed
> (Broken pipe) offset: 0 wlen: 5
> 2009-10-14 18:44:36 LOG:   pid 2842: starting degeneration. shutdown
> host my-database-server(5432)
> 2009-10-14 18:44:36 ERROR: pid 2842: failover_handler: no valid DB node found
> 2009-10-14 18:44:36 LOG:   pid 2891: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:44:36 LOG:   pid 2842: execute command: echo
> my-database-server 5432 /usr/local/pgsql/data/ 0 >>
> /var/log/pgpool/failover; echo "pgpool failover
> event:  my-database-server 5432 /usr/local/pgsql/data/ 0 " | mail -s
> "pgpool failover event:  my-database-server 5432
> /usr/local/pgsql/data/ 0 " root,ddcsw
> 2009-10-14 18:44:37 LOG:   pid 2842: failover_handler: set new master node: 1
> 2009-10-14 18:44:37 LOG:   pid 22762: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:44:37 LOG:   pid 22763: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:44:37 LOG:   pid 22764: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:44:37 LOG:   pid 22765: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:44:37 LOG:   pid 22766: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:44:37 LOG:   pid 2842: failover done. shutdown host
> my-database-server(5432)
>
> Then I noticed the failure and restarted pgpool (stop and start):
>
>
> 2009-10-14 18:49:09 LOG:   pid 24122: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:49:09 LOG:   pid 24119: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:49:10 LOG:   pid 24123: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:49:11 LOG:   pid 24129: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:49:11 LOG:   pid 24154: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:49:11 LOG:   pid 24124: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:49:11 LOG:   pid 24156: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:49:12 LOG:   pid 24157: Cannot accept() new connection.
> all backends are down
> 2009-10-14 18:49:12 LOG:   pid 2842: received smart shutdown request
> 2009-10-14 18:49:27 LOG:   pid 24185: pgpool successfully started
>
>
> Nothing wrong with the database, when I restarted pgpool, it
> reconnected OK and has been running fine since.
>
> Why did it fail out the database, please?
>
> Truly,
> -at
>