[Pgpool-general] pgpool-II failed out my database - why?

[Tatsuo Ishii]

What I can observe from the log is, pgpool tried to send a packet to
backend and failed with "pool_flush_it: write failed (Broken pipe)
offset: 0 wlen: 5". This suggests that the socket connection between
pgpool and PostgreSQL was terminated at this point.

Did you find anything unusual with pgpool or system? For example, some
kernel errors, network errors or pgpool errors (for example, segfault)?
--
Tatsuo Ishii
SRA OSS, Inc. Japan

> Any suggestions for me?   We're about to take this system live, but I
> am uncomfortable
> leaving pgpool-II in if it fails out the database with no apparent
> reason.  Are we using pgpool-II wrong or did we run into a bug or ?
> 
> I've upgraded to v2.2.5 but would like some idea of what went wrong.
> This happened twice in the last 2 months, that pgpool-II failed out
> the single DB back end for no apparent reason.
> 
> How can I get insight into this?
> 
> I tried turning on debug mode but that generated too many messages and
> filled up the disk, so I had to turn it off.
> 
> Thanks very much,
> Aleksey
> 
> 
> On Wed, Oct 14, 2009 at 7:02 PM, Aleksey Tsalolikhin
> <atsaloli.tech at gmail.com> wrote:
> > Hi.  pgpool-II v2.2.4 has failed out my PostgreSQL 8.3.7 database again.
> >
> > I found the following in pgpool log:
> >
> > 2009-10-14 13:16:29 ERROR: pid 2877: pool_read: read failed
> > (Connection reset by peer)
> > 2009-10-14 13:16:29 LOG:   pid 2877: ProcessFrontendResponse: failed
> > to read kind from frontend. frontend abnormally exited
> > 2009-10-14 13:30:47 LOG:   pid 2889: ProcessFrontendResponse: failed
> > to read kind from frontend. frontend abnormally exited
> > 2009-10-14 18:44:36 LOG:   pid 2845: ProcessFrontendResponse: failed
> > to read kind from frontend. frontend abnormally exited
> > 2009-10-14 18:44:36 LOG:   pid 2891: ProcessFrontendResponse: failed
> > to read kind from frontend. frontend abnormally exited
> > 2009-10-14 18:44:36 LOG:   pid 2913: ProcessFrontendResponse: failed
> > to read kind from frontend. frontend abnormally exited
> > 2009-10-14 18:44:36 ERROR: pid 2913: pool_read: read failed
> > (Connection reset by peer)
> > 2009-10-14 18:44:36 LOG:   pid 2892: ProcessFrontendResponse: failed
> > to read kind from frontend. frontend abnormally exited
> > 2009-10-14 18:44:36 LOG:   pid 2913: notice_backend_error: 0 fail over
> > request from pid 2913
> > 2009-10-14 18:44:36 ERROR: pid 2913: pool_flush_it: write failed
> > (Broken pipe) offset: 0 wlen: 5
> > 2009-10-14 18:44:36 LOG:   pid 2842: starting degeneration. shutdown
> > host my-database-server(5432)
> > 2009-10-14 18:44:36 ERROR: pid 2842: failover_handler: no valid DB node found
> > 2009-10-14 18:44:36 LOG:   pid 2891: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:44:36 LOG:   pid 2842: execute command: echo
> > my-database-server 5432 /usr/local/pgsql/data/ 0 >>
> > /var/log/pgpool/failover; echo "pgpool failover
> > event:  my-database-server 5432 /usr/local/pgsql/data/ 0 " | mail -s
> > "pgpool failover event:  my-database-server 5432
> > /usr/local/pgsql/data/ 0 " root,ddcsw
> > 2009-10-14 18:44:37 LOG:   pid 2842: failover_handler: set new master node: 1
> > 2009-10-14 18:44:37 LOG:   pid 22762: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:44:37 LOG:   pid 22763: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:44:37 LOG:   pid 22764: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:44:37 LOG:   pid 22765: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:44:37 LOG:   pid 22766: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:44:37 LOG:   pid 2842: failover done. shutdown host
> > my-database-server(5432)
> >
> > Then I noticed the failure and restarted pgpool (stop and start):
> >
> >
> > 2009-10-14 18:49:09 LOG:   pid 24122: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:49:09 LOG:   pid 24119: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:49:10 LOG:   pid 24123: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:49:11 LOG:   pid 24129: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:49:11 LOG:   pid 24154: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:49:11 LOG:   pid 24124: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:49:11 LOG:   pid 24156: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:49:12 LOG:   pid 24157: Cannot accept() new connection.
> > all backends are down
> > 2009-10-14 18:49:12 LOG:   pid 2842: received smart shutdown request
> > 2009-10-14 18:49:27 LOG:   pid 24185: pgpool successfully started
> >
> >
> > Nothing wrong with the database, when I restarted pgpool, it
> > reconnected OK and has been running fine since.
> >
> > Why did it fail out the database, please?
> >
> > Truly,
> > -at
> >
> _______________________________________________
> Pgpool-general mailing list
> Pgpool-general at pgfoundry.org
> http://pgfoundry.org/mailman/listinfo/pgpool-general