[Pgpool-general] "failed to read kind from frontend. frontend abnormally exited"?

[Tomasz Chmielewski]

I just saw that one of my nodes has status 1, which means, it never received connections.

I inspected the logs and saw:


Dec 18 12:05:21 pg1 pgpool: 2009-12-18 12:05:21 LOG:   pid 4233: connection received: host=[local]
Dec 18 12:05:27 pg1 pgpool: 2009-12-18 12:05:27 LOG:   pid 4233: connection received: host=[local]
Dec 18 12:05:33 pg1 pgpool: 2009-12-18 12:05:33 LOG:   pid 4233: ProcessFrontendResponse: failed to read kind from frontend. frontend abnormally exited
Dec 18 12:06:21 pg1 pgpool: 2009-12-18 12:06:21 LOG:   pid 4233: connection received: host=[local]
Dec 18 12:06:28 pg1 pgpool: 2009-12-18 12:06:28 LOG:   pid 4121: connection received: host=[local]
Dec 18 12:06:28 pg1 pgpool: 2009-12-18 12:06:28 ERROR: pid 4121: read_kind_from_backend: 1 th kind E does not match with master or majority connection kind C
Dec 18 12:06:28 pg1 pgpool: 2009-12-18 12:06:28 ERROR: pid 4121: kind mismatch among backends. Possible last query was: "INSERT INTO "ghistoryday" ("data_id", "date", "bytes", "protocol_id") VALUES (1, E'2009-12-18', 2802555, 2)" kind details are: 0[C] 1[E]
Dec 18 12:06:28 pg1 pgpool: 2009-12-18 12:06:28 LOG:   pid 4121: notice_backend_error: 1 fail over request from pid 4121
Dec 18 12:06:28 pg1 pgpool[4319]: Executing pgpool-failover as user postgres
Dec 18 12:06:28 pg1 pgpool[4320]: Failover of node 1 at hostname 10.226.230.163. New master node is 0. Old master node was 0.
Dec 18 12:06:28 pg1 pgpool: 2009-12-18 12:06:28 LOG:   pid 16828: starting degeneration. shutdown host 10.226.230.163(5432)
Dec 18 12:06:28 pg1 pgpool: 2009-12-18 12:06:28 LOG:   pid 16828: execute command: /vol/data/pgpool-failover 1 10.226.230.163 5432 /var/lib/postgresql/8.3/main 0 0
Dec 18 12:06:28 pg1 pgpool: 2009-12-18 12:06:28 LOG:   pid 16828: failover_handler: set new master node: 0
Dec 18 12:06:28 pg1 pgpool: 2009-12-18 12:06:28 LOG:   pid 16828: failover done. shutdown host 10.226.230.163(5432)
Dec 18 12:06:38 pg1 pgpool: 2009-12-18 12:06:38 LOG:   pid 4352: connection received: host=[local]
Dec 18 12:06:49 pg1 pgpool: 2009-12-18 12:06:49 LOG:   pid 4352: connection received: host=[local]


It doesn't look too healthy to me - what exactly happened there?

This is a setup of two postgres servers, connections go through pgpool-ii 2.3.0.
Setup is very lightly loaded.


-- 
Tomasz Chmielewski
http://wpkg.org