[Pgpool-general] Mismatch among backends

[Jaume Sabater]

Hi all!

Just tried to connect to my pgpool-II 2.2/PostgreSQL 8.3 cluster and
saw an error, which I forgot to copy and paste somewhere, that said
something like "error in catalog with relid 26243" (I only copied the
number). I checked the cluster and, again, there had been a kind
mismatch among backends, so the slave node was down and the cluster
was working only with the master node.

This is what I found on the syslog:

Mar  6 08:10:27 pgsql1 pgpool: ERROR: pid 26306:
read_kind_from_backend: 1 th kind E does not match with master or
majority connection kind C
Mar  6 08:10:27 pgsql1 pgpool: ERROR: pid 26306: kind mismatch among
backends. Possible last query was: "COPY "TSearcherServices"
("IdSearcherServices", "IdSearcher" ,"IdService", "SearcherNumber" )
Mar  6 08:10:27 pgsql1 pgpool: FROM '/opt/pgpool2/TSearcherServices.csv'
Mar  6 08:10:27 pgsql1 pgpool: WITH DELIMITER AS '|' CSV;" kind
details are: 0[C] 1[E]
Mar  6 08:10:27 pgsql1 pgpool: LOG:   pid 26306: notice_backend_error:
1 fail over request from pid 26306
Mar  6 08:10:27 pgsql1 pgpool: LOG:   pid 5315: starting degeneration.
shutdown host pgsql2.freyatest.domain(5432)
Mar  6 08:10:27 pgsql1 pgpool: LOG:   pid 5315: execute command:
/var/lib/postgresql/8.3/main/pgpool-failover 1 pgsql2.freyatest.domain
5432 /var/lib/postgresql/8.3/main 0 0
Mar  6 08:10:27 pgsql1 pgpool[32211]: Executing pgpool-failover as user postgres
Mar  6 08:10:27 pgsql1 pgpool[32212]: Failover of node 1 at hostname
pgsql2.freyatest.domain. New master node is 0. Old master node was 0.
Mar  6 08:10:27 pgsql1 pgpool: LOG:   pid 5315: failover_handler: set
new master node: 0
Mar  6 08:10:27 pgsql1 pgpool: LOG:   pid 5315: failover done.
shutdown host pgsql2.freyatest.domain(5432)


These COPY operations have been very frequent during the last three or
four months, with developers constantly dumping information here and
there. With version 2.1 I never had a mismatch among backends, but now
I have had 2 of those this very same week, plus a few more the
previous couple of weeks (we were working with betas or RCs of version
2.2). I can't really point at version 2.2 regarding the issue, but I
promise I don't recall it happening with version 2.1. It is true that
the number of operations on the PostgreSQL cluster have increased a
lot in the last 4 weeks, too.

Tatsuo, could you please check it out? Here you are the other error
that happened this week. Notice the query was different. Logs from
past week are gone, unfortunately.

Mar  5 14:50:26 pgsql1 pgpool: ERROR: pid 20120: pool_read: read
failed (Connection reset by peer)
Mar  5 14:50:26 pgsql1 pgpool: LOG:   pid 20120:
ProcessFrontendResponse: failed to read kind from frontend. frontend
abnormally exited
Mar  5 14:50:26 pgsql1 pgpool: LOG:   pid 20120:
read_kind_from_backend: parameter name: is_superuser value: on
Mar  5 14:50:26 pgsql1 pgpool: LOG:   pid 20120:
read_kind_from_backend: parameter name: session_authorization value:
pgpool2
Mar  5 14:50:26 pgsql1 pgpool: LOG:   pid 20120:
read_kind_from_backend: parameter name: is_superuser value: on
Mar  5 14:50:26 pgsql1 pgpool: LOG:   pid 20120:
read_kind_from_backend: parameter name: session_authorization value:
pgpool2
Mar  5 14:50:57 pgsql1 pgpool: LOG:   pid 19950:
read_kind_from_backend: parameter name: is_superuser value: on
Mar  5 14:50:57 pgsql1 pgpool: LOG:   pid 19950:
read_kind_from_backend: parameter name: session_authorization value:
pgpool2
Mar  5 14:50:57 pgsql1 pgpool: LOG:   pid 19950:
read_kind_from_backend: parameter name: is_superuser value: on
Mar  5 14:50:57 pgsql1 pgpool: LOG:   pid 19950:
read_kind_from_backend: parameter name: session_authorization value:
pgpool2
Mar  5 14:51:08 pgsql1 pgpool: ERROR: pid 19538:
read_kind_from_backend: 1 th kind E does not match with master or
majority connection kind C
Mar  5 14:51:08 pgsql1 pgpool: ERROR: pid 19538: kind mismatch among
backends. Possible last query was: "delete from "TSearcher"" kind
details are: 0[C] 1[E]
Mar  5 14:51:08 pgsql1 pgpool: LOG:   pid 19538: notice_backend_error:
1 fail over request from pid 19538
Mar  5 14:51:08 pgsql1 pgpool: LOG:   pid 5315: starting degeneration.
shutdown host pgsql2.freyatest.domain(5432)
Mar  5 14:51:08 pgsql1 pgpool: LOG:   pid 5315: execute command:
/var/lib/postgresql/8.3/main/pgpool-failover 1 pgsql2.freyatest.domain
5432 /var/lib/postgresql/8.3/main 0 0
Mar  5 14:51:08 pgsql1 pgpool[20704]: Executing pgpool-failover as user postgres
Mar  5 14:51:08 pgsql1 pgpool[20705]: Failover of node 1 at hostname
pgsql2.freyatest.domain. New master node is 0. Old master node was 0.
Mar  5 14:51:08 pgsql1 pgpool: LOG:   pid 5315: failover_handler: set
new master node: 0
Mar  5 14:51:08 pgsql1 pgpool: LOG:   pid 5315: failover done.
shutdown host pgsql2.freyatest.domain(5432)


Anyone else having this problem?

-- 
Jaume Sabater
http://linuxsilo.net/

"Ubi sapientas ibi libertas"