[Pgpool-general] pgpool 2.2.4: DEALLOCATED other problem

Sun Sep 27 21:30:15 UTC 2009

Hi,

Got different problem with DEALLOCATED then other one here. After few
hours (4-5h) normal working start see alot DEALLOCATED process at
pgpool and all db backends got such errors.

#v+
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8036eec4"
ERROR:  prepared statement "pdo_pgsql_stmt_8063113c" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8063113c"
ERROR:  prepared statement "pdo_pgsql_stmt_8046a53c" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8046a53c"
ERROR:  prepared statement "pdo_pgsql_stmt_8036eec4" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8036eec4"
ERROR:  prepared statement "pdo_pgsql_stmt_8063113c" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8063113c"
ERROR:  prepared statement "pdo_pgsql_stmt_8046a53c" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8046a53c"
ERROR:  prepared statement "pdo_pgsql_stmt_8036eec4" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8036eec4"
ERROR:  prepared statement "pdo_pgsql_stmt_8063113c" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8063113c"
ERROR:  prepared statement "pdo_pgsql_stmt_8046a53c" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8046a53c"
ERROR:  prepared statement "pdo_pgsql_stmt_8036eec4" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8036eec4"
ERROR:  prepared statement "pdo_pgsql_stmt_8063113c" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8063113c"
ERROR:  prepared statement "pdo_pgsql_stmt_8046a53c" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8046a53c"
ERROR:  prepared statement "pdo_pgsql_stmt_8036eec4" does not exist
STATEMENT:  DEALLOCATE "pdo_pgsql_stmt_8036eec4"
ERROR:  prepared statement "pdo_pgsql_stmt_8063113c" does not exist
#v-

One day such logs got ~10GB, traffic grow up 2x times and load grow up
from around 1 to 4-5

pgpool log when that starts:

#v+
2009-09-27 18:35:08 ERROR: pid 21511: pool_read: read failed
(Connection reset by peer)
2009-09-27 18:35:08 LOG:   pid 21511: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:35:11 LOG:   pid 21512: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:35:11 ERROR: pid 21471: pool_read: read failed
(Connection reset by peer)
2009-09-27 18:35:11 LOG:   pid 21471: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:35:12 ERROR: pid 21566: pool_read: read failed
(Connection reset by peer)
2009-09-27 18:35:12 LOG:   pid 21566: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:35:12 ERROR: pid 21568: pool_read: read failed
(Connection reset by peer)
2009-09-27 18:35:12 LOG:   pid 21568: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:35:12 LOG:   pid 21570: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:35:12 LOG:   pid 21510: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:35:12 ERROR: pid 21511: pool_read: read failed
(Connection reset by peer)
2009-09-27 18:35:12 LOG:   pid 21511: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 ERROR: pid 21511: pool_read: read failed
(Connection reset by peer)
2009-09-27 18:36:34 LOG:   pid 21511: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 LOG:   pid 21605: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 LOG:   pid 21566: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 LOG:   pid 21591: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 ERROR: pid 21594: pool_flush_it: write failed
(Broken pipe) offset: 0 wlen: 740
2009-09-27 18:36:34 ERROR: pid 21563: pool_flush_it: write failed
(Broken pipe) offset: 0 wlen: 30
2009-09-27 18:36:34 ERROR: pid 21606: pool_flush_it: write failed
(Broken pipe) offset: 0 wlen: 156
2009-09-27 18:36:34 LOG:   pid 21563: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 LOG:   pid 21594: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 LOG:   pid 21606: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 ERROR: pid 21563: pool_flush_it: write failed
(Broken pipe) offset: 0 wlen: 377
2009-09-27 18:36:34 ERROR: pid 21606: pool_flush_it: write failed
(Broken pipe) offset: 0 wlen: 12
2009-09-27 18:36:34 ERROR: pid 21563: pool_read_kind: kind does not
match between master(73) slot[1] (84)
2009-09-27 18:36:34 ERROR: pid 21606: pool_read_kind: kind does not
match between master(84) slot[1] (73)
2009-09-27 18:36:34 LOG:   pid 21587: ProcessFrontendResponse: failed
to read kind from frontend. frontend abnormally exited
2009-09-27 18:36:34 ERROR: pid 21594: pool_flush_it: write failed
(Broken pipe) offset: 0 wlen: 3590
2009-09-27 18:36:34 ERROR: pid 21594: pool_read_kind: kind does not
match between master(84) slot[1] (73)
2009-09-27 18:36:36 LOG:   pid 21566: connection closed. retry to
create new connection pool.
#v-

At both backends:
# rpm -qa | grep -ir postgresql
postgresql-libs-8.3.8-1.fc11.x86_64
postgresql-8.3.8-1.fc11.x86_64
postgresql-devel-8.3.8-1.fc11.x86_64
postgresql-server-8.3.8-1.fc11.x86_64

At pool host pgpool-2.2.4

-- 
Łukasz Jagiełło
System Administrator
G-Forces Web Management Polska sp. z o.o. (www.gforces.pl)

Ul. Kruczkowskiego 12, 80-288 Gdańsk
Spółka wpisana do KRS pod nr 246596 decyzją Sądu Rejonowego Gdańsk-Północ