[Pgpool-general] forcing pool_flush_it error

[Marcelo Martins]

Does anyone know how I could force this pool_flush_it error to happen ?

Aug  9 06:15:24 pr2-pgpool1-1 pgpool: 2009-08-09 06:15:24 LOG:   pid  
4802: authentication is timeout
Aug  9 06:25:10 pr2-pgpool1-1 pgpool: 2009-08-09 06:25:10 ERROR: pid  
4470: pool_flush_it: write failed (Broken pipe) offset: 0 wlen: 8192
Aug  9 06:25:10 pr2-pgpool1-1 pgpool: 2009-08-09 06:25:10 ERROR: pid  
1193: pool_flush_it: write failed (Broken pipe) offset: 0 wlen: 8192
Aug  9 06:25:17 pr2-pgpool1-1 pgpool: 2009-08-09 06:25:17 LOG:   pid  
4470: connection closed. retry to create new connection pool.
Aug  9 06:25:25 pr2-pgpool1-1 pgpool: 2009-08-09 06:25:25 LOG:   pid  
4845: authentication is timeout
Aug  9 06:25:33 pr2-pgpool1-1 pgpool: 2009-08-09 06:25:33 LOG:   pid  
1193: connection closed. retry to create new connection pool.


On a staging environment I have tried exhaust pgpool the best I can  
with several QA tests using our in-house apps.
I have tried killing apache, restarting it gracefully and non- 
gracefully ... killing PostgreSQL backend processes (which causes a  
pool_read error) .
I have tried killing pgbouncer processes which talk to pgpool (we have  
apache -> pgbouncer -> pgpool-II 2.2.2 -> postgresql 8.3.7)

I have gone through some of the code on pool_process_query.c and  
pool_stream.c to grasp some understanding of the flush, got a bit but  
any clarifications welcome.

It seems like the databases don't get out of sync every time a  
pool_flush_it error occurs but most of the time it does cause an issue.
Once that happens the PG logs start to spit out several of the lines  
below found below.

....
2009-08-09 17:47:29 UTC 10.7.5.6(42839) 8764ERROR:  syntax error at or  
near "send" at character 1
2009-08-09 17:47:29 UTC 10.7.5.6(42839) 8764STATEMENT:  send invalid  
query from pgpool to abort transaction
2009-08-09 17:47:29 UTC 10.7.5.6(42839) 8764ERROR:  syntax error at or  
near "send" at character 1
.....


So far, this pool_flush_it errors that happened two sundays in a row  
at same time coincides with the same time that logrotate restarts  
apache2 after weekly rotating logs.
That is the first assumption right now  but doesn't quite explain why  
it only happens on one set of pgpool/database systems when it should  
happen on all sets of pgpool/database since rotation
happens on all systems.

Any ideas on how I could replicate this pool_flush_it error is very  
appreciated

Marcelo
PostgreSQL DBA
Linux/Solaris System Administrator