[Pgpool-general] Recovery randomly failing - could not open relation with OID

[Fernando Morgenstern]

Hello,

I have pgpool with the following recovery script http://pastebin.ca/1745584 for 1st and 2nd stage command.

I had a node that was dettached for a long time and was trying to recover it during this weekend. While the recovery was run, i also had the following script being run:

#!/bin/bash
COUNT=50000
TOTAL=15000

while [ $COUNT != $TOTAL ];
do
        psql -h 192.168.100.50 test -c "INSERT INTO test VALUES ( $COUNT , $COUNT );" -U postgres
        COUNT=`expr $COUNT + 1`
done

The first time that i tried to recover the node, it worked perfectly. I verified test table and it had the same number of rows on all nodes.

But, when i tried to recover it again, i got the following error:

ERROR:  kind mismatch among backends. Possible last query was: "INSERT INTO test VALUES ( 54600 , 54600 );" kind details are: 0[C] 1[C] 2[E: could not open relation with OID 326264]
HINT:  check data consistency among db nodes
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
connection to server was lost

Could someone tell me what causes this error and what is the correct way to fix and avoid having it in the future?

I am not sure how can i should proceed. I just dettached node 2 ( the one that was being recovered ) and have left pgpool running with the other 2 nodes. I verified that i can't use table test on node 2 anymore.

Best Regards,
---

Fernando Marcelo
www.consultorpc.com
fernando at consultorpc.com