[Pgpool-general] FailoverInProgress message on executing pcp_recovery_node in pgpool-II 3.1

Wed Nov 16 12:57:15 UTC 2011

I found this in your pgpool.conf:

pool_conn_dbname = 'postgres'

Any version of pgpool-II does not have the directive. Is your
pgpool-II a customized one?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> I see some more logs:
> 
> 2011-11-16 14:33:29 LOG:   pid 21932: failover: set new primary node: -1
> 2011-11-16 14:33:29 LOG:   pid 21932: failover: set new master node: 1
> 2011-11-16 14:33:29 ERROR: pid 21932: fork() failed. reason: Cannot allocate memory
> 
> 
> Attaching the pgpool.conf and pgpool.log for your reference.
> 
>  
> 
> 
> ________________________________
> From: Sandeep Thakkar <sandeeptt at yahoo.com>
> To: Sandeep Thakkar <sandeeptt at yahoo.com>; Tatsuo Ishii <ishii at sraoss.co.jp>
> Cc: "pgpool-general at pgfoundry.org" <pgpool-general at pgfoundry.org>
> Sent: Tuesday, November 15, 2011 3:12 PM
> Subject: Re: [Pgpool-general] FailoverInProgress message on executing pcp_recovery_node in pgpool-II 3.1
> 
> 
> Hi Tatsuo
> 
> Please help here. Do I need to set any more directives here? BTW, I guess for pgpool-II 3.1, we just need pgpool-recovery and not pgpool-walrecrunning, right?
> 
> 
> 
> 
> ________________________________
> From: Sandeep Thakkar <sandeeptt at yahoo.com>
> To: Tatsuo Ishii <ishii at sraoss.co.jp>
> Cc: "pgpool-general at pgfoundry.org" <pgpool-general at pgfoundry.org>
> Sent: Monday, November 14, 2011 11:55 AM
> Subject: Re: [Pgpool-general] FailoverInProgress message on executing pcp_recovery_node in pgpool-II 3.1
> 
> 
> I executed the test case again today and I see the following more lines in the log:
> 
> ....
> 
> 2011-11-14 11:21:37 DEBUG: pid 31233: reap_handler called
> 2011-11-14 11:21:37 DEBUG: pid 31233: reap_handler: exited due to switching
> 2011-11-14 11:21:38 DEBUG: pid 31233: pool_ssl: SSL requested but SSL support is not available
> 2011-11-14 11:21:38 DEBUG: pid 31233: s_do_auth: auth kind: 0
> 2011-11-14 11:21:38 ERROR: pid 31233: s_do_auth: unknown response "E" before processing BackendKeyData
> 2011-11-14 11:21:38 ERROR: pid 31233: s_do_auth: unknown response "^@" before processing BackendKeyData
> 2011-11-14 11:21:38 ERROR: pid 31233: s_do_auth: unknown response "^@" before processing BackendKeyData
> 2011-11-14 11:21:38 ERROR: pid 31233: s_do_auth: unknown response "^@" before processing BackendKeyData
> 2011-11-14 11:21:38 ERROR: pid 31233: s_do_auth: unknown response "[" before processing BackendKeyData
> 2011-11-14 11:21:38 DEBUG: pid 31233: s_do_auth: parameter status data received
> 2011-11-14 11:21:38 ERROR: pid 31233: pool_read2: failed to realloc
> .....
> 
> Moreover, the behaviour is same even when I do not set follow_master_command.
> 
> 
> ________________________________
> From: Tatsuo Ishii <ishii at sraoss.co.jp>
> To: sandeeptt at yahoo.com
> Cc: pgpool-general at pgfoundry.org
> Sent: Saturday, November 12, 2011 5:25 PM
> Subject: Re: [Pgpool-general] FailoverInProgress message on executing pcp_recovery_node in pgpool-II 3.1
> 
>> I'm using pgpool-II 3.1 and configured it in Streaming Replication mode. What I notice is that as soon as the
>> 
>> Master goes down, the pgpool server process and all the client processes die. I see only "pgpool: PCP: wait for connection request" 
>> 
>> and "pgpool: worker process" processes.  Though the failover actually happens and I see that Standby is promoted to Master now.
>> 
>> When I try to recover the old Master, I see the following message:
>> 
>> 
>> [edb at localhost regress]$ $PGPOOL_INST/bin/pcp_recovery_node  -d 20 localhost $PCP_PORT edb edb 0
>> DEBUG: send: tos="R", len=41
>> DEBUG: recv: tos="r", len=21, data=AuthenticationOK
>> DEBUG: send: tos="D", len=6
>> DEBUG: recv: tos="e", len=23, data=FailoverInProgress
>> DEBUG: command failed. reason=FailoverInProgress
>> BackendError
>> DEBUG: send: tos="X", len=4
>> 
>> 
>>
>   My question is "why the pgpool server process and child processes got terminated?" The same configuration 
>> works fine against pgpool-II 3.0.3. The only difference between the two configuration is that we added 2 new directives in pgpool.conf,
>> backend_flag and follow_master_command (set same as failover_command).
>> 
>> I see the following lines in the pgpool.log:
>> 2011-11-11 15:41:44 ERROR: pid 27642: health check failed. 0 th host localhost at port 5445 is down
>> 2011-11-11 15:41:44 LOG:   pid 27642: set 0 th backend down status
>> 2011-11-11 15:41:44 DEBUG: pid 27642: failover_handler called
>> 2011-11-11 15:41:44 DEBUG: pid 27642: failover_handler: starting to select new master node
>> 2011-11-11 15:41:44 LOG:   pid 27642: starting degeneration. shutdown host localhost(5445)
>> 2011-11-11 15:41:44 LOG:   pid 27642: Restart all children
>> 2011-11-11
>  15:41:44 DEBUG: pid 27643: child received shutdown request signal 3
>> 2011-11-11 15:41:44 DEBUG: pid 27642: failover_handler: kill 27643
>> ...
>> ...
>> 2011-11-11 15:41:44 DEBUG: pid 27642: failover_handler: kill 27674
>> 2011-11-11 15:41:44 LOG:   pid 27642: execute command: touch /home/edb/AS91/edb-postgres/edb-postgres/src/test/regress/tmp_pgpool/failover.log
>> 2011-11-11 15:41:45 DEBUG: pid 27661: child received shutdown request signal 3
>> 2011-11-11 15:41:45 LOG:   pid 27642: find_primary_node_repeatedly: waiting for finding a primary node
>> 2011-11-11 15:41:45 ERROR: pid 27642: make_persistent_db_connection: s_do_auth failed
>> 2011-11-11 15:41:45 ERROR: pid 27642: find_primary_node: make_persistent_connection failed
> 
> Do you have anything after this line in your log? If so, please show me.
> 
> Also if you do not set follow_master_command, does it work?
> --
> Tatsuo
>  Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
> 
> 
> 
> _______________________________________________
> Pgpool-general mailing list
> Pgpool-general at pgfoundry.org
> http://pgfoundry.org/mailman/listinfo/pgpool-general