<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<p class="MsoNormal">This question has gotten lost… <o:p></o:p></p>
<p class="MsoNormal">Thank you in advance. The attachment is a renamed zip file.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="line-height:15.0pt"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt"><span lang="EN-US" style="font-size:12.0pt;color:black">Regards,<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt"><b><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:12.0pt;color:black">Todd Stein</span></b><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> Todd Stein
<br>
<b>Sent:</b> Wednesday, March 15, 2023 2:33 PM<br>
<b>To:</b> pgpool-general@pgpool.net<br>
<b>Subject:</b> online recovery causes node to loose replication state.<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<p class="MsoNormal">I’m running a 3 node cluster based on PostgreSQL 14 and pgpool 4.4.1. I’m having an issue while testing failover scenarios.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>At start of scenario:<o:p></o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">node_id | hostname | port | status | pg_status | lb_weight | role | pg_role | select_cnt | load_balance_node | replication_delay | replication_state
| replication_sync_state | last_status_change<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">---------+------------------------------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">0 | catvmtspg04a.domain.net | 5432 | up | up | 0.333333 | primary | primary | 0 | true | 0 | |
| 2023-03-15 12:56:31<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">1 | catvmtspg04b.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | streaming |
async | 2023-03-15 12:56:38<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">2 | catvmtspg04c.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | streaming |
async | 2023-03-15 13:46:29<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">(3 rows)<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Times New Roman",serif">After stopping postgres on node0
<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">node_id | hostname | port | status | pg_status | lb_weight | role | pg_role | select_cnt | load_balance_node | replication_delay | replication_state
| replication_sync_state | last_status_change<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">---------+------------------------------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">0 | catvmtspg04a.domain.net | 5432 | down | down | 0.333333 | standby | unknown | 0 | false | 0 | |
| 2023-03-15 12:34:05<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">1 | catvmtspg04b.domain.net | 5432 | up | up | 0.333333 | primary | primary | 0 | true | 0 | |
| 2023-03-15 12:34:05<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">2 | catvmtspg04c.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | streaming |
async | 2023-03-15 12:34:09<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">(3 rows)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Times New Roman",serif">After running recovery on node 0<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Times New Roman",serif">pcp_recovery_node -h 10.78.121.3 -p 9898 -U postgres -n 0<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">node_id | hostname | port | status | pg_status | lb_weight | role | pg_role | select_cnt | load_balance_node | replication_delay | replication_state
| replication_sync_state | last_status_change<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">---------+------------------------------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">0 | catvmtspg04a.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | streaming |
async | 2023-03-15 12:55:59<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">1 | catvmtspg04b.domain.net | 5432 | up | up | 0.333333 | primary | primary | 0 | true | 0 | |
| 2023-03-15 12:34:05<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">2 | catvmtspg04c.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | streaming |
async | 2023-03-15 12:34:09<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">(3 rows)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Times New Roman",serif">After promoting node1 so that node0 becomes primary again:<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Times New Roman",serif">pcp_promote_node -s -n 0 -U pgpool<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Times New Roman",serif">Note that Ioose the replication_state and replication_sync_state on node2, would like to know why this happens.</span></b><span style="font-size:10.0pt;font-family:"Times New Roman",serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">node_id | hostname | port | status | pg_status | lb_weight | role | pg_role | select_cnt | load_balance_node | replication_delay | replication_state
| replication_sync_state | last_status_change<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">---------+------------------------------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">0 | catvmtspg04a.domain.net | 5432 | up | up | 0.333333 | primary | primary | 0 | true | 0 | |
| 2023-03-15 12:56:31<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">1 | catvmtspg04b.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | streaming |
async | 2023-03-15 12:56:38<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">2 | catvmtspg04c.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | |
| 2023-03-15 12:56:38<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">(3 rows)<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>My process to get the cluster back to normal. Don’t understand why the recovery process is failing… Eventually, it works…<o:p></o:p></b></p>
<p class="MsoNormal"><b>pcp_recovery_node -h 10.78.121.3 -p 9898 -U postgres -n 2<o:p></o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">ERROR: recovery is checking if postmaster is started<o:p></o:p></p>
<p class="MsoNormal">DETAIL: postmaster on hostname:"catvmtspg04c.domain.net" database:"template1" user:"postgres" failed to start in 0 second<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[postgres@catvmtspg04a pgpool-II]$ pcp_recovery_node -h 10.78.121.3 -p 9898 -U postgres -n 2<o:p></o:p></p>
<p class="MsoNormal">Password:<o:p></o:p></p>
<p class="MsoNormal">ERROR: recovery is checking if postmaster is started<o:p></o:p></p>
<p class="MsoNormal">DETAIL: postmaster on hostname:"catvmtspg04c.domain.net" database:"template1" user:"postgres" failed to start in 0 second<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[postgres@catvmtspg04a pgpool-II]$ pcp_recovery_node -h 10.78.121.3 -p 9898 -U postgres -n 2<o:p></o:p></p>
<p class="MsoNormal">Password:<o:p></o:p></p>
<p class="MsoNormal"><b>pcp_recovery_node -- Command Successful<o:p></o:p></b></p>
<p class="MsoNormal"><b>cluster back to normal.<o:p></o:p></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">[postgres@catvmtspg04a pgpool-II]$ psql -h 10.78.121.3 -p 9999 -U pgpool postgres -c "show pool_nodes"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">Password for user pgpool:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">node_id | hostname | port | status | pg_status | lb_weight | role | pg_role | select_cnt | load_balance_node | replication_delay | replication_state
| replication_sync_state | last_status_change<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">---------+------------------------------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">0 | catvmtspg04a.domain.net | 5432 | up | up | 0.333333 | primary | primary | 0 | true | 0 | |
| 2023-03-15 12:56:31<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">1 | catvmtspg04b.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | streaming |
async | 2023-03-15 12:56:38<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">2 | catvmtspg04c.domain.net | 5432 | up | up | 0.333333 | standby | standby | 0 | false | 0 | streaming |
async | 2023-03-15 13:46:29<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Times New Roman",serif">(3 rows)<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt"><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt"><span lang="EN-US" style="font-size:12.0pt;color:black">Regards,<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt"><b><span lang="EN-US" style="font-size:12.0pt;color:black"><o:p> </o:p></span></b></p>
<p class="MsoNormal" style="line-height:15.0pt"><b><span lang="EN-US" style="font-size:12.0pt;color:black">Todd Stein</span></b><span lang="EN-US" style="font-size:12.0pt;color:#58595B"><br>
OpsBridge Technical Success<o:p></o:p></span></p>
<p class="MsoNormal" style="line-height:15.0pt"><b><span lang="EN-US" style="font-size:14.0pt">OpenText<o:p></o:p></span></b></p>
<p class="MsoNormal" style="line-height:15.0pt"><span lang="EN-US" style="font-size:12.0pt;color:#58595B">(Cell)</span><span lang="EN-US" style="font-size:12.0pt;color:red">
</span><span lang="EN-US" style="font-size:12.0pt">+1 (941) 248-8752<span style="color:#58595B"><br>
</span><span style="color:#0179EF"><a href="mailto:tstein2@opentext.com">tstein2@opentext.com</a></span></span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>