<div dir="ltr">Hi Ishii-San,<div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 8, 2021 at 7:33 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Usama,<br>
<br>
Thank you for the work. I have added it to the Pgpool-II 4.3 release<br>
note. Can you please take a look at to check if I have<br>
misunderstood anything regarding the commit.<br></blockquote><div><br></div><div>Overall it looks good. I have tried to rephrase it a tiny bit. Can you see the attached</div><div>release_notes.diff if it looks ok to you? </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
One thing I noticed in the doc you added is, you did not mention about<br>
the risk of split-brain because of this feature enabled. Should we add<br>
that?<br>
<br></blockquote><div><br></div>You are absolutely right. We should definitely mention the risks.<br><br>What do you think about this below caution message for documentation?</div><div class="gmail_quote"><br>--quote--<br><font face="monospace">Using the dynamic cluster membership has an associated risk of causing a split-brain.<br>So it is strongly recommended to carefully consider if the setup requires the dynamic cluster<br>membership and try to use conservative values for related settings.<br></font>--un-quote--<br><br>If you agree with the above message. I will put that in the respective documentation file.<br><br>Thanks<br>Best regards<br>Muhammad Usama</div><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> Hi Ishii-San,<br>
> <br>
> On Tue, Nov 2, 2021 at 5:58 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
> <br>
>> Hi Usama,<br>
>><br>
>> I confirmed you patch works as expected. Thank you for your great work!<br>
>><br>
> <br>
> Many thanks for the confirmation. I have made a few cosmetic changes and<br>
> committed the patch and documentation update.<br>
> <br>
> Best Regards<br>
> Muhammad Usama<br>
> <br>
> <br>
> <br>
>> > Hi Tatsuo,<br>
>> ><br>
>> > On Mon, Nov 1, 2021 at 12:21 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
>> ><br>
>> >> Hi Usama,<br>
>> >><br>
>> >> Thank you for the patch. Unfortunately the patch does not apply to the<br>
>> >> master branch anymore. Can you please rebase it?<br>
>> >><br>
>> ><br>
>> > Please find the rebased patch<br>
>> ><br>
>> > Thanks<br>
>> > Best regards<br>
>> > Muhammad Usama<br>
>> ><br>
>> ><br>
>> >> --<br>
>> >> Tatsuo Ishii<br>
>> >> SRA OSS, Inc. Japan<br>
>> >> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>> >> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>> >><br>
>> >> > Hi,<br>
>> >> ><br>
>> >> > So I have cooked up a WIP patch that implements the above discussed<br>
>> >> > behavior.<br>
>> >> ><br>
>> >> > The attached patch adds three new configuration parameters<br>
>> >> ><br>
>> >> > #wd_remove_shutdown_nodes = off<br>
>> >> > # when enabled properly shutdown<br>
>> >> > watchdog nodes get<br>
>> >> > # removed from the cluster and<br>
>> does<br>
>> >> not<br>
>> >> > count towards<br>
>> >> > # the quorum and consensus<br>
>> >> computations<br>
>> >> ><br>
>> >> > #wd_lost_node_removal_timeout = 0s<br>
>> >> > # Time after which the LOST<br>
>> watchdog<br>
>> >> > nodes get<br>
>> >> > # removed from the cluster and<br>
>> does<br>
>> >> not<br>
>> >> > count towards<br>
>> >> > # the quorum and consensus<br>
>> >> computations<br>
>> >> > # setting it to 0 will never<br>
>> remove<br>
>> >> the<br>
>> >> > LOST nodes<br>
>> >> ><br>
>> >> > #wd_initial_node_showup_time = 0s<br>
>> >> ><br>
>> >> > # Time to wait for Watchdog nodes<br>
>> to<br>
>> >> > connect to the cluster.<br>
>> >> > # After that time the nodes are<br>
>> >> > considered to be not part of<br>
>> >> > # the cluster and will not count<br>
>> >> towards<br>
>> >> > # the quorum and consensus<br>
>> >> computations<br>
>> >> > # setting it to 0 will wait<br>
>> forever<br>
>> >> ><br>
>> >> ><br>
>> >> > Keeping the default values for these parameters retains the existing<br>
>> >> > behavior.<br>
>> >> ><br>
>> >> ><br>
>> >> > Moreover, the patch also enhances the wd_watchdog_info utility to<br>
>> output<br>
>> >> > the current "Quorum State"<br>
>> >> ><br>
>> >> > for each watchdog node and "number of nodes require for quorum" and<br>
>> >> "valid<br>
>> >> > remote nodes count" as per<br>
>> >> ><br>
>> >> > the current status of watchdog cluster. This change might also require<br>
>> >> the<br>
>> >> > bump of pcp lib version.<br>
>> >> ><br>
>> >> ><br>
>> >> ><br>
>> >> > bin/pcp_watchdog_info -U postgres -v<br>
>> >> > Watchdog Cluster Information<br>
>> >> > Total Nodes : 3<br>
>> >> > Remote Nodes : 2<br>
>> >> ><br>
>> >> > *Valid Remote Nodes : 1*Alive Remote Nodes : 0<br>
>> >> ><br>
>> >> > *Nodes required for quorum: 2*Quorum state : QUORUM ABSENT<br>
>> >> > VIP up on local node : NO<br>
>> >> > Leader Node Name : localhost:9990 Darwin<br>
>> Usama-Macbook-Pro.local<br>
>> >> > Leader Host Name : localhost<br>
>> >> ><br>
>> >> > Watchdog Node Information<br>
>> >> > Node Name : localhost:9990 Darwin Usama-Macbook-Pro.local<br>
>> >> > ...<br>
>> >> > Status Name : LEADER<br>
>> >> ><br>
>> >> > *Quorum State : ACTIVE*<br>
>> >> > Node Name : localhost:9991 Darwin Usama-Macbook-Pro.local<br>
>> >> > ...<br>
>> >> > Status : 10<br>
>> >> > Status Name : SHUTDOWN<br>
>> >> > *Quorum State : ACTIVE*<br>
>> >> ><br>
>> >> > Node Name : Not_Set<br>
>> >> > ...<br>
>> >> > Status Name : DEAD<br>
>> >> ><br>
>> >> > *Quorum State : REMOVED-NO-SHOW*<br>
>> >> ><br>
>> >> > The patch is still in WIP state mainly because it lacks the<br>
>> documentation<br>
>> >> > updates, and I am<br>
>> >> > sharing it to get an opinion and suggestions on the behavior and<br>
>> >> > configuration parameter names.<br>
>> >> ><br>
>> >> > Thanks<br>
>> >> > Best regards<br>
>> >> > Muhammad Usama<br>
>> >> ><br>
>> >> ><br>
>> >> > On Mon, Aug 23, 2021 at 6:05 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>> wrote:<br>
>> >> ><br>
>> >> >> Hi Usama,<br>
>> >> >><br>
>> >> >> Sorry for late reply.<br>
>> >> >><br>
>> >> >> From: Muhammad Usama <<a href="mailto:m.usama@gmail.com" target="_blank">m.usama@gmail.com</a>><br>
>> >> >> Subject: Re: [pgpool-hackers: 3898] Re: [pgpool-general: 7543] VIP<br>
>> with<br>
>> >> >> one node<br>
>> >> >> Date: Thu, 22 Jul 2021 14:12:59 +0500<br>
>> >> >> Message-ID: <<br>
>> >> >> <a href="mailto:CAEJvTzXsKE2B0QMd0AjGBmXK6zocWZZcGU7yzzkSnmff0iAfqA@mail.gmail.com" target="_blank">CAEJvTzXsKE2B0QMd0AjGBmXK6zocWZZcGU7yzzkSnmff0iAfqA@mail.gmail.com</a>><br>
>> >> >><br>
>> >> >> > On Tue, Jul 20, 2021 at 4:40 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>> >> wrote:<br>
>> >> >> ><br>
>> >> >> >> >> Is it possible to configure watchdog to enable the lost node<br>
>> >> removal<br>
>> >> >> >> >> function only when a node is properly shutdown?<br>
>> >> >> >> >><br>
>> >> >> >><br>
>> >> >> >> > Yes if we disable the wd_lost_node_to_remove_timeout (by<br>
>> setting it<br>
>> >> >> to 0)<br>
>> >> >> >> > the lost node removal will only happen for properly shutdown<br>
>> nodes.<br>
>> >> >> >><br>
>> >> >> >> Oh, I thought setting wd_lost_node_to_remove_timeout to 0 will<br>
>> keep<br>
>> >> >> >> the existing behavior.<br>
>> >> >> >><br>
>> >> >> ><br>
>> >> >> > As there are two parts of the proposal, First one deals with<br>
>> removing<br>
>> >> the<br>
>> >> >> > lost node<br>
>> >> >> > from the cluster after wd_lost_node_to_remove_timeout amount of<br>
>> time.<br>
>> >> >> While<br>
>> >> >> > the<br>
>> >> >> > second part is about removing the properly shutdown nodes from the<br>
>> >> >> cluster.<br>
>> >> >> ><br>
>> >> >> > Now disabling the wd_lost_node_to_remove_timeout (setting it to 0)<br>
>> >> will<br>
>> >> >> > keep the<br>
>> >> >> > existing behaviour as far as removing the lost node portion of<br>
>> >> proposal<br>
>> >> >> is<br>
>> >> >> > concerned.<br>
>> >> >> ><br>
>> >> >> > While not counting the properly shutdown node as part of watchdog<br>
>> >> cluster<br>
>> >> >> > is not configurable (as per original proposal), So if we want to<br>
>> make<br>
>> >> >> this<br>
>> >> >> > part configurable<br>
>> >> >> > as well so that we can switch to 100% current behaviour then we can<br>
>> >> add<br>
>> >> >> > another<br>
>> >> >> > config parameter for that. like<br>
>> >> >> consider_shutdown_nodes_part_of_wd_cluster<br>
>> >> >> > = [on|off]<br>
>> >> >><br>
>> >> >> +1 to add the new parameter.<br>
>> >> >><br>
>> >> >> The reason is, some users may want to avoid split brain problem even<br>
>> >> >> if quorum/VIP is lost. Suppose there are two admins A for the system<br>
>> >> >> (OS), B for the database. B never wants to have the split brain<br>
>> >> >> possibility. If A shutdowns the system, B may not notice there are<br>
>> not<br>
>> >> >> enough nodes to form consensus anymore because if<br>
>> >> >> consider_shutdown_nodes_part_of_wd_cluster is on because the<br>
>> >> >> quorum/VIP will be kept until no node remains.<br>
>> >> >><br>
>> >> >> In summary I think there are two use-cases for both<br>
>> >> >> consider_shutdown_nodes_part_of_wd_cluster is on and off.<br>
>> >> >> --<br>
>> >> >> Tatsuo Ishii<br>
>> >> >> SRA OSS, Inc. Japan<br>
>> >> >> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>> >> >> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>> >> >><br>
>> >> >><br>
>> >><br>
>><br>
</blockquote></div></div></div>