<div dir="ltr">Hi Ishii-San<div><br></div><div>I have tried to rephrase your suggestion for clarity. Please have a look at the attached patch if you see it fit</div><div><br></div><div><br></div><div>Thanks</div><div>Best Regards</div><div>Muhammad Usama</div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 11, 2018 at 1:30 PM, Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Ok, here is a proposal for addition to the doc.<br>
<div class="HOEnZb"><div class="h5"><br>
Best regards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
<br>
> Usama,<br>
> <br>
> Do we want to add some notes to the doc regarding this? The behavior<br>
> described below may not be obvious to users.<br>
> <br>
> Best regards,<br>
> --<br>
> Tatsuo Ishii<br>
> SRA OSS, Inc. Japan<br>
> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_<wbr>en.php</a><br>
> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.<wbr>jp</a><br>
> <br>
>> Hi<br>
>> <br>
>> Thanks for the logs and config files.<br>
>> As per the logs and pgpool.conf files, This is what is happening.<br>
>> <br>
>> You have health check disabled on all Pgpool-II nodes, So only way to<br>
>> detect the backend failure is through fail_over_on_backend error( which<br>
>> only works when client connection<br>
>> detects the error) . But since the clients are only connecting to the<br>
>> master Pgpool-II node, so only master Pgpool-II node can notice the backend<br>
>> PostgreSQL node failure<br>
>> and because of consensus requirement it will keep waiting for the detection<br>
>> of backend failure by other Pgpool-II nodes, Which never arrives because<br>
>> other two Pgpool-II nodes<br>
>> are sitting idle and didn't detected the error.<br>
>> So you either need to enable the health check on all pgpool-II nodes (<br>
>> Which is the recommended setting for HA) or just disable the consensus<br>
>> requirements (as you did when failover<br>
>> was working fine)<br>
>> <br>
>> Thanks<br>
>> Best Rgeards<br>
>> Muhammad Usama<br>
>> <br>
>> On Tue, May 8, 2018 at 7:54 PM, Vlad G <<a href="mailto:omenvlad@gmail.com">omenvlad@gmail.com</a>> wrote:<br>
>> <br>
>>> Hey Guys.<br>
>>> Thank you for your answer.<br>
>>> I attached the configuration files of pgpool and logs.<br>
>>> I hope you сan help.<br>
>>><br>
>> <br>
>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> Best regards,<br>
>>> Vladyslav<br>
>>><br>
>>><br>
>>> On May 7, 2018, at 16:05, Muhammad Usama <<a href="mailto:m.usama@gmail.com">m.usama@gmail.com</a>> wrote:<br>
>>><br>
>>> Hi<br>
>>><br>
>>> From the log snippet you shared it seems that the the failure was never<br>
>>> detected by the other Pgpool-II node, Can you please share the pgpoo.conf<br>
>>> files and log files for all Pgpool nodes.<br>
>>><br>
>>> Thanks<br>
>>> Best Regards<br>
>>> Muhammad Usama<br>
>>><br>
>>> On Thu, May 3, 2018 at 5:20 PM, Vlad G <<a href="mailto:omenvlad@gmail.com">omenvlad@gmail.com</a>> wrote:<br>
>>><br>
>>>> Hey Guys,<br>
>>>> I have a cluster with Pgpool-II-pg96-3.7.3 and postgresql-9.6.<br>
>>>> (3 x pgpool and 3 x postgresql<br>
>>>> The same scheme as:<br>
>>>> <a href="http://www.pgpool.net/docs/latest/en/html/example-cluster.html" rel="noreferrer" target="_blank">http://www.pgpool.net/docs/<wbr>latest/en/html/example-<wbr>cluster.html</a><br>
>>>><br>
>>>> When master node of postgresql (pgpoolpsql-1) goes down the master node<br>
>>>> of pgpool ( pgpool-1) does not get second vote from one of the standby<br>
>>>> pgpool nodes (pgpool-2 and pgpool-3).<br>
>>>><br>
>>>> If I set:<br>
>>>> failover_require_consensus = off<br>
>>>> Everything works fine.<br>
>>>><br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237:<br>
>>>> LOG: failed to connect to PostgreSQL server on "pgpoolpsql-1:5432",<br>
>>>> getsockopt() detected error "Connection refused"<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237:<br>
>>>> LOG: received degenerate backend request for node_id: 0 from pid [24237]<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217:<br>
>>>> LOG: new IPC connection received<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217:<br>
>>>> LOG: watchdog received the failover command from local pgpool-II on IPC<br>
>>>> interface<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217:<br>
>>>> LOG: watchdog is processing the failover command<br>
>>>> [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217:<br>
>>>> LOG: failover requires the majority vote, waiting for consensus<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217:<br>
>>>> DETAIL: failover request noted<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24217:<br>
>>>> LOG: failover command [DEGENERATE_BACKEND_REQUEST] request from pgpool-II<br>
>>>> node "pgpool-1:9999 Linux pgpool-1" is queued, waiting for the confirmation<br>
>>>> from other nodes<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237:<br>
>>>> LOG: degenerate backend request for node_id: 0 from pid [24237], will be<br>
>>>> handled by watchdog, which is building consensus for request<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237:<br>
>>>> FATAL: failed to create a backend connection<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24237:<br>
>>>> DETAIL: executing failover on backend<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24216:<br>
>>>> LOG: child process with pid: 24237 exits with status 256<br>
>>>> May 03 13:02:45 pgpool-1 pgpool[24216]: 2018-05-03 13:02:45: pid 24216:<br>
>>>> LOG: fork a new child process with pid: 24268<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228:<br>
>>>> LOG: failed to connect to PostgreSQL server on "pgpoolpsql-1:5432",<br>
>>>> getsockopt() detected error "Connection refused"<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228:<br>
>>>> LOG: received degenerate backend request for node_id: 0 from pid [24228]<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: new IPC connection received<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: watchdog received the failover command from local pgpool-II on IPC<br>
>>>> interface<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: watchdog is processing the failover command<br>
>>>> [DEGENERATE_BACKEND_REQUEST] received from local pgpool-II on IPC interface<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: Duplicate failover request from "pgpool-1:9999 Linux pgpool-1" node<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> DETAIL: request ignored<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: failover requires the majority vote, waiting for consensus<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> DETAIL: failover request noted<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228:<br>
>>>> LOG: degenerate backend request for 1 node(s) from pid [24228], is changed<br>
>>>> to quarantine node request by watchdog<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228:<br>
>>>> DETAIL: watchdog is taking time to build consensus<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228:<br>
>>>> FATAL: failed to create a backend connection<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24228:<br>
>>>> DETAIL: executing failover on backend<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216:<br>
>>>> LOG: Pgpool-II parent process has received failover request<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: new IPC connection received<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: received the failover indication from Pgpool-II on IPC interface<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: watchdog is informed of failover end by the main process<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216:<br>
>>>> LOG: starting quarantine. shutdown host pgpoolpsql-1(5432)<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216:<br>
>>>> LOG: Restart all children<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216:<br>
>>>> LOG: failover: set new primary node: -1<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24216:<br>
>>>> LOG: failover: set new master node: 1<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24252:<br>
>>>> LOG: worker process received restart request<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: new IPC connection received<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: received the failover indication from Pgpool-II on IPC interface<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: 2018-05-03 13:02:46: pid 24217:<br>
>>>> LOG: watchdog is informed of failover start by the main process<br>
>>>> May 03 13:02:46 pgpool-1 pgpool[24216]: quarantine done. shutdown host<br>
>>>> pgpoolpsql-1(5432)2018-05-03 13:02:46: pid 24216: LOG: quarantine done.<br>
>>>> shutdown host pgpoolpsql-1(5432)<br>
>>>> May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24251:<br>
>>>> LOG: restart request received in pcp child process<br>
>>>> May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216:<br>
>>>> LOG: PCP child 24251 exits with status 0 in failover()<br>
>>>> May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216:<br>
>>>> LOG: fork a new PCP child pid 24301 in failover()<br>
>>>> May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216:<br>
>>>> LOG: child process with pid: 24219 exits with status 0<br>
>>>> May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216:<br>
>>>> LOG: child process with pid: 24219 exited with success and will not be<br>
>>>> restarted<br>
>>>> May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216:<br>
>>>> LOG: child process with pid: 24220 exits with status 0<br>
>>>> May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216:<br>
>>>> LOG: child process with pid: 24220 exited with success and will not be<br>
>>>> restarted<br>
>>>> May 03 13:02:47 pgpool-1 pgpool[24216]: 2018-05-03 13:02:47: pid 24216:<br>
>>>> LOG: child process with pid: 24221 exits with status 0<br>
>>>><br>
>>>> Around a month ago it woked fine (It seems I tested it on pgpool-3.7.2),<br>
>>>> but now it does not work. Could you tell me some parameters what it depends<br>
>>>> on or you have other thoughts.<br>
>>>><br>
>>>> Best regards,<br>
>>>> Vladyslav<br>
>>>><br>
>>>> ______________________________<wbr>_________________<br>
>>>> pgpool-general mailing list<br>
>>>> <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
>>>> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-general</a><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> ______________________________<wbr>_________________<br>
>>> pgpool-general mailing list<br>
>>> <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
>>> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-general</a><br>
>>><br>
>>><br>
> ______________________________<wbr>_________________<br>
> pgpool-general mailing list<br>
> <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/<wbr>listinfo/pgpool-general</a><br>
</div></div><br>diff --git a/doc/src/sgml/watchdog.sgml b/doc/src/sgml/watchdog.sgml<br>
index 7e7adc9..041686b 100644<br>
--- a/doc/src/sgml/watchdog.sgml<br>
+++ b/doc/src/sgml/watchdog.sgml<br>
@@ -442,6 +442,16 @@<br>
<para><br>
Default is on.<br>
</para><br>
+<br>
+ <caution><br>
+ <para><br>
+ To make <varname>failover_require_<wbr>consensus</varname><br>
+ workable, You need to enable health check. For more<br>
+ details of health check,<br>
+ see <xref linkend="runtime-config-<wbr>health-check">.<br>
+ </para><br>
+ </caution><br>
+<br>
<para><br>
<varname>failover_require_<wbr>consensus</varname> is not available prior to<br>
<productname>Pgpool-II </productname><emphasis>V3.7</<wbr>emphasis>. and it is only<br>
<br></blockquote></div><br></div></div></div>