[pgpool-general: 6874] Re: watchdog fails to start pgpool-4.1.0

Thu Feb 13 01:08:29 JST 2020

I turned on enable_consensus_with_half_votes which I’m getting the
acquired delegate IP on node 0. But now when I start pgpool on node 1
getting this in the log file which is repeating - see below.  When I check
which node has the virtual IP is show that node 0 does which is the master
node. 

2020-02-12 08:11:52: pid 29493: LOG:  watchdog node state changed from
[INITIALIZING] to [MASTER]
2020-02-12 08:11:52: pid 29493: LOG:  I am announcing my self as
master/coordinator watchdog node
2020-02-12 08:11:52: pid 29493: LOG:  remote node "" decided it is the
true master
2020-02-12 08:11:52: pid 29493: DETAIL:  re-initializing the local
watchdog cluster state because of split-brain
2020-02-12 08:11:52: pid 29493: LOG:  watchdog node state changed from
[MASTER] to [JOINING]
2020-02-12 08:11:53: pid 29493: LOG:  new watchdog node connection is
received from "10.11.0.202:12399"
2020-02-12 08:11:56: pid 29493: LOG:  watchdog node state changed from
[JOINING] to [INITIALIZING]
2020-02-12 08:11:57: pid 29493: LOG:  I am the only alive node in the
watchdog cluster
2020-02-12 08:11:57: pid 29493: HINT:  skipping stand for coordinator state

My environment
2 pgpool hosts on Ubuntu 18
2 postgresql hosts on Ubuntu 18 postgreSQL 11

Wolfgang Schwurack
Database/System Administrator
Utah Education Network
801-587-9444
Wolf at uen.org

On 2/11/20, 3:50 PM, "Tatsuo Ishii" <ishii at sraoss.co.jp> wrote:

>Have you turned on enable_consensus_with_half_votes?
>From 4.1 you need to turn on this if you use even number of Pgpool-II
>nodes.
>It's documented in the migration section in the doc:
>https://www.pgpool.net/docs/latest/en/html/release-4-1-0.html
>
>Best regards,
>--
>Tatsuo Ishii
>SRA OSS, Inc. Japan
>English: http://www.sraoss.co.jp/index_en.php
>Japanese:http://www.sraoss.co.jp
>
>From: Wolf Schwurack <wolf at uen.org>
>Subject: [pgpool-general: 6865] Re: watchdog fails to start pgpool-4.1.0
>Date: Tue, 11 Feb 2020 18:10:25 +0000
>Message-ID: <56216C05-00F8-4C10-A32A-C793411C7891 at umail.utah.edu>
>
>> After doing some more testing on version 4.1.0 I have notice that if
>>node 0 fails, node 1 never acquires the delegate IP. I compared this to
>>version 4.0.5 which when node 0 fails, node 1 acquires the delegate IP
>> 
>> Wolfgang Schwurack
>> Database/System Administrator
>> Utah Education Network
>> 801-587-9444
>> wolf at uen.org<mailto:wolf at uen.org>
>> 
>> From: "pgpool-general-bounces at pgpool.net"
>><pgpool-general-bounces at pgpool.net> on behalf of Wolfgang Schwurack
>><wolf at uen.org>
>> Date: Tuesday, February 11, 2020 at 10:54 AM
>> To: "pgpool-general at pgpool.net" <pgpool-general at pgpool.net>
>> Subject: [pgpool-general: 6864] Re: watchdog fails to start pgpool-4.1.0
>> 
>> It seem that version 4.1.0 requires the second node to be started
>>before acquired the delegate IP
>> After starting pgpool on the node 1 I?m seeing that watchdog
>>successfully acquired the delegate IP on node 0
>> 
>> 2020-02-11 10:45:26: pid 9928: LOG:  watchdog: escalation started
>> 2020-02-11 10:45:33: pid 9928: LOG:  successfully acquired the delegate
>>IP:"10.11.0.204"
>> 2020-02-11 10:45:33: pid 9928: DETAIL:  'if_up_cmd' returned with
>>success
>> 2020-02-11 10:45:33: pid 9577: LOG:  watchdog escalation process with
>>pid: 9928 exit with SUCCESS.
>> 
>> On previous versions watchdog would always acquire the delegate IP
>>without the second node being started.
>> 
>> 
>> From: "pgpool-general-bounces at pgpool.net"
>><pgpool-general-bounces at pgpool.net> on behalf of Wolfgang Schwurack
>><wolf at uen.org>
>> Date: Tuesday, February 11, 2020 at 10:22 AM
>> To: "pgpool-general at pgpool.net" <pgpool-general at pgpool.net>
>> Subject: [pgpool-general: 6863] watchdog fails to start pgpool-4.1.0
>> 
>> I?m trying to get watchdog to start using pgpool-4.1.0 but fails to
>>start. I have been using pgpool-4.0.5 with watchdog no issues.
>> Has something changed in version 4.1.0 for watchdog?
>> Hosts  - Ubuntu 18.0.4
>> PostgreSQL 11
>> 
>> I?ve been using pgpool for a long time on each new release I have
>>always just done ./configure, make, make install
>> 
>> This is my start command
>> 
>> /usr/local/bin/pgpool -n -D -f /usr/local/etc/pgpool.conf >
>>/var/log/pgpool/pgpool.log 2>&1 &
>> In pgpool.log it would always show if acquired the delegate ip
>> Version 4.0.5 start up watchdog
>> 
>> 2020-02-11 10:13:05: pid 2195: LOG:  pgpool-II successfully started.
>>version 4.0.5 (torokiboshi)
>> 
>> 2020-02-11 10:13:05: pid 2195: LOG:  node status[0]: 1
>> 
>> 2020-02-11 10:13:05: pid 2195: LOG:  node status[1]: 2
>> 
>> 2020-02-11 10:13:06: pid 2228: LOG:  creating socket for sending
>>heartbeat
>> 
>> 2020-02-11 10:13:06: pid 2228: DETAIL:  bind send socket to device: eth0
>> 
>> 2020-02-11 10:13:06: pid 2228: LOG:  set SO_REUSEPORT option to the
>>socket
>> 
>> 2020-02-11 10:13:06: pid 2228: LOG:  creating socket for sending
>>heartbeat
>> 
>> 2020-02-11 10:13:06: pid 2228: DETAIL:  set SO_REUSEPORT
>> 
>> 2020-02-11 10:13:06: pid 2227: LOG:  createing watchdog heartbeat
>>receive socket.
>> 
>> 2020-02-11 10:13:06: pid 2227: DETAIL:  bind receive socket to device:
>>"eth0"
>> 
>> 2020-02-11 10:13:06: pid 2227: LOG:  set SO_REUSEPORT option to the
>>socket
>> 
>> 2020-02-11 10:13:06: pid 2227: LOG:  creating watchdog heartbeat
>>receive socket.
>> 
>> 2020-02-11 10:13:06: pid 2227: DETAIL:  set SO_REUSEPORT
>> 
>> 2020-02-11 10:13:12: pid 2200: LOG:  successfully acquired the delegate
>>IP:"10.11.0.204"
>> 
>> 2020-02-11 10:13:12: pid 2200: DETAIL:  'if_up_cmd' returned with
>>success
>> 
>> 2020-02-11 10:13:12: pid 2197: LOG:  watchdog escalation process with
>>pid: 2200 exit with SUCCESS.
>> 
>> Version 4.1.0 fails to start watchdog
>> 
>> 2020-02-11 10:15:54: pid 8392: LOG:  pgpool-II successfully started.
>>version 4.1.0 (karasukiboshi)
>> 
>> 2020-02-11 10:15:54: pid 8392: LOG:  node status[0]: 1
>> 
>> 2020-02-11 10:15:54: pid 8392: LOG:  node status[1]: 2
>> 
>> 2020-02-11 10:15:55: pid 8425: LOG:  creating socket for sending
>>heartbeat
>> 
>> 2020-02-11 10:15:55: pid 8425: DETAIL:  bind send socket to device: eth0
>> 
>> 2020-02-11 10:15:55: pid 8425: LOG:  set SO_REUSEPORT option to the
>>socket
>> 
>> 2020-02-11 10:15:55: pid 8425: LOG:  creating socket for sending
>>heartbeat
>> 
>> 2020-02-11 10:15:55: pid 8425: DETAIL:  set SO_REUSEPORT
>> 
>> 2020-02-11 10:15:55: pid 8424: LOG:  createing watchdog heartbeat
>>receive socket.
>> 
>> 2020-02-11 10:15:55: pid 8424: DETAIL:  bind receive socket to device:
>>"eth0"
>> 
>> 2020-02-11 10:15:55: pid 8424: LOG:  set SO_REUSEPORT option to the
>>socket
>> 
>> 2020-02-11 10:15:55: pid 8424: LOG:  creating watchdog heartbeat
>>receive socket.
>> 
>> 2020-02-11 10:15:55: pid 8424: DETAIL:  set SO_REUSEPORT
>> 
>> 
>> Wolfgang Schwurack
>> Database/System Administrator
>> Utah Education Network
>> 801-587-9444
>> wolf at uen.org<mailto:wolf at uen.org>
>> 