[pgpool-general: 6884] Re: watchdog fails to start pgpool-4.1.0

Tue Feb 18 16:13:10 JST 2020

Hi Usama,

Any opinion on this?

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

> I turned on enable_consensus_with_half_votes which I’m getting the
> acquired delegate IP on node 0. But now when I start pgpool on node 1
> getting this in the log file which is repeating - see below.  When I check
> which node has the virtual IP is show that node 0 does which is the master
> node. 
> 
> 2020-02-12 08:11:52: pid 29493: LOG:  watchdog node state changed from
> [INITIALIZING] to [MASTER]
> 2020-02-12 08:11:52: pid 29493: LOG:  I am announcing my self as
> master/coordinator watchdog node
> 2020-02-12 08:11:52: pid 29493: LOG:  remote node "" decided it is the
> true master
> 2020-02-12 08:11:52: pid 29493: DETAIL:  re-initializing the local
> watchdog cluster state because of split-brain
> 2020-02-12 08:11:52: pid 29493: LOG:  watchdog node state changed from
> [MASTER] to [JOINING]
> 2020-02-12 08:11:53: pid 29493: LOG:  new watchdog node connection is
> received from "10.11.0.202:12399"
> 2020-02-12 08:11:56: pid 29493: LOG:  watchdog node state changed from
> [JOINING] to [INITIALIZING]
> 2020-02-12 08:11:57: pid 29493: LOG:  I am the only alive node in the
> watchdog cluster
> 2020-02-12 08:11:57: pid 29493: HINT:  skipping stand for coordinator state
> 
> My environment
> 2 pgpool hosts on Ubuntu 18
> 2 postgresql hosts on Ubuntu 18 postgreSQL 11
> 
> 
> Wolfgang Schwurack
> Database/System Administrator
> Utah Education Network
> 801-587-9444
> Wolf at uen.org
> 
> 
> 
> 
> 
> On 2/11/20, 3:50 PM, "Tatsuo Ishii" <ishii at sraoss.co.jp> wrote:
> 
>>Have you turned on enable_consensus_with_half_votes?
>>From 4.1 you need to turn on this if you use even number of Pgpool-II
>>nodes.
>>It's documented in the migration section in the doc:
>>https://www.pgpool.net/docs/latest/en/html/release-4-1-0.html
>>
>>Best regards,
>>--
>>Tatsuo Ishii
>>SRA OSS, Inc. Japan
>>English: http://www.sraoss.co.jp/index_en.php
>>Japanese:http://www.sraoss.co.jp
>>
>>From: Wolf Schwurack <wolf at uen.org>
>>Subject: [pgpool-general: 6865] Re: watchdog fails to start pgpool-4.1.0
>>Date: Tue, 11 Feb 2020 18:10:25 +0000
>>Message-ID: <56216C05-00F8-4C10-A32A-C793411C7891 at umail.utah.edu>
>>
>>> After doing some more testing on version 4.1.0 I have notice that if
>>>node 0 fails, node 1 never acquires the delegate IP. I compared this to
>>>version 4.0.5 which when node 0 fails, node 1 acquires the delegate IP
>>> 
>>> Wolfgang Schwurack
>>> Database/System Administrator
>>> Utah Education Network
>>> 801-587-9444
>>> wolf at uen.org<mailto:wolf at uen.org>
>>> 
>>> From: "pgpool-general-bounces at pgpool.net"
>>><pgpool-general-bounces at pgpool.net> on behalf of Wolfgang Schwurack
>>><wolf at uen.org>
>>> Date: Tuesday, February 11, 2020 at 10:54 AM
>>> To: "pgpool-general at pgpool.net" <pgpool-general at pgpool.net>
>>> Subject: [pgpool-general: 6864] Re: watchdog fails to start pgpool-4.1.0
>>> 
>>> It seem that version 4.1.0 requires the second node to be started
>>>before acquired the delegate IP
>>> After starting pgpool on the node 1 I?m seeing that watchdog
>>>successfully acquired the delegate IP on node 0
>>> 
>>> 2020-02-11 10:45:26: pid 9928: LOG:  watchdog: escalation started
>>> 2020-02-11 10:45:33: pid 9928: LOG:  successfully acquired the delegate
>>>IP:"10.11.0.204"
>>> 2020-02-11 10:45:33: pid 9928: DETAIL:  'if_up_cmd' returned with
>>>success
>>> 2020-02-11 10:45:33: pid 9577: LOG:  watchdog escalation process with
>>>pid: 9928 exit with SUCCESS.
>>> 
>>> On previous versions watchdog would always acquire the delegate IP
>>>without the second node being started.
>>> 
>>> 
>>> From: "pgpool-general-bounces at pgpool.net"
>>><pgpool-general-bounces at pgpool.net> on behalf of Wolfgang Schwurack
>>><wolf at uen.org>
>>> Date: Tuesday, February 11, 2020 at 10:22 AM
>>> To: "pgpool-general at pgpool.net" <pgpool-general at pgpool.net>
>>> Subject: [pgpool-general: 6863] watchdog fails to start pgpool-4.1.0
>>> 
>>> I?m trying to get watchdog to start using pgpool-4.1.0 but fails to
>>>start. I have been using pgpool-4.0.5 with watchdog no issues.
>>> Has something changed in version 4.1.0 for watchdog?
>>> Hosts  - Ubuntu 18.0.4
>>> PostgreSQL 11
>>> 
>>> I?ve been using pgpool for a long time on each new release I have
>>>always just done ./configure, make, make install
>>> 
>>> This is my start command
>>> 
>>> /usr/local/bin/pgpool -n -D -f /usr/local/etc/pgpool.conf >
>>>/var/log/pgpool/pgpool.log 2>&1 &
>>> In pgpool.log it would always show if acquired the delegate ip
>>> Version 4.0.5 start up watchdog
>>> 
>>> 2020-02-11 10:13:05: pid 2195: LOG:  pgpool-II successfully started.
>>>version 4.0.5 (torokiboshi)
>>> 
>>> 2020-02-11 10:13:05: pid 2195: LOG:  node status[0]: 1
>>> 
>>> 2020-02-11 10:13:05: pid 2195: LOG:  node status[1]: 2
>>> 
>>> 2020-02-11 10:13:06: pid 2228: LOG:  creating socket for sending
>>>heartbeat
>>> 
>>> 2020-02-11 10:13:06: pid 2228: DETAIL:  bind send socket to device: eth0
>>> 
>>> 2020-02-11 10:13:06: pid 2228: LOG:  set SO_REUSEPORT option to the
>>>socket
>>> 
>>> 2020-02-11 10:13:06: pid 2228: LOG:  creating socket for sending
>>>heartbeat
>>> 
>>> 2020-02-11 10:13:06: pid 2228: DETAIL:  set SO_REUSEPORT
>>> 
>>> 2020-02-11 10:13:06: pid 2227: LOG:  createing watchdog heartbeat
>>>receive socket.
>>> 
>>> 2020-02-11 10:13:06: pid 2227: DETAIL:  bind receive socket to device:
>>>"eth0"
>>> 
>>> 2020-02-11 10:13:06: pid 2227: LOG:  set SO_REUSEPORT option to the
>>>socket
>>> 
>>> 2020-02-11 10:13:06: pid 2227: LOG:  creating watchdog heartbeat
>>>receive socket.
>>> 
>>> 2020-02-11 10:13:06: pid 2227: DETAIL:  set SO_REUSEPORT
>>> 
>>> 2020-02-11 10:13:12: pid 2200: LOG:  successfully acquired the delegate
>>>IP:"10.11.0.204"
>>> 
>>> 2020-02-11 10:13:12: pid 2200: DETAIL:  'if_up_cmd' returned with
>>>success
>>> 
>>> 2020-02-11 10:13:12: pid 2197: LOG:  watchdog escalation process with
>>>pid: 2200 exit with SUCCESS.
>>> 
>>> Version 4.1.0 fails to start watchdog
>>> 
>>> 2020-02-11 10:15:54: pid 8392: LOG:  pgpool-II successfully started.
>>>version 4.1.0 (karasukiboshi)
>>> 
>>> 2020-02-11 10:15:54: pid 8392: LOG:  node status[0]: 1
>>> 
>>> 2020-02-11 10:15:54: pid 8392: LOG:  node status[1]: 2
>>> 
>>> 2020-02-11 10:15:55: pid 8425: LOG:  creating socket for sending
>>>heartbeat
>>> 
>>> 2020-02-11 10:15:55: pid 8425: DETAIL:  bind send socket to device: eth0
>>> 
>>> 2020-02-11 10:15:55: pid 8425: LOG:  set SO_REUSEPORT option to the
>>>socket
>>> 
>>> 2020-02-11 10:15:55: pid 8425: LOG:  creating socket for sending
>>>heartbeat
>>> 
>>> 2020-02-11 10:15:55: pid 8425: DETAIL:  set SO_REUSEPORT
>>> 
>>> 2020-02-11 10:15:55: pid 8424: LOG:  createing watchdog heartbeat
>>>receive socket.
>>> 
>>> 2020-02-11 10:15:55: pid 8424: DETAIL:  bind receive socket to device:
>>>"eth0"
>>> 
>>> 2020-02-11 10:15:55: pid 8424: LOG:  set SO_REUSEPORT option to the
>>>socket
>>> 
>>> 2020-02-11 10:15:55: pid 8424: LOG:  creating watchdog heartbeat
>>>receive socket.
>>> 
>>> 2020-02-11 10:15:55: pid 8424: DETAIL:  set SO_REUSEPORT
>>> 
>>> 
>>> Wolfgang Schwurack
>>> Database/System Administrator
>>> Utah Education Network
>>> 801-587-9444
>>> wolf at uen.org<mailto:wolf at uen.org>
>>> 
> 