[pgpool-general: 6900] Re: watchdog fails to start pgpool-4.1.0

Wed Mar 4 18:20:12 JST 2020

Hi Wolfgang,

Sorry for the late reply. I just realized the email was sitting in my
drafts folder and was never sent.

Is it possible if you can share the Pppool log files for both nodes
preferably with the debug enabled?
Meanwhile, I am also trying to reproduce the scenario locally.

Thanks
Best regards
Muhammad Usama

On Tue, Feb 18, 2020 at 12:13 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:

> Hi Usama,
>
> Any opinion on this?
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp
>
> > I turned on enable_consensus_with_half_votes which I’m getting the
> > acquired delegate IP on node 0. But now when I start pgpool on node 1
> > getting this in the log file which is repeating - see below.  When I
> check
> > which node has the virtual IP is show that node 0 does which is the
> master
> > node.
> >
> > 2020-02-12 08:11:52: pid 29493: LOG:  watchdog node state changed from
> > [INITIALIZING] to [MASTER]
> > 2020-02-12 08:11:52: pid 29493: LOG:  I am announcing my self as
> > master/coordinator watchdog node
> > 2020-02-12 08:11:52: pid 29493: LOG:  remote node "" decided it is the
> > true master
> > 2020-02-12 08:11:52: pid 29493: DETAIL:  re-initializing the local
> > watchdog cluster state because of split-brain
> > 2020-02-12 08:11:52: pid 29493: LOG:  watchdog node state changed from
> > [MASTER] to [JOINING]
> > 2020-02-12 08:11:53: pid 29493: LOG:  new watchdog node connection is
> > received from "10.11.0.202:12399"
> > 2020-02-12 08:11:56: pid 29493: LOG:  watchdog node state changed from
> > [JOINING] to [INITIALIZING]
> > 2020-02-12 08:11:57: pid 29493: LOG:  I am the only alive node in the
> > watchdog cluster
> > 2020-02-12 08:11:57: pid 29493: HINT:  skipping stand for coordinator
> state
> >
> > My environment
> > 2 pgpool hosts on Ubuntu 18
> > 2 postgresql hosts on Ubuntu 18 postgreSQL 11
> >
> >
> > Wolfgang Schwurack
> > Database/System Administrator
> > Utah Education Network
> > 801-587-9444
> > Wolf at uen.org
> >
> >
> >
> >
> >
> > On 2/11/20, 3:50 PM, "Tatsuo Ishii" <ishii at sraoss.co.jp> wrote:
> >
> >>Have you turned on enable_consensus_with_half_votes?
> >>From 4.1 you need to turn on this if you use even number of Pgpool-II
> >>nodes.
> >>It's documented in the migration section in the doc:
> >>https://www.pgpool.net/docs/latest/en/html/release-4-1-0.html
> >>
> >>Best regards,
> >>--
> >>Tatsuo Ishii
> >>SRA OSS, Inc. Japan
> >>English: http://www.sraoss.co.jp/index_en.php
> >>Japanese:http://www.sraoss.co.jp
> >>
> >>From: Wolf Schwurack <wolf at uen.org>
> >>Subject: [pgpool-general: 6865] Re: watchdog fails to start pgpool-4.1.0
> >>Date: Tue, 11 Feb 2020 18:10:25 +0000
> >>Message-ID: <56216C05-00F8-4C10-A32A-C793411C7891 at umail.utah.edu>
> >>
> >>> After doing some more testing on version 4.1.0 I have notice that if
> >>>node 0 fails, node 1 never acquires the delegate IP. I compared this to
> >>>version 4.0.5 which when node 0 fails, node 1 acquires the delegate IP
> >>>
> >>> Wolfgang Schwurack
> >>> Database/System Administrator
> >>> Utah Education Network
> >>> 801-587-9444
> >>> wolf at uen.org<mailto:wolf at uen.org>
> >>>
> >>> From: "pgpool-general-bounces at pgpool.net"
> >>><pgpool-general-bounces at pgpool.net> on behalf of Wolfgang Schwurack
> >>><wolf at uen.org>
> >>> Date: Tuesday, February 11, 2020 at 10:54 AM
> >>> To: "pgpool-general at pgpool.net" <pgpool-general at pgpool.net>
> >>> Subject: [pgpool-general: 6864] Re: watchdog fails to start
> pgpool-4.1.0
> >>>
> >>> It seem that version 4.1.0 requires the second node to be started
> >>>before acquired the delegate IP
> >>> After starting pgpool on the node 1 I?m seeing that watchdog
> >>>successfully acquired the delegate IP on node 0
> >>>
> >>> 2020-02-11 10:45:26: pid 9928: LOG:  watchdog: escalation started
> >>> 2020-02-11 10:45:33: pid 9928: LOG:  successfully acquired the delegate
> >>>IP:"10.11.0.204"
> >>> 2020-02-11 10:45:33: pid 9928: DETAIL:  'if_up_cmd' returned with
> >>>success
> >>> 2020-02-11 10:45:33: pid 9577: LOG:  watchdog escalation process with
> >>>pid: 9928 exit with SUCCESS.
> >>>
> >>> On previous versions watchdog would always acquire the delegate IP
> >>>without the second node being started.
> >>>
> >>>
> >>> From: "pgpool-general-bounces at pgpool.net"
> >>><pgpool-general-bounces at pgpool.net> on behalf of Wolfgang Schwurack
> >>><wolf at uen.org>
> >>> Date: Tuesday, February 11, 2020 at 10:22 AM
> >>> To: "pgpool-general at pgpool.net" <pgpool-general at pgpool.net>
> >>> Subject: [pgpool-general: 6863] watchdog fails to start pgpool-4.1.0
> >>>
> >>> I?m trying to get watchdog to start using pgpool-4.1.0 but fails to
> >>>start. I have been using pgpool-4.0.5 with watchdog no issues.
> >>> Has something changed in version 4.1.0 for watchdog?
> >>> Hosts  - Ubuntu 18.0.4
> >>> PostgreSQL 11
> >>>
> >>> I?ve been using pgpool for a long time on each new release I have
> >>>always just done ./configure, make, make install
> >>>
> >>> This is my start command
> >>>
> >>> /usr/local/bin/pgpool -n -D -f /usr/local/etc/pgpool.conf >
> >>>/var/log/pgpool/pgpool.log 2>&1 &
> >>> In pgpool.log it would always show if acquired the delegate ip
> >>> Version 4.0.5 start up watchdog
> >>>
> >>> 2020-02-11 10:13:05: pid 2195: LOG:  pgpool-II successfully started.
> >>>version 4.0.5 (torokiboshi)
> >>>
> >>> 2020-02-11 10:13:05: pid 2195: LOG:  node status[0]: 1
> >>>
> >>> 2020-02-11 10:13:05: pid 2195: LOG:  node status[1]: 2
> >>>
> >>> 2020-02-11 10:13:06: pid 2228: LOG:  creating socket for sending
> >>>heartbeat
> >>>
> >>> 2020-02-11 10:13:06: pid 2228: DETAIL:  bind send socket to device:
> eth0
> >>>
> >>> 2020-02-11 10:13:06: pid 2228: LOG:  set SO_REUSEPORT option to the
> >>>socket
> >>>
> >>> 2020-02-11 10:13:06: pid 2228: LOG:  creating socket for sending
> >>>heartbeat
> >>>
> >>> 2020-02-11 10:13:06: pid 2228: DETAIL:  set SO_REUSEPORT
> >>>
> >>> 2020-02-11 10:13:06: pid 2227: LOG:  createing watchdog heartbeat
> >>>receive socket.
> >>>
> >>> 2020-02-11 10:13:06: pid 2227: DETAIL:  bind receive socket to device:
> >>>"eth0"
> >>>
> >>> 2020-02-11 10:13:06: pid 2227: LOG:  set SO_REUSEPORT option to the
> >>>socket
> >>>
> >>> 2020-02-11 10:13:06: pid 2227: LOG:  creating watchdog heartbeat
> >>>receive socket.
> >>>
> >>> 2020-02-11 10:13:06: pid 2227: DETAIL:  set SO_REUSEPORT
> >>>
> >>> 2020-02-11 10:13:12: pid 2200: LOG:  successfully acquired the delegate
> >>>IP:"10.11.0.204"
> >>>
> >>> 2020-02-11 10:13:12: pid 2200: DETAIL:  'if_up_cmd' returned with
> >>>success
> >>>
> >>> 2020-02-11 10:13:12: pid 2197: LOG:  watchdog escalation process with
> >>>pid: 2200 exit with SUCCESS.
> >>>
> >>> Version 4.1.0 fails to start watchdog
> >>>
> >>> 2020-02-11 10:15:54: pid 8392: LOG:  pgpool-II successfully started.
> >>>version 4.1.0 (karasukiboshi)
> >>>
> >>> 2020-02-11 10:15:54: pid 8392: LOG:  node status[0]: 1
> >>>
> >>> 2020-02-11 10:15:54: pid 8392: LOG:  node status[1]: 2
> >>>
> >>> 2020-02-11 10:15:55: pid 8425: LOG:  creating socket for sending
> >>>heartbeat
> >>>
> >>> 2020-02-11 10:15:55: pid 8425: DETAIL:  bind send socket to device:
> eth0
> >>>
> >>> 2020-02-11 10:15:55: pid 8425: LOG:  set SO_REUSEPORT option to the
> >>>socket
> >>>
> >>> 2020-02-11 10:15:55: pid 8425: LOG:  creating socket for sending
> >>>heartbeat
> >>>
> >>> 2020-02-11 10:15:55: pid 8425: DETAIL:  set SO_REUSEPORT
> >>>
> >>> 2020-02-11 10:15:55: pid 8424: LOG:  createing watchdog heartbeat
> >>>receive socket.
> >>>
> >>> 2020-02-11 10:15:55: pid 8424: DETAIL:  bind receive socket to device:
> >>>"eth0"
> >>>
> >>> 2020-02-11 10:15:55: pid 8424: LOG:  set SO_REUSEPORT option to the
> >>>socket
> >>>
> >>> 2020-02-11 10:15:55: pid 8424: LOG:  creating watchdog heartbeat
> >>>receive socket.
> >>>
> >>> 2020-02-11 10:15:55: pid 8424: DETAIL:  set SO_REUSEPORT
> >>>
> >>>
> >>> Wolfgang Schwurack
> >>> Database/System Administrator
> >>> Utah Education Network
> >>> 801-587-9444
> >>> wolf at uen.org<mailto:wolf at uen.org>
> >>>
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20200304/a79b1d71/attachment-0001.html>