<div dir="ltr">The network config for one of the watch_dog node is :<div><br><div>1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000<br> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br> inet <a href="http://127.0.0.1/8">127.0.0.1/8</a> scope host lo<br> valid_lft forever preferred_lft forever<br> inet6 ::1/128 scope host<br> valid_lft forever preferred_lft forever<br>2: eno145: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000<br> link/ether 20:04:0f:f1:c2:48 brd ff:ff:ff:ff:ff:ff<br> altname enp24s0f0<br>3: eno146: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000<br> link/ether 20:04:0f:f1:c2:49 brd ff:ff:ff:ff:ff:ff<br> altname enp24s0f1<br>4: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000<br> link/ether 20:04:0f:f1:c2:4a brd ff:ff:ff:ff:ff:ff<br> altname enp25s0f0<br>5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000<br> link/ether 20:04:0f:f1:c2:4b brd ff:ff:ff:ff:ff:ff<br> altname enp25s0f1<br>6: ens10f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000<br> link/ether 00:1b:21:bd:58:0e brd ff:ff:ff:ff:ff:ff<br> altname enp177s0f0<br>7: ens10f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000<br> link/ether 00:1b:21:bd:58:0f brd ff:ff:ff:ff:ff:ff<br> altname enp177s0f1<br> inet <a href="http://192.168.1.121/24">192.168.1.121/24</a> brd 192.168.1.255 scope global noprefixroute ens10f1<br> valid_lft forever preferred_lft forever<br> inet6 fe80::1614:8143:8362:6611/64 scope link noprefixroute<br> valid_lft forever preferred_lft forever<br></div><div><br></div><div><br></div></div><div>in pgpoo.conf, after changing <span style="color:rgb(73,73,73);font-size:11pt">wd_monitoring_interfaces_list from emtpy ("") to </span>ens10f1, the problem seems is resovled.. I am wondering other network interface's status change will caused this issue.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Bo Peng <<a href="mailto:pengbo@sraoss.co.jp">pengbo@sraoss.co.jp</a>> 于2021年10月4日周一 上午11:45写道：<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
Sorry for the late response.<br>
<br>
> Hi,<br>
> <br>
> I have two PG severs and three watch_dog nodes to setup a PG HA<br>
> environment.<br>
> <br>
> - OS: Ubuntu 20.04<br>
> - PG version:12.8<br>
> - Pgpool version: 4.1.4<br>
> <br>
> <br>
> - PG -primary: 192.168.1.122<br>
> - PG -slave: 192.168.1.121<br>
> - Watch_dog node0: 192.168.1.122<br>
> - Watch_dog node1: 192.168.1.121<br>
> - Watch_dog node2: 192.168.1.101<br>
> <br>
> <br>
> the HA environment works fine while after 3-4 hours, two watch_dog nodes<br>
> downs, remaining only 1 watch_dog node (192.168.1.101) running. the<br>
> leader of watch_dog's log shows below error althought the network ip<br>
> 192.168.1.122 is alive.<br>
> <br>
> 2021-09-20 15:53:37: pid 1900172: WARNING: network IP is removed and<br>
> system has no IP is assigned<br>
> 2021-09-20 15:53:37: pid 1900172: DETAIL: changing the state to in network<br>
> trouble<br>
> 2021-09-20 15:53:37: pid 1900172: DEBUG: removing all watchdog nodes from<br>
> the standby list<br>
<br>
I think it may be caused by a temporary network problem.<br>
Does this issue occur every time? <br>
<br>
> 2021-09-20 15:53:37: pid 1900172: DETAIL: standby list contains 1 nodes<br>
> 2021-09-20 15:53:37: pid 1900172: DEBUG: Removing all failover objects<br>
> 2021-09-20 15:53:37: pid 1900172: LOG: watchdog node state changed from<br>
> [MASTER] to [IN NETWORK TROUBLE]<br>
> 2021-09-20 15:53:37: pid 1900172: DEBUG: STATE MACHINE INVOKED WITH EVENT<br>
> = STATE CHANGED Current State = IN NETWORK TROUBLE<br>
> 2021-09-20 15:53:37: pid 1900172: FATAL: system has lost the network<br>
> 2021-09-20 15:53:37: pid 1900172: LOG: Watchdog is shutting down<br>
> 2021-09-20 15:53:37: pid 1900172: DEBUG: sending packet, watchdog node:[<br>
> <a href="http://192.168.1.101:9999" rel="noreferrer" target="_blank">192.168.1.101:9999</a> Linux dell-PowerEdge-R740] command id:[1113]<br>
> type:[INFORM I AM GOING DOWN] state:[IN NETWORK TROUBLE]<br>
> 2021-09-20 15:53:37: pid 1900172: DEBUG: sending watchdog packet to<br>
> socket:8, type:[X], command ID:1113, data Length:0<br>
> 2021-09-20 15:53:37: pid 1933141: LOG: watchdog: de-escalation started<br>
> 2021-09-20 15:53:37: pid 1933141: DEBUG: watchdog exec interface up/down<br>
> command: '/usr/bin/sudo /sbin/ip addr del $_IP_$/24 dev ens2f0' succeeded<br>
> 2021-09-20 15:53:37: pid 1933141: LOG: successfully released the delegate<br>
> IP:"192.168.1.129"<br>
> 2021-09-20 15:53:37: pid 1933141: DETAIL: 'if_down_cmd' returned with<br>
> success<br>
> 2021-09-20 15:53:37: pid 1900168: DEBUG: reaper handler<br>
> 2021-09-20 15:53:37: pid 1900168: DEBUG: watchdog child process with pid:<br>
> 1900172 exit with FATAL ERROR. pgpool-II will be shutdown<br>
> 2021-09-20 15:53:37: pid 1900168: LOG: watchdog child process with pid:<br>
> 1900172 exits with status 768<br>
> 2021-09-20 15:53:37: pid 1900168: FATAL: watchdog child process exit with<br>
> fatal error. exiting pgpool-II<br>
> 2021-09-20 15:53:37: pid 1933148: LOG: setting the local watchdog node<br>
> name to "<a href="http://192.168.1.122:9999" rel="noreferrer" target="_blank">192.168.1.122:9999</a> Linux dell-PowerEdge-R740"<br>
> 2021-09-20 15:53:37: pid 1933148: LOG: watchdog cluster is configured with<br>
> 2 remote nodes<br>
> 2021-09-20 15:53:37: pid 1933148: LOG: watchdog remote node:0 on<br>
> <a href="http://192.168.1.121:9000" rel="noreferrer" target="_blank">192.168.1.121:9000</a><br>
> 2021-09-20 15:53:37: pid 1933148: LOG: watchdog remote node:1 on<br>
> <a href="http://192.168.1.101:9000" rel="noreferrer" target="_blank">192.168.1.101:9000</a><br>
> 2021-09-20 15:53:37: pid 1933148: LOG: interface monitoring is disabled in<br>
> watchdog<br>
> 2021-09-20 15:53:37: pid 1933148: INFO: IPC socket path:<br>
> "/tmp/.s.PGPOOLWD_CMD.9000"<br>
> 2021-09-20 15:53:37: pid 1933148: LOG: watchdog node state changed from<br>
> [DEAD] to [LOADING]<br>
> 2021-09-20 15:53:37: pid 1933148: DEBUG: STATE MACHINE INVOKED WITH EVENT<br>
> = STATE CHANGED Current State = LOADING<br>
> 2021-09-20 15:53:37: pid 1933148: DEBUG: error in outbound connection to<br>
> <a href="http://192.168.1.121:9000" rel="noreferrer" target="_blank">192.168.1.121:9000</a><br>
> 2021-09-20 15:53:37: pid 1933148: DETAIL: Connection refused<br>
> 2021-09-20 15:53:37: pid 1933148: LOG: new outbound connection to<br>
> <a href="http://192.168.1.101:9000" rel="noreferrer" target="_blank">192.168.1.101:9000</a><br>
> 2021-09-20 15:53:37: pid 1900189: DEBUG: lifecheck child receives shutdown<br>
> request signal 2, forwarding to all children<br>
> 2021-09-20 15:53:37: pid 1900189: DEBUG: lifecheck child receives fast<br>
> shutdown request<br>
> 2021-09-20 15:53:37: pid 1933148: LOG: Watchdog is shutting down<br>
> <br>
> Please refer the pgpool.conf and running log on each server. Any advice<br>
> to fix it?<br>
<br>
<br>
-- <br>
Bo Peng <<a href="mailto:pengbo@sraoss.co.jp" target="_blank">pengbo@sraoss.co.jp</a>><br>
SRA OSS, Inc. Japan<br>
<a href="http://www.sraoss.co.jp/" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/</a><br>
</blockquote></div>