[pgpool-general: 4356] Watchdog on Debian Jessie platform with PGpool version(3.3.4, 3.4, 3.5) wont run when defining virtual IP?

[Michael Akkermans]

Hi,   English is not my native language; please excuse typing errors.   === 
Summary ===  Goal: Use Watchdog with a virtual-IP between 2 PGpool 
instances. Reason i ask my question in General, is cause im sure im missing 
something very obvious.  Problem description: Watchdog and PGpool wont run 
after configuring (any) virtual IP.  I checked the following sources: - 
Mailinglist General archives PGpool - Web search "setup and Watchdog 
topics" (Startpage.com) - Stack Overflow, Server Fault searches - Manual: 
http://www.pgpool.net/docs/latest/pgpool-en.html - FAQ: 
http://www.pgpool.net/mediawiki/index.php/FAQ - Read this before sending 
this email: http://www.catb.org/esr/faqs/smart-questions.html - Checked 
sourcecode from Debian-os-repo(3.3.4) and PGpool-git-repo(3.5) NOTE: Im not 
a code guru, but i can read it and somewhat understand what it should do.  
I created a labsetup:  - Labsetup (2x PGpool VMs, 2x PG-9.4 VM's)   === Lab 
Setup === pgpool01 -- VM(proxmox), OS(Debian Jessie 8.2 - 
3.16.0-4-686-pae), Hostname(pgpool01.lab.local), IP(10.100.4.81) pgpool02 
-- VM(proxmox), OS(Debian Jessie 8.2 - 3.16.0-4-686-pae), 
Hostname(pgpool02.lab.local), IP(10.100.4.82) pgsql01 -- VM(proxmox), 
OS(Debian Jessie 8.2 - 3.16.0-4-686-pae), Hostname(pgsql01.lab.local), 
IP(10.100.4.91) pgsql02 -- VM(proxmox), OS(Debian Jessie 8.2 - 
3.16.0-4-686-pae), Hostname(pgsql01.lab.local), IP(10.100.4.92)  
********************************************* *** 
pgpool01(/etc/pgpool.conf) highlights *** 
********************************************* listen_addresses = '*' port = 
5432 pcp_port = 9898 -- backend_hostname0 = '10.100.4.91' backend_port0 = 
5432 backend_weight0 = 1 backend_data_directory0 = 
'/var/lib/postgresql/9.4/main' backend_flag0 = 'DISALLOW_TO_FAILOVER' -- 
backend_hostname1 = '10.100.4.92' backend_port1 = 5432 backend_weight1 = 1 
backend_data_directory1 = '/var/lib/postgresql/9.4/main' backend_flag1 = 
'DISALLOW_TO_FAILOVER' -- log_destination = 'stderr' debug_level = 1 -- 
connection_cache = on replication_mode = off load_balance_mode = on 
master_slave_mode = off parallel_mode = off -- use_watchdog = on 
trusted_servers = '' wd_hostname = '10.100.4.81' wd_port = 9000 delegate_IP 
= '10.100.4.85' if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 
255.255.255.0' if_down_cmd = 'ifconfig eth0:0 down' arping_cmd = 'arping -U 
$_IP_$ -w 1' clear_memqcache_on_escalation = on -- wd_lifecheck_method = 
'heartbeat' wd_heartbeat_port = 9694 heartbeat_destination0 = '10.100.4.82' 
heartbeat_destination_port0 = 9694 heartbeat_device0 = 'eth0'   
********************************************* *** 
pgpool02(/etc/pgpool.conf) highlights *** 
********************************************* Has a similar setup as 
pgpool01 with subtle differences listed below: wd_hostname = '10.100.4.82' 
heartbeat_destination0 = '10.100.4.81'  ************************ *** 
pgsql01, pgsql02 *** ************************ Both clusters are online and 
respond to queries from PGpool.  """ psql -h 10.100.4.81 -U postgres -d 
postgres -c "SELECT * FROM t1"  col1 |    col2      ------+-------------    
 1 | a (1 row)  """ NOTE: I did not configured hot_standby, just 2 
standalone clusters with the same t1 table + tuples. First i want Watchdog 
operational so i can test some PGpool failover scenario's.   === Behaviour 
in Lab Setup === With the settings as described in this email this is the 
output of pgpool, """ 2016-01-23 12:38:58 LOG:   pid 15187: wd_chk_setuid: 
ifup[/sbin/ifconfig] doesn't have setuid bit 2016-01-23 12:39:01 LOG:   pid 
15187: wd_create_send_socket: connect() reports failure (No route to host). 
You can safely ignore this while starting up. 2016-01-23 12:39:01 LOG:   
pid 15187: send_packet_4_nodes: packet for 10.100.4.82:9000 is canceled 
2016-01-23 12:39:04 DEBUG: pid 15187: exec_ping: failed to ping 
10.100.4.85: exit code 1 2016-01-23 12:39:04 LOG:   pid 15187: 
wd_escalation: escalating to master pgpool 2016-01-23 12:39:04 DEBUG: pid 
15187: exec_ifconfig: 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0' 
succeeded """  This is the output of netstat, """ netstat -tulpen | grep 
'5432\|9000\|9694' tcp        0      0 0.0.0.0:5432 LISTEN 15187/pgpool """ 
 For some (to me unknown) reason Watchdog and Heartbeat are not running. 
The subinterface with virtual-IP 10.100.4.85 is up. Also the pid files are 
there in /var/run/postgresql/.  Output (snippets) with strace (with 
settings as described in this email) """ access("/etc/ld.so.nohwcap", F_OK) 
     = -1 ENOENT (No such file or directory) write(2, "2016-01-23 17:50:38 
LOG:   pid 1"..., 992016-01-23 17:50:38 LOG:   pid 14996: 
send_packet_4_nodes: packet for 10.100.4.82:9000 is canceled write(2, 
"2016-01-23 17:50:41 LOG:   pid 1"..., 812016-01-23 17:50:41 LOG:   pid 
14996: wd_escalation: escalating to master pgpool """  === Findings === - 
The message "send_packet_4_nodes is cancelled" means the other host is not 
responding. But even when i check the other host and make sure port 9000 is 
up and listening for incomming packets, the message wont change. I should 
not prevent me from brining the master up? - The output states 
"wd_escalation: escalating to master pgpool", but why does Watchdog stay 
down? - Tried with PGpool version 3.5, version 3.4 stable and version 
3.3.4, same problem, what am i missing here? - Searched on keywords from 
the logfiles with no luck. Many happy-flow examples, not so much Watchdog 
troubleshoot examples. - pcp_* commands wont work in this setup.   === 
Questions === - Would someone provide a pointer? - What is my example 
missing? - What site should I have checked? - How can i get Watchdog to 
work?  With kind regards, Michael Akkermans
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20160123/04ec5854/attachment.htm>