[pgpool-general: 4357] Re: Watchdog on Debian Jessie platform with PGpool version(3.3.4, 3.4, 3.5) wont run when defining virtual IP?

[Michael Akkermans]

All,
  
 Very very  sorry!! Im ashamed. My mail-client rendered my email horribly. 
Wil resend my message with other mailclient.
  

----------------------------------------
 From: "Michael Akkermans" <info at michael-akkermans.nl>
Sent: Saturday, January 23, 2016 6:32 PM
To: pgpool-general at pgpool.net
Subject: [pgpool-general: 4356] Watchdog on Debian Jessie platform with 
PGpool version(3.3.4, 3.4, 3.5) wont run when defining virtual IP?   
Hi, English is not my native language; please excuse typing errors. === 
Summary === Goal: Use Watchdog with a virtual-IP between 2 PGpool 
instances. Reason i ask my question in General, is cause im sure im missing 
something very obvious. Problem description: Watchdog and PGpool wont run 
after configuring (any) virtual IP. I checked the following sources: - 
Mailinglist General archives PGpool - Web search "setup and Watchdog 
topics" (Startpage.com) - Stack Overflow, Server Fault searches - Manual: 
http://www.pgpool.net/docs/latest/pgpool-en.html - FAQ: 
http://www.pgpool.net/mediawiki/index.php/FAQ - Read this before sending 
this email: http://www.catb.org/esr/faqs/smart-questions.html - Checked 
sourcecode from Debian-os-repo(3.3.4) and PGpool-git-repo(3.5) NOTE: Im not 
a code guru, but i can read it and somewhat understand what it should do. I 
created a labsetup: - Labsetup (2x PGpool VMs, 2x PG-9.4 VM's) === Lab 
Setup === pgpool01 -- VM(proxmox), OS(Debian Jessie 8.2 - 
3.16.0-4-686-pae), Hostname(pgpool01.lab.local), IP(10.100.4.81) pgpool02 
-- VM(proxmox), OS(Debian Jessie 8.2 - 3.16.0-4-686-pae), 
Hostname(pgpool02.lab.local), IP(10.100.4.82) pgsql01 -- VM(proxmox), 
OS(Debian Jessie 8.2 - 3.16.0-4-686-pae), Hostname(pgsql01.lab.local), 
IP(10.100.4.91) pgsql02 -- VM(proxmox), OS(Debian Jessie 8.2 - 
3.16.0-4-686-pae), Hostname(pgsql01.lab.local), IP(10.100.4.92) 
********************************************* *** 
pgpool01(/etc/pgpool.conf) highlights *** 
********************************************* listen_addresses = '*' port = 
5432 pcp_port = 9898 -- backend_hostname0 = '10.100.4.91' backend_port0 = 
5432 backend_weight0 = 1 backend_data_directory0 = 
'/var/lib/postgresql/9.4/main' backend_flag0 = 'DISALLOW_TO_FAILOVER' -- 
backend_hostname1 = '10.100.4.92' backend_port1 = 5432 backend_weight1 = 1 
backend_data_directory1 = '/var/lib/postgresql/9.4/main' backend_flag1 = 
'DISALLOW_TO_FAILOVER' -- log_destination = 'stderr' debug_level = 1 -- 
connection_cache = on replication_mode = off load_balance_mode = on 
master_slave_mode = off parallel_mode = off -- use_watchdog = on 
trusted_servers = '' wd_hostname = '10.100.4.81' wd_port = 9000 delegate_IP 
= '10.100.4.85' if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 
255.255.255.0' if_down_cmd = 'ifconfig eth0:0 down' arping_cmd = 'arping -U 
$_IP_$ -w 1' clear_memqcache_on_escalation = on -- wd_lifecheck_method = 
'heartbeat' wd_heartbeat_port = 9694 heartbeat_destination0 = '10.100.4.82' 
heartbeat_destination_port0 = 9694 heartbeat_device0 = 'eth0' 
********************************************* *** 
pgpool02(/etc/pgpool.conf) highlights *** 
********************************************* Has a similar setup as 
pgpool01 with subtle differences listed below: wd_hostname = '10.100.4.82' 
heartbeat_destination0 = '10.100.4.81' ************************ *** 
pgsql01, pgsql02 *** ************************ Both clusters are online and 
respond to queries from PGpool. """ psql -h 10.100.4.81 -U postgres -d 
postgres -c "SELECT * FROM t1" col1 | col2 ------+------------- 1 | a (1 
row) """ NOTE: I did not configured hot_standby, just 2 standalone clusters 
with the same t1 table + tuples. First i want Watchdog operational so i can 
test some PGpool failover scenario's. === Behaviour in Lab Setup === With 
the settings as described in this email this is the output of pgpool, """ 
2016-01-23 12:38:58 LOG: pid 15187: wd_chk_setuid: ifup[/sbin/ifconfig] 
doesn't have setuid bit 2016-01-23 12:39:01 LOG: pid 15187: 
wd_create_send_socket: connect() reports failure (No route to host). You 
can safely ignore this while starting up. 2016-01-23 12:39:01 LOG: pid 
15187: send_packet_4_nodes: packet for 10.100.4.82:9000 is canceled 
2016-01-23 12:39:04 DEBUG: pid 15187: exec_ping: failed to ping 
10.100.4.85: exit code 1 2016-01-23 12:39:04 LOG: pid 15187: wd_escalation: 
escalating to master pgpool 2016-01-23 12:39:04 DEBUG: pid 15187: 
exec_ifconfig: 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0' 
succeeded """ This is the output of netstat, """ netstat -tulpen | grep 
'5432\|9000\|9694' tcp 0 0 0.0.0.0:5432 LISTEN 15187/pgpool """ For some 
(to me unknown) reason Watchdog and Heartbeat are not running. The 
subinterface with virtual-IP 10.100.4.85 is up. Also the pid files are 
there in /var/run/postgresql/. Output (snippets) with strace (with settings 
as described in this email) """ access("/etc/ld.so.nohwcap", F_OK) = -1 
ENOENT (No such file or directory) write(2, "2016-01-23 17:50:38 LOG: pid 
1"..., 992016-01-23 17:50:38 LOG: pid 14996: send_packet_4_nodes: packet 
for 10.100.4.82:9000 is canceled write(2, "2016-01-23 17:50:41 LOG: pid 
1"..., 812016-01-23 17:50:41 LOG: pid 14996: wd_escalation: escalating to 
master pgpool """ === Findings === - The message "send_packet_4_nodes is 
cancelled" means the other host is not responding. But even when i check 
the other host and make sure port 9000 is up and listening for incomming 
packets, the message wont change. I should not prevent me from brining the 
master up? - The output states "wd_escalation: escalating to master 
pgpool", but why does Watchdog stay down? - Tried with PGpool version 3.5, 
version 3.4 stable and version 3.3.4, same problem, what am i missing here? 
- Searched on keywords from the logfiles with no luck. Many happy-flow 
examples, not so much Watchdog troubleshoot examples. - pcp_* commands wont 
work in this setup. === Questions === - Would someone provide a pointer? - 
What is my example missing? - What site should I have checked? - How can i 
get Watchdog to work? With kind regards, Michael Akkermans

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20160123/f25b2c04/attachment.htm>