[pgpool-general: 2500] Re: wd_escalation_command exit code

Sergey Arlashin sergeyarl.maillist at gmail.com
Wed Jan 29 15:26:00 JST 2014


Hi!

I'm testing this patch on a vagrant/virtualbox based VM. 

# uname -a
Linux lb-node1 3.2.0-55-generic #85-Ubuntu SMP Wed Oct 2 12:29:27 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/issue
Ubuntu 12.04.3 LTS \n \l

This is the output of ifconfig before starting pgpool:

eth0      Link encap:Ethernet  HWaddr 08:00:27:03:2b:89
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe03:2b89/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7198 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4853 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:553607 (553.6 KB)  TX bytes:722721 (722.7 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:70:46:a0
          inet addr:192.168.33.11  Bcast:192.168.33.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe70:46a0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:23682 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4876 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2551344 (2.5 MB)  TX bytes:646217 (646.2 KB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1636 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1636 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:109868 (109.8 KB)  TX bytes:109868 (109.8 KB)


/etc/pgpool2/pgpool.conf:
...
debug_level                   = 9
…
delegate_IP                   = '192.168.33.200'
...
ifconfig_path                 = '/sbin'
if_up_cmd                     = 'ifconfig eth1:0 $_IP_$ netmask 255.255.255.0'
if_down_cmd                   = 'ifconfig eth1:0 down'
...


Once I start pgpool I get the following ifconfig output


eth0      Link encap:Ethernet  HWaddr 08:00:27:03:2b:89
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe03:2b89/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7939 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5404 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:606232 (606.2 KB)  TX bytes:816924 (816.9 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:70:46:a0
          inet addr:192.168.33.11  Bcast:192.168.33.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe70:46a0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:25179 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5204 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2704567 (2.7 MB)  TX bytes:690834 (690.8 KB)

eth1:0    Link encap:Ethernet  HWaddr 08:00:27:70:46:a0
          inet addr:192.168.33.200  Bcast:192.168.33.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1745 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1745 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:117264 (117.2 KB)  TX bytes:117264 (117.2 KB)



# ping 192.168.33.200
PING 192.168.33.200 (192.168.33.200) 56(84) bytes of data.
64 bytes from 192.168.33.200: icmp_req=1 ttl=64 time=0.060 ms
^C
--- 192.168.33.200 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.060/0.060/0.060/0.000 ms


And these are some messages from pgpool.log:

pgpool[4152]: wd_chk_setuid all commands have setuid bit
pgpool[4152]: watchdog might call network commands which using setuid bit.
pgpool[4152]: exec_ping: failed to ping 192.168.33.200
pgpool[4152]: wd_escalation: escalating to master pgpool
pgpool[4152]: wd_IP_up: ifconfig up failed
pgpool[4152]: wd_declare: send the packet to declare the new master
pgpool[4152]: wd_escalation: escalated to master pgpool with some errors


When I stop pgpool I get the following messages in pgpool.log:

pgpool[4163]: wd_IP_down: not delegate IP holder
pgpool[4161]: hb_receiver child receives shutdown request signal 2
pgpool[4162]: hb_sender child receives shutdown request signal 2
pgpool[4152]: shmem_exit(0)





On Jan 29, 2014, at 6:42 AM, Yugo Nagata <nagata at sraoss.co.jp> wrote:

> On Tue, 28 Jan 2014 23:03:20 +0400
> Sergey Arlashin <sergeyarl.maillist at gmail.com> wrote:
> 
>> Hi!
>> This patch applied successfully. But now a new problem. When I start pgpool service I get a new interface eth0:0 with failover IP address assigned as expected. But when I stop pgpool service eth0:0 won't go down. It remains even after complete shutdown of pgpool.
> 
> Odd, I can't reproduce this. Are there any error message?
> What ifconfig command do you use?
> 
>> 
>> I tried 3.3.2 without this patch and everything worked well. 
>> 
>> 
>> 
>> 
>> On Jan 27, 2014, at 5:18 AM, Yugo Nagata <nagata at sraoss.co.jp> wrote:
>> 
>>> On Sat, 25 Jan 2014 15:31:44 +0400
>>> Sergey Arlashin <sergeyarl.maillist at gmail.com> wrote:
>>> 
>>>> 
>>>> On Jan 24, 2014, at 1:25 PM, Yugo Nagata <nagata at sraoss.co.jp> wrote:
>>>> 
>>>>> On Tue, 21 Jan 2014 15:24:02 +0400
>>>>> Sergey Arlashin <sergeyarl.maillist at gmail.com> wrote:
>>>>> 
>>>>>> Great! Now it is working!
>>>>>> 
>>>>>> pgpool[31903]: wd_escalation: escalation command failed. exit status: 1
>>>>>> 
>>>>>> Thank you!
>>>>>> 
>>>>>> Will this patch be included in 3.3.3 ?
>>>>>> 
>>>>>> Also, what about failed if_up_cmd and further pgpool behaviour (my second message in the thread.) ?
>>>>> 
>>>>> I attached the patch. Could you try this? In this fix, pgpool outputs a error 
>>>>> message for if_up_cmd failure. This patch should be applied after the previous
>>>>> patch. This fix will be included in 3.3.3.
>>>> 
>>>> 
>>>> Hi!
>>>> 
>>>> I tried to apply the patch against both 3.3.1 and 3.3.2
>>>> 
>>>> this is what I got:
>>> 
>>> Hmm.. Could you try the attached patch to 3.3.2? This includes allthe fix
>>> for escalation command and ifconfig errors.
>>> 
>>>> 
>>>> node1:~/pgpool-orig# patch -p1 < /root/op/esc.patch
>>>> 
>>>> patching file src/watchdog/wd_packet.c
>>>> Hunk #1 succeeded at 954 (offset 23 lines).
>>>> 
>>>> node1:~/pgpool-orig# patch -p1 < /root/op/ifup.patch
>>>> 
>>>> patching file src/watchdog/wd_if.c
>>>> Hunk #1 succeeded at 42 with fuzz 1 (offset 3 lines).
>>>> Hunk #2 succeeded at 62 (offset 3 lines).
>>>> Hunk #3 succeeded at 117 (offset 3 lines).
>>>> patching file src/watchdog/wd_packet.c
>>>> Hunk #1 succeeded at 654 (offset 23 lines).
>>>> Hunk #2 succeeded at 939 (offset 23 lines).
>>>> Hunk #3 FAILED at 932.
>>>> Hunk #4 succeeded at 976 (offset 18 lines).
>>>> 1 out of 4 hunks FAILED -- saving rejects to file src/watchdog/wd_packet.c.rej
>>>> 
>>>> 
>>>> src/watchdog/wd_packet.c.rej:
>>>> 
>>>> 
>>>> --- src/watchdog/wd_packet.c
>>>> +++ src/watchdog/wd_packet.c
>>>> @@ -932,22 +933,31 @@
>>>> 	/* execute escalation command */
>>>> 	if (strlen(pool_config->wd_escalation_command))
>>>> 	{
>>>> -		int r;
>>>> 		r = system(pool_config->wd_escalation_command);
>>>> 		if (WIFEXITED(r))
>>>> 		{
>>>> 			if (WEXITSTATUS(r) == EXIT_SUCCESS)
>>>> 				pool_log("wd_escalation: escalation command succeeded");
>>>> 			else
>>>> +			{
>>>> 				pool_error("wd_escalation: escalation command failed. exit status: %d", WEXITSTATUS(r));
>>>> +				has_error = true;
>>>> +			}
>>>> 		}
>>>> 		else
>>>> +		{
>>>> 			pool_error("wd_escalation: escalation command exit abnormally");
>>>> +			has_error = true;
>>>> +		}
>>>> 	}
>>>> 
>>>> 	/* interface up as delegate IP */
>>>> 	if (strlen(pool_config->delegate_IP) != 0)
>>>> -		wd_IP_up();
>>>> +	{
>>>> +		r = wd_IP_up();
>>>> +		if (r == WD_NG)
>>>> +			has_error = true;
>>>> +	}
>>>> 
>>>> 	/* set master status to the wd list */
>>>> 	wd_set_wd_list(pool_config->wd_hostname, pool_config->port,
>>>> 
>>>> 
>>>>> 
>>>>> In addition, I consider that pgpool shoud go to down status when if_up_cmd fails, 
>>>>> since this is worthless as a member of watchdog cluster. I'll make this fix for
>>>>> either 3.3.3 or 3.4.0.
>>>>> 
>>>>> 
>>>> 
>>>> Sounds reasonable. 
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> -- 
>>> Yugo Nagata <nagata at sraoss.co.jp>
>>> <escalation_error_all.patch>
>> 
> 
> 
> -- 
> Yugo Nagata <nagata at sraoss.co.jp>



More information about the pgpool-general mailing list