[pgpool-general: 4740] 【Not execute failover with kamailio】
Noriyuki Hayashi
nhayashi at wats.gr.jp
Mon Jun 27 22:27:37 JST 2016
Hi
Environment are as follows.
1. CentOS-6.6 x86_64 E5-2670*2 384GB Memory.
2. postgresql-server-9.4.6
3. pgpool-II-pg94-3.5.2-1
4. kamailio-4.3.5-2.1
5. asterisk-11.21.2-1
I am testing failover by 1 server shutdown.
pgpool could not execute failover.sh with kamailio.
child process received shutdown request signal 3 could execute failover.
But child process received shutdown request signal 2 could not execute failover.
If I setup FloatingIP:9999 to kamailio.cfg,
I could not connect to
"psql-h FloatingIP -p 9999 -U postgres DB.
pgpool with asterisk has no problem.
If you give me any advise or hint with much appreciated.
I am looking forward from any hint and advise.
Kind regards,
Noriyuki Hayashi
Please see below logs.(There is no any error on the kamailio.)
failover without kamailio
-----------------------------------------------------------------
2016-06-27 21:40:54: pid 22119: LOG: read from socket failed, remote end closed the connection
2016-06-27 21:40:54: pid 22119: LOG: remote node "Linux_sp0.voiplus.net_9999" is shutting down
2016-06-27 21:40:54: pid 22119: LOG: read from socket failed, remote end closed the connection
2016-06-27 21:40:55: pid 22155: LOG: reading and processing packets
2016-06-27 21:40:55: pid 22155: DETAIL: postmaster on DB node 0 was shutdown by administrative command
2016-06-27 21:40:55: pid 22130: LOG: reading and processing packets
2016-06-27 21:40:55: pid 22130: DETAIL: postmaster on DB node 0 was shutdown by administrative command
2016-06-27 21:40:55: pid 22130: LOG: received degenerate backend request for node_id: 0 from pid [22130]
2016-06-27 21:40:55: pid 22119: LOG: new IPC connection received
2016-06-27 21:40:55: pid 22117: LOG: starting degeneration. shutdown host 192.168.192.136(5432)
2016-06-27 21:40:55: pid 22155: LOG: received degenerate backend request for node_id: 0 from pid [22155]
2016-06-27 21:40:55: pid 22119: LOG: new IPC connection received
2016-06-27 21:40:55: pid 22117: LOG: Restart all children
2016-06-27 21:40:55: pid 22130: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22126: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22133: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22124: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22128: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22137: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22135: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22150: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22141: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22144: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22148: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 26646: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22153: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22156: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 27242: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22152: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 26644: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22119: LOG: new IPC connection received
2016-06-27 21:40:55: pid 22147: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22149: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22139: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22119: LOG: processing sync request from IPC socket
2016-06-27 21:40:55: pid 22151: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22132: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22119: LOG: watchdog node "Linux_sp9.voiplus.net_9999" is requesting to aquire lock for failover command start
2016-06-27 21:40:55: pid 22119: LOG: watchdog node "Linux_sp9.voiplus.net_9999" is requesting to aquire lock for failover command start
2016-06-27 21:40:55: pid 22117: LOG: execute command: /etc/pgpool-II/sbin/failover.sh 0 0 192.168.192.137 /var/lib/pgsql/9.4/data 192.168.192.136
2016-06-27 21:40:55: pid 22145: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22154: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22129: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22143: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22140: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 22146: LOG: child process received shutdown request signal 3
2016-06-27 21:40:55: pid 26645: LOG: child process received shutdown request signal 3
+ FALLING_NODE=0
+ OLDPRIMARY_NODE=0
+ NEW_PRIMARY=192.168.192.137
+ PGDATA=/var/lib/pgsql/9.4/data
+ FALLING_HOST=192.168.192.136
+ LOGFILE=/var/log/failover.log
+ echo '###---------------------------------- starting failover --------------------------###'
+ date
failover with kamailio
-------------------------------------------------------------------------------------------
2016-06-27 20:48:37: pid 8011: ERROR: unable to read data from frontend
2016-06-27 20:48:37: pid 8011: DETAIL: EOF encountered with frontend
2016-06-27 20:48:49: pid 7976: LOG: received fast shutdown request
2016-06-27 20:48:49: pid 21370: LOG: stop request sent to pgpool. waiting for termination...
2016-06-27 20:48:49: pid 7976: LOG: shutdown request. closing listen socket
.2016-06-27 20:48:49: pid 17768: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 18079: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 21245: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 18981: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 19507: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 20720: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 20895: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 13788: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 15988: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 8000: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 15695: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 13085: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 7983: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 19095: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 17949: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 19564: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 21316: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 20088: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 20543: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 20519: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 17473: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 19270: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 19623: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 20603: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 18076: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 18570: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 17005: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 20309: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 7978: LOG: Watchdog is shutting down
2016-06-27 20:48:49: pid 14487: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 18453: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 21385: LOG: watchdog: de-escalation started
2016-06-27 20:48:49: pid 8011: LOG: child process received shutdown request signal 2
2016-06-27 20:48:49: pid 19332: LOG: child process received shutdown request signal 2
............2016-06-27 20:49:01: pid 21385: WARNING: watchdog failed to ping host"192.168.192.142"
2016-06-27 20:49:01: pid 21385: DETAIL: ping process exits with code: 1
2016-06-27 20:49:01: pid 21385: LOG: watchdog bringing down delegate IP
2016-06-27 20:49:01: pid 21385: DETAIL: if_down_cmd succeeded
done.
2016-06-27 21:23:30: pid 22117: LOG: Backend status file /var/log/pgpool/pgpool_status does not exist
2016-06-27 21:23:30: pid 22117: LOG: waiting for watchdog to initialize
2016-06-27 21:23:30: pid 22119: LOG: setting the local watchdog node name to "Linux_sp9.voiplus.net_9999"
2016-06-27 21:23:30: pid 22119: LOG: watchdog cluster configured with 1 remote nodes
2016-06-27 21:23:30: pid 22119: LOG: watchdog remote node:0 on 192.168.192.136:9000
2016-06-27 21:23:30: pid 22119: LOG: interface monitoring is disabled in watchdog
2016-06-27 21:23:30: pid 22119: LOG: IPC socket path: "/tmp/.s.PGPOOLWD_CMD.9000"
2016-06-27 21:23:30: pid 22119: LOG: new outbond connection to 192.168.192.136:9000
2016-06-27 21:23:30: pid 22119: LOG: watchdog node state changed from [LOADING] to [INITIALIZING]
2016-06-27 21:23:31: pid 22119: LOG: watchdog node state changed from [INITIALIZING] to [STANDING FOR MASTER]
2016-06-27 21:23:31: pid 22119: LOG: watchdog node state changed from [STANDING FOR MASTER] to [MASTER]
2016-06-27 21:23:31: pid 22119: LOG: I am announcing my self as master/coordinator watchdog node
2016-06-27 21:23:31: pid 22119: LOG: I am the cluster leader node
2016-06-27 21:23:31: pid 22119: DETAIL: our declare coordinator message is accepted by all nodes
2016-06-27 21:23:31: pid 22119: LOG: I am the cluster leader node. Starting escalation process
2016-06-27 21:23:31: pid 22117: LOG: watchdog process is initialized
2016-06-27 21:23:31: pid 22119: LOG: escalation process started with PID:22120
2016-06-27 21:23:31: pid 22120: LOG: watchdog: escalation started
2016-06-27 21:23:31: pid 22117: LOG: Setting up socket for 0.0.0.0:9999
2016-06-27 21:23:31: pid 22117: LOG: Setting up socket for :::9999
2016-06-27 21:23:31: pid 22119: LOG: new IPC connection received
2016-06-27 21:23:31: pid 22121: LOG: 2 watchdog nodes are configured for lifecheck
2016-06-27 21:23:31: pid 22121: LOG: watchdog nodes ID:0 Name:"Linux_sp9.voiplus.net_9999"
2016-06-27 21:23:31: pid 22121: DETAIL: Host:"192.168.192.137" WD Port:9000 pgpool-II port:9999
2016-06-27 21:23:31: pid 22121: LOG: watchdog nodes ID:1 Name:"Linux_sp0.voiplus.net_9999"
2016-06-27 21:23:31: pid 22121: DETAIL: Host:"192.168.192.136" WD Port:9000 pgpool-II port:9999
2016-06-27 21:23:31: pid 22117: LOG: pgpool-II successfully started. version 3.5.2 (ekieboshi)
2016-06-27 21:23:31: pid 22117: LOG: find_primary_node: checking backend no 0
WARNING: interface is ignored: Operation not permitted
2016-06-27 21:23:31: pid 22117: LOG: find_primary_node: primary node id is 0
2016-06-27 21:23:31: pid 22134: LOG: statement: select min(rates_cntry_area_num) as rates_cntry_area_num,min(rates_jpy_voiplus_rate) as rates_jpy_voiplus_rate,rates_unit_time, rates_destination_jp,rates_destination_en from rates where (rates_destination_en like '%Proper%' or rates_destination_en like '%Domestic%') and (rates_jpy_voiplus_rate > 0 and rates_jpy_voiplus_rate < 23) and rates_intl_num='002' group by rates_destination_en,rates_destination_jp,rates_unit_time order by random() limit 6
2016-06-27 21:23:32: pid 22125: LOG: set SO_REUSEPORT option to the socket
2016-06-27 21:23:32: pid 22125: LOG: creating watchdog heartbeat receive socket.
2016-06-27 21:23:32: pid 22125: DETAIL: set SO_REUSEPORT
2016-06-27 21:23:32: pid 22127: LOG: set SO_REUSEPORT option to the socket
2016-06-27 21:23:32: pid 22127: LOG: creating socket for sending heartbeat
2016-06-27 21:23:32: pid 22127: DETAIL: set SO_REUSEPORT
2016-06-27 21:23:34: pid 22119: LOG: new watchdog node connection is received from "192.168.192.136:4535"
2016-06-27 21:23:45: pid 22120: WARNING: watchdog failed to ping
host"192.168.192.142"
2016-06-27 21:23:45: pid 22120: DETAIL: ping process exits with code: 1
-------------------------------------------------------------------------------------------
More information about the pgpool-general
mailing list