Difference between revisions of "pgpool-II 3.5 watchdog test"

From pgpool Wiki
Jump to: navigation, search
Line 134: Line 134:
 
|----
 
|----
 
|7.3
 
|7.3
|Cheking other functionality of watchdog  
+
|Checking other functionality of watchdog  
 
|Automatic registration of a server as standby in recovery
 
|Automatic registration of a server as standby in recovery
 
|
 
|
Line 156: Line 156:
 
|9
 
|9
 
|Networking Isolation scenarion
 
|Networking Isolation scenarion
|
+
|In this scenario we need to check if the pgpool primary and secondary node loose contact due to interruption in network connectivity. This means that two instances loose connectivity.
|
+
|The watchdog process should wait based on some timeout value in order to see if the network comes backup. Need to make sure that we don't have a situation where both instances are acting as primary.
 
|
 
|
 
|
 
|
 
|----
 
|----
|9.1
+
|10
|
+
|Testing watchdog on cloud
|
+
|In this case we want to test watchdog functionality by deploying watchdog, pgpool instances and database server on Amazon cloud. The goal should be perform all the test scenario on AWS that are performed on-premise 
|
+
|All the watchdog functionality should work on AWS similar to how it works in on-premise deployment.
 
|
 
|
 
|
 
|
 
|----
 
|----
|10
+
|11
 
|Database Failure
 
|Database Failure
|
+
|In the test case, the PG database running on the secondary dies. This can be done by stopping PG services on the secondary node.
|
+
|The expected scenario is that pgpool primary node will function as is and continue to serving queries from the client. A pgpool secondary node can be added later on to the watchdog cluster, i believe pgpool needs to be restarted after adding back the secondary node.
 
|
 
|
 
|
 
|
 
|----
 
|----
|10.1
+
|12
|
+
|Database Failure
|
+
|In the test case, the PG database running on the primary node dies. This can be done by stopping PG services on the primary node.
|
+
|The expected scenario is that pgpool instance running secondary node will be promoted to primary node and it  will serve queries from the client. A secondary node can be added later on, i believe pgpool needs to be restarted after adding back the secondary node.
 
|
 
|
 
|
 
|

Revision as of 13:07, 5 November 2015

Test Num Category Test Description Expected Output How to test Assigned-to
1 Installation Make sure that the new watchdog is installed and configured successfully
2 Upgrade Make sure that pgpool II with new watchdog can be installed on a system running pgpool II with the old watchdog
3 Configuration Make sure that pgpool II can be configured successfully with one primary and one stand-by configuration
4 Setup Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a sample query Configruation
4.1 Functional testing Shutdown Host-1's pgpool instanse and execute query again Host-2 should take and respond to query
4.2 Functional testing Shutdown Host-2's pgpool instanse and execute query again Host-3 should take and respond to query
4.3 Functional testing Start Host-1's pgpool instanse and execute query again Need to see which host will respond to query
5 Failover scenarions / Setup Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a sample query Configruation
5.1 Failover scenarios Un-Plug Host-1's network cable and execute query again Host-2 should take and respond to query
5.2 Failover scenarios Un-Plug Host-2's network cable and execute query again Host-3 should take and respond to query
5.3 Failover scenarios Plug Host-1's network cable execute query again Need to see which host will respond to query
6 Functional testing / Setup Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a long query
6.1 Functional testing Shutdown / Power Off Host-1's instanse and execute query again Host-2 should take over and start responding, need to see the already running query response.
6.2 Functional testing Shutdown / Power Off Host-2's instanse and execute query again Host-3 should take over and start responding, need to see the already running query response.
6.3 Functional testing Start Host-1's pgpool instanse and execute query again Need to see which host will respond to query
7.1 Cheking other functionality of watchdog Changing active/standby state in case of certain faults detected
7.2 Cheking other functionality of watchdog Automatic virtual IP address assigning synchronous to server switching
7.3 Checking other functionality of watchdog Automatic registration of a server as standby in recovery
8 Isolated master scenario / Setup Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a query
8.1 Isolated master scenario Break the connectivity between pgpool watchdog primary and stand node by bringing down connectivity on the stand-by. Split brain testing ensures that their is only one master at a time that the clients can connect to. The stand-by should be promoted as the primary and the clients shouldn't be able to connect to the old master.
9 Networking Isolation scenarion In this scenario we need to check if the pgpool primary and secondary node loose contact due to interruption in network connectivity. This means that two instances loose connectivity. The watchdog process should wait based on some timeout value in order to see if the network comes backup. Need to make sure that we don't have a situation where both instances are acting as primary.
10 Testing watchdog on cloud In this case we want to test watchdog functionality by deploying watchdog, pgpool instances and database server on Amazon cloud. The goal should be perform all the test scenario on AWS that are performed on-premise All the watchdog functionality should work on AWS similar to how it works in on-premise deployment.
11 Database Failure In the test case, the PG database running on the secondary dies. This can be done by stopping PG services on the secondary node. The expected scenario is that pgpool primary node will function as is and continue to serving queries from the client. A pgpool secondary node can be added later on to the watchdog cluster, i believe pgpool needs to be restarted after adding back the secondary node.
12 Database Failure In the test case, the PG database running on the primary node dies. This can be done by stopping PG services on the primary node. The expected scenario is that pgpool instance running secondary node will be promoted to primary node and it will serve queries from the client. A secondary node can be added later on, i believe pgpool needs to be restarted after adding back the secondary node.
10.2
11 Watchdog agent failure
11.1
11.2
12 Pgpool-II Watchdog integration
12.1 PG backend failover with watchdog Test the command interlocking, when pgpool-II watchdog in enabled. The failover and follow master scripts should only be executed by one pgpool-II node
12.2 PG backend failback with watchdog Test the command interlocking, when pgpool-II watchdog in enabled. The failback scripts should only be executed by one pgpool-II node
12.3 online recovery with watchdog enabled Execute online recovery with watchdog enabled.
12.4 pgpool-II configuration integrity Perform test with changing pgpool-II configurations on different pgpool-II nodes Standby watchdog shoud report and fail to start if the configurations on master node is differnet from this node
13 integrate external node health checking Test if watchdog can successfully integrate with external health-checking system send the node down and node alive messages to watchdog ipc socket and it should handle these appropriately