[pgpool-general: 1223] Re: pgpool 3.2.1 - Health check failing to connect

Wed Dec 5 07:59:11 JST 2012

> this is what i have which just repeats
> 
> ec  4 14:36:41 pgpool1 pgpool[70606]: connect_inet_domain_socket: connect() failed: Socket is already connected

This is it. Since pgpool-II 3.2, it uses a non blocking socket to
connect to backend.  I suspect the implementation has difficulty with
some platforms. What OS/version are you using?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> Dec  4 14:36:41 pgpool1 pgpool[70606]: make_persistent_db_connection: connection to prod1.db.power-soft.net(5433) failed
> Dec  4 14:36:41 pgpool1 pgpool[70606]: health check failed. 0 th host prod1.db.power-soft.net at port 5433 is down
> Dec  4 14:36:41 pgpool1 pgpool[70606]: health_check: 0 failover is canceld because failover is disallowed
> Dec  4 14:36:44 pgpool1 pgpool[70606]: connect_inet_domain_socket: connect() failed: Socket is already connected
> Dec  4 14:36:44 pgpool1 pgpool[70606]: make_persistent_db_connection: connection to prod1.db.power-soft.net(5433) failed
> Dec  4 14:36:44 pgpool1 pgpool[70606]: health check failed. 0 th host prod1.db.power-soft.net at port 5433 is down
> Dec  4 14:36:44 pgpool1 pgpool[70606]: health_check: 0 failover is canceld because failover is disallowed
> Dec  4 14:36:45 pgpool1 pgpool[70619]: connect_inet_domain_socket: connect() failed: Socket is already connected
> Dec  4 14:36:45 pgpool1 pgpool[70619]: make_persistent_db_connection: connection to prod1.db.power-soft.net(5433) failed
> Dec  4 14:36:45 pgpool1 pgpool[70619]: connect_inet_domain_socket: connect() failed: Socket is already connected
> Dec  4 14:36:45 pgpool1 pgpool[70619]: make_persistent_db_connection: connection to prod2.db.power-soft.net(5434) failed
> Dec  4 14:36:45 pgpool1 pgpool[70619]: check_replication_time_lag: could not connect to DB node 0, check sr_check_user and sr_check_password
> 
> 
> 
> On 2012-12-04, at 2:39 PM, Tatsuo Ishii <ishii at postgresql.org>
>  wrote:
> 
>> You should have more error messages before this:
>>
>>> Dec  4 11:49:47 pgpool1 pgpool[67018]: make_persistent_db_connection: connection to prod1.db.power-soft.net<http://prod1.db.power-soft.net>(5433) failed
>>
>> Can you show it?
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese: http://www.sraoss.co.jp
>>
>>> Hello List,
>>>
>>> I am having some difficulty with heath check in pgpool. It appears that it can not make a persistent connection so it fails health check.
>>>
>>> The servers are listed as up
>>> prod1.db.power-soft.net5433Up[Detail]
>>> prod2.db.power-soft.net5434Up[Detail]
>>>
>>> IP Address
>>>
>>> Port
>>>
>>> Status
>>>
>>> Weight
>>>
>>> prod1.db.power-soft.net<http://prod1.db.power-soft.net>
>>>
>>> 5433
>>>
>>> Up. Disconnect. Running as primary server
>>>
>>> 0.500
>>>
>>> [Disconnect]
>>>
>>> prod2.db.power-soft.net<http://prod2.db.power-soft.net>
>>>
>>> 5434
>>>
>>> Up. Disconnect. Running as standby server
>>>
>>> 0.500
>>>
>>> [Disconnect] [Promote]
>>>
>>>
>>>
>>> Dec  4 11:49:47 pgpool1 pgpool[67018]: make_persistent_db_connection: connection to prod1.db.power-soft.net<http://prod1.db.power-soft.net>(5433) failed
>>> Dec  4 11:49:47 pgpool1 pgpool[67018]: health check failed. 0 th host prod1.db.power-soft.net<http://prod1.db.power-soft.net> at port 5433 is down
>>> Dec  4 11:49:56 pgpool1 pgpool[67030]: check_replication_time_lag: could not connect to DB node 0, check sr_check_user and sr_check_password
>>>
>>> I am using postgres/postgres for health_check_user/password which I can connect with manually to both the databases.
>>> sr_check_user/password are using postgres/postgres as well.
>>>
>>>
>>> Any suggestions out there?  The only thing I can find in the FAQ that is related to this is if the user does not exist that its trying to connect with and the return error is completely different.
>>>
>>> Both databases are postgresql 9.0.10
>>>
>>> pgpool is on a separate machine pgpool1.db.power-soft.net<http://pgpool1.db.power-soft.net>  there is no firewall blocking connections...
>>>
>>> If I enable failover it will make both prod1 and 2 down.
>>>
>>> here is the debug from a restart, if this helps any
>>>
>>>
>>> Any help would be greatly appreciated!
>>>
>>>
>>>
>>> pgpool restart succeed
>>>
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: listen_addresses
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '*' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: port
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5432 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: socket_dir
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/tmp' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: pcp_port
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 9898 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: pcp_socket_dir
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/tmp' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: enable_pool_hba
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: authentication_timeout
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 60 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: num_init_children
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 10 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: max_pool
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 4 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: child_life_time
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 300 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: child_max_connections
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: connection_life_time
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: client_idle_limit
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_destination
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'syslog' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: print_timestamp
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_connections
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_hostname
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_statement
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_per_node_statement
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_standby_delay
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'if_over_threshold' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: syslog_facility
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'LOCAL0' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: syslog_ident
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'pgpool' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: debug_level
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 10 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: pid_file_name
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/var/run/pgpool/pgpool.pid' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: logdir
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/tmp' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: connection_cache
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: reset_query_list
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'ABORT; DISCARD ALL' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token: ABORT
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token: DISCARD ALL
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: replication_mode
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: replicate_select
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: replicate_select: 0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: insert_lock
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: lobj_lock_table
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: replication_stop_on_mismatch
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: replication_stop_on_mismatch: 0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: failover_if_affected_tuples_mismatch
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: failover_if_affected_tuples_mismatch: 0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: load_balance_mode
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ignore_leading_white_space
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: white_function_list
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: black_function_list
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'currval,lastval,nextval,setval' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token: currval
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token: lastval
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token: nextval
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token: setval
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: add_to_patterns: regex pattern: ^currval$
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: add_to_patterns: regex pattern: ^lastval$
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: add_to_patterns: regex pattern: ^nextval$
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: add_to_patterns: regex pattern: ^setval$
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: master_slave_mode
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: master_slave_sub_mode
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'stream' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: sr_check_period
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: sr_check_user
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: sr_check_password
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: delay_threshold
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: follow_master_command
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: parallel_mode
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: pgpool2_hostname
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_period
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_timeout
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_user
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_password
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_max_retries
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 3 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_retry_delay
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: failover_command
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/usr/local/pgsql/bin/failover_stream.sh %d %H /db-array/db_repo/tmp/trigger_file0' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: failback_command
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: fail_over_on_backend_error
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: recovery_user
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: recovery_password
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: recovery_1st_stage_command
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: recovery_2nd_stage_command
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: recovery_timeout
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 90 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: client_idle_limit_in_recovery
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: use_watchdog
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: trusted_servers
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: delegate_IP
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_hostname
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_port
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 9000 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_interval
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 10 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ping_path
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/bin' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ifconfig_path
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/sbin' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: if_up_cmd
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: if_down_cmd
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'ifconfig eth0:0 down' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: arping_path
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/usr/sbin # arping command path' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: arping_cmd
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'arping -U $_IP_$ -w 1' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_life_point
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 3 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_lifecheck_query
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'SELECT 1' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: relcache_expire
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: relcache_size
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 256 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: check_temp_table
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memory_cache_enabled
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_method
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'shmem' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_memcached_host
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'localhost' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_memcached_port
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 11211 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_total_size
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 67108864 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_max_num_cache
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1000000 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_expire
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_auto_cache_invalidation
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_maxcache
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 409600 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_cache_block_size
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1048576 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_oiddir
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/var/log/pgpool/oiddir' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: white_memqcache_table_list
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: black_memqcache_table_list
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl_key
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl_cert
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl_ca_cert
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl_ca_cert_dir
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: enable_query_cache
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_hostname
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'system_db_hostname' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_port
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5432 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_dbname
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'pgpool' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_schema
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'pgpool_catalog' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_user
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'pgpool' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_password
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_socket_dir
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/tmp' kind: 4
>>> 2012-12-04 12:08:24 LOG: pid 70605: pool_config: backend_socket_dir is deprecated, please use backend_hostname
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_hostname0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'prod1.db.power-soft.net<http://prod1.db.power-soft.net>' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_port0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5433 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: port slot number 0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_weight0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: weight slot number 0 weight: 1.000000
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_data_directory0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/db-array/db_repo/prod1/data' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_flag0
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'DISALLOW_TO_FAILOVER' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token: DISALLOW_TO_FAILOVER
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: disallow_to_failover on
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: slot number 0 flag: 0001
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_hostname1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'prod2.db.power-soft.net<http://prod2.db.power-soft.net>' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_port1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5434 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: port slot number 1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_weight1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1 kind: 2
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: weight slot number 1 weight: 1.000000
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_data_directory1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/db-array/db_repo/prod2/data' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_flag1
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'DISALLOW_TO_FAILOVER' kind: 4
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token: DISALLOW_TO_FAILOVER
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: disallow_to_failover on
>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: slot number 1 flag: 0001
>>>
>>>
>