[pgpool-general: 1226] Re: pgpool 3.2.1 - Health check failing to connect

Wed Dec 5 08:25:20 JST 2012

All that I can see in the sytem log is;

Dec  4 15:22:20 pgpool1 pgpool[70606]: connect_inet_domain_socket: connect() failed: Socket is already connected

If you can tell me how to gather the info I'll get it for you.

On 2012-12-04, at 3:17 PM, Tatsuo Ishii <ishii at postgresql.org>
 wrote:

> Great. We alreay have similar bug report from BSD user
> (http://www.pgpool.net/mantisbt/view.php?id=29). Since OS X is based
> on BSD, it is not surprising that you have same problem. I asked the
> reporter this question:
>
>                 "Socket is already connected": what is the error no and error
>                 symbol(macro)? I couldn't find it on Linux.
>
> and got no answer so far. I need this info to fix the problem. Do you have any idea?
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
>
>> OSX Snow Leopard Server
>>
>>
>>
>> -----Original Message-----
>> From: Tatsuo Ishii [mailto:ishii at postgresql.org]
>> Sent: Tuesday, December 04, 2012 2:59 PM
>> To: Mikola Rose
>> Cc: pgpool-general at pgpool.net
>> Subject: Re: [pgpool-general: 1218] pgpool 3.2.1 - Health check failing to connect
>>
>>> this is what i have which just repeats
>>>
>>> ec  4 14:36:41 pgpool1 pgpool[70606]: connect_inet_domain_socket:
>>> connect() failed: Socket is already connected
>>
>> This is it. Since pgpool-II 3.2, it uses a non blocking socket to connect to backend.  I suspect the implementation has difficulty with some platforms. What OS/version are you using?
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese: http://www.sraoss.co.jp
>>
>>> Dec  4 14:36:41 pgpool1 pgpool[70606]: make_persistent_db_connection:
>>> connection to prod1.db.power-soft.net(5433) failed Dec  4 14:36:41
>>> pgpool1 pgpool[70606]: health check failed. 0 th host
>>> prod1.db.power-soft.net at port 5433 is down Dec  4 14:36:41 pgpool1
>>> pgpool[70606]: health_check: 0 failover is canceld because failover is
>>> disallowed Dec  4 14:36:44 pgpool1 pgpool[70606]:
>>> connect_inet_domain_socket: connect() failed: Socket is already
>>> connected Dec  4 14:36:44 pgpool1 pgpool[70606]:
>>> make_persistent_db_connection: connection to
>>> prod1.db.power-soft.net(5433) failed Dec  4 14:36:44 pgpool1
>>> pgpool[70606]: health check failed. 0 th host prod1.db.power-soft.net
>>> at port 5433 is down Dec  4 14:36:44 pgpool1 pgpool[70606]:
>>> health_check: 0 failover is canceld because failover is disallowed Dec
>>> 4 14:36:45 pgpool1 pgpool[70619]: connect_inet_domain_socket:
>>> connect() failed: Socket is already connected Dec  4 14:36:45 pgpool1
>>> pgpool[70619]: make_persistent_db_connection: connection to
>>> prod1.db.power-soft.net(5433) failed Dec  4 14:36:45 pgpool1
>>> pgpool[70619]: connect_inet_domain_socket: connect() failed: Socket is
>>> already connected Dec  4 14:36:45 pgpool1 pgpool[70619]:
>>> make_persistent_db_connection: connection to
>>> prod2.db.power-soft.net(5434) failed Dec  4 14:36:45 pgpool1
>>> pgpool[70619]: check_replication_time_lag: could not connect to DB
>>> node 0, check sr_check_user and sr_check_password
>>>
>>>
>>>
>>> On 2012-12-04, at 2:39 PM, Tatsuo Ishii <ishii at postgresql.org>
>>> wrote:
>>>
>>>> You should have more error messages before this:
>>>>
>>>>> Dec  4 11:49:47 pgpool1 pgpool[67018]:
>>>>> make_persistent_db_connection: connection to
>>>>> prod1.db.power-soft.net<http://prod1.db.power-soft.net>(5433) failed
>>>>
>>>> Can you show it?
>>>> --
>>>> Tatsuo Ishii
>>>> SRA OSS, Inc. Japan
>>>> English: http://www.sraoss.co.jp/index_en.php
>>>> Japanese: http://www.sraoss.co.jp
>>>>
>>>>> Hello List,
>>>>>
>>>>> I am having some difficulty with heath check in pgpool. It appears that it can not make a persistent connection so it fails health check.
>>>>>
>>>>> The servers are listed as up
>>>>> prod1.db.power-soft.net5433Up[Detail]
>>>>> prod2.db.power-soft.net5434Up[Detail]
>>>>>
>>>>> IP Address
>>>>>
>>>>> Port
>>>>>
>>>>> Status
>>>>>
>>>>> Weight
>>>>>
>>>>> prod1.db.power-soft.net<http://prod1.db.power-soft.net>
>>>>>
>>>>> 5433
>>>>>
>>>>> Up. Disconnect. Running as primary server
>>>>>
>>>>> 0.500
>>>>>
>>>>> [Disconnect]
>>>>>
>>>>> prod2.db.power-soft.net<http://prod2.db.power-soft.net>
>>>>>
>>>>> 5434
>>>>>
>>>>> Up. Disconnect. Running as standby server
>>>>>
>>>>> 0.500
>>>>>
>>>>> [Disconnect] [Promote]
>>>>>
>>>>>
>>>>>
>>>>> Dec  4 11:49:47 pgpool1 pgpool[67018]:
>>>>> make_persistent_db_connection: connection to
>>>>> prod1.db.power-soft.net<http://prod1.db.power-soft.net>(5433) failed
>>>>> Dec  4 11:49:47 pgpool1 pgpool[67018]: health check failed. 0 th
>>>>> host prod1.db.power-soft.net<http://prod1.db.power-soft.net> at port
>>>>> 5433 is down Dec  4 11:49:56 pgpool1 pgpool[67030]:
>>>>> check_replication_time_lag: could not connect to DB node 0, check
>>>>> sr_check_user and sr_check_password
>>>>>
>>>>> I am using postgres/postgres for health_check_user/password which I can connect with manually to both the databases.
>>>>> sr_check_user/password are using postgres/postgres as well.
>>>>>
>>>>>
>>>>> Any suggestions out there?  The only thing I can find in the FAQ that is related to this is if the user does not exist that its trying to connect with and the return error is completely different.
>>>>>
>>>>> Both databases are postgresql 9.0.10
>>>>>
>>>>> pgpool is on a separate machine pgpool1.db.power-soft.net<http://pgpool1.db.power-soft.net>  there is no firewall blocking connections...
>>>>>
>>>>> If I enable failover it will make both prod1 and 2 down.
>>>>>
>>>>> here is the debug from a restart, if this helps any
>>>>>
>>>>>
>>>>> Any help would be greatly appreciated!
>>>>>
>>>>>
>>>>>
>>>>> pgpool restart succeed
>>>>>
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: listen_addresses
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '*' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: port
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5432 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: socket_dir
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/tmp' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: pcp_port
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 9898 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: pcp_socket_dir
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/tmp' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: enable_pool_hba
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: authentication_timeout
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 60 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: num_init_children
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 10 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: max_pool
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 4 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: child_life_time
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 300 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: child_max_connections
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: connection_life_time
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: client_idle_limit
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_destination
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'syslog' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: print_timestamp
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_connections
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_hostname
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_statement
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_per_node_statement
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: log_standby_delay
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'if_over_threshold'
>>>>> kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: syslog_facility
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'LOCAL0' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: syslog_ident
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'pgpool' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: debug_level
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 10 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: pid_file_name
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value:
>>>>> '/var/run/pgpool/pgpool.pid' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: logdir
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/tmp' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: connection_cache
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: reset_query_list
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'ABORT; DISCARD ALL'
>>>>> kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token:
>>>>> ABORT
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token:
>>>>> DISCARD ALL
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: replication_mode
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: replicate_select
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: replicate_select: 0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: insert_lock
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: lobj_lock_table
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> replication_stop_on_mismatch
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: replication_stop_on_mismatch:
>>>>> 0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> failover_if_affected_tuples_mismatch
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605:
>>>>> failover_if_affected_tuples_mismatch: 0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: load_balance_mode
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> ignore_leading_white_space
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: white_function_list
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: black_function_list
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value:
>>>>> 'currval,lastval,nextval,setval' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token:
>>>>> currval
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token:
>>>>> lastval
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token:
>>>>> nextval
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token:
>>>>> setval
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: add_to_patterns: regex
>>>>> pattern: ^currval$
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: add_to_patterns: regex
>>>>> pattern: ^lastval$
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: add_to_patterns: regex
>>>>> pattern: ^nextval$
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: add_to_patterns: regex
>>>>> pattern: ^setval$
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: master_slave_mode
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: master_slave_sub_mode
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'stream' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: sr_check_period
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: sr_check_user
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: sr_check_password
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: delay_threshold
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: follow_master_command
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: parallel_mode
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: pgpool2_hostname
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_period
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_timeout
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_user
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_password
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_max_retries
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 3 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: health_check_retry_delay
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: failover_command
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value:
>>>>> '/usr/local/pgsql/bin/failover_stream.sh %d %H
>>>>> /db-array/db_repo/tmp/trigger_file0' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: failback_command
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> fail_over_on_backend_error
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: recovery_user
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: recovery_password
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'postgres' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> recovery_1st_stage_command
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> recovery_2nd_stage_command
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: recovery_timeout
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 90 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> client_idle_limit_in_recovery
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: use_watchdog
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: trusted_servers
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: delegate_IP
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_hostname
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_port
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 9000 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_interval
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 10 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ping_path
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/bin' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ifconfig_path
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/sbin' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: if_up_cmd
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'ifconfig eth0:0 inet
>>>>> $_IP_$ netmask 255.255.255.0' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: if_down_cmd
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'ifconfig eth0:0 down'
>>>>> kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: arping_path
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/usr/sbin # arping
>>>>> command path' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: arping_cmd
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'arping -U $_IP_$ -w 1'
>>>>> kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_life_point
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 3 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: wd_lifecheck_query
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'SELECT 1' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: relcache_expire
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: relcache_size
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 256 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: check_temp_table
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memory_cache_enabled
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_method
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'shmem' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_memcached_host
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'localhost' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_memcached_port
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 11211 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_total_size
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 67108864 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_max_num_cache
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1000000 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_expire
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 0 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> memqcache_auto_cache_invalidation
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: on kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_maxcache
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 409600 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> memqcache_cache_block_size
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1048576 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: memqcache_oiddir
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value:
>>>>> '/var/log/pgpool/oiddir' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> white_memqcache_table_list
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key:
>>>>> black_memqcache_table_list
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl_key
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl_cert
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl_ca_cert
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: ssl_ca_cert_dir
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: enable_query_cache
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: off kind: 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_hostname
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'system_db_hostname'
>>>>> kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_port
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5432 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_dbname
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'pgpool' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_schema
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'pgpool_catalog' kind:
>>>>> 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_user
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'pgpool' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: system_db_password
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_socket_dir
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: '/tmp' kind: 4
>>>>> 2012-12-04 12:08:24 LOG: pid 70605: pool_config: backend_socket_dir
>>>>> is deprecated, please use backend_hostname
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_hostname0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value:
>>>>> 'prod1.db.power-soft.net<http://prod1.db.power-soft.net>' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_port0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5433 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: port slot number
>>>>> 0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_weight0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: weight slot
>>>>> number 0 weight: 1.000000
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_data_directory0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value:
>>>>> '/db-array/db_repo/prod1/data' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_flag0
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'DISALLOW_TO_FAILOVER'
>>>>> kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token:
>>>>> DISALLOW_TO_FAILOVER
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config:
>>>>> disallow_to_failover on
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: slot number 0
>>>>> flag: 0001
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_hostname1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value:
>>>>> 'prod2.db.power-soft.net<http://prod2.db.power-soft.net>' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_port1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 5434 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: port slot number
>>>>> 1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_weight1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 1 kind: 2
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: weight slot
>>>>> number 1 weight: 1.000000
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_data_directory1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value:
>>>>> '/db-array/db_repo/prod2/data' kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: key: backend_flag1
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: value: 'DISALLOW_TO_FAILOVER'
>>>>> kind: 4
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: extract_string_tokens: token:
>>>>> DISALLOW_TO_FAILOVER
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config:
>>>>> disallow_to_failover on
>>>>> 2012-12-04 12:08:24 DEBUG: pid 70605: pool_config: slot number 1
>>>>> flag: 0001
>>>>>
>>>>>
>>>