View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000403 | Pgpool-II | Bug | public | 2018-06-04 13:31 | 2018-11-15 11:30 |
| Reporter | Parkbyunggyu | Assigned To | t-ishii | ||
| Priority | urgent | Severity | crash | Reproducibility | random |
| Status | assigned | Resolution | open | ||
| Platform | X86_64 | OS | CentOS | OS Version | 7.3 |
| Product Version | 3.6.5 | ||||
| Target Version | 3.6.12 | ||||
| Summary | 0000403: About : 0000402: write on backend 0 failed with error :"Success" | ||||
| Description | Hi, I am currently experiencing problems using pg-pool II 3.6.5. Database is using PostgreSQL version 9.6.5 And I'm using H/A streaming replication mode Master - Slave structure. The following warning occasionally occurs , and the connection is temporarily unavailable. When I check other similar issues(issue_ID=0000395), the answer is called Bug. But I can not confirm the solution to the bug, so I issue a new issue. 1. I want to know what circumstances these bugs occur. 2. How can I prevent these bugs from happening? 3. When this bug occurs, what should be done to resolve the problem? I write pg-pool err and upload OS log , postgresql.log Please help me,.. ================================================================================================ 2018-04-18 15:58:02: pid 3307: FATAL: unable to read data from DB node 0 2018-04-18 15:58:02: pid 3307: DETAIL: EOF encountered with backend 2018-04-18 15:58:02: pid 3307: WARNING: write on backend 0 failed with error :"Success" 2018-04-18 15:58:02: pid 3307: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-04-18 15:58:02: pid 10737: LOG: child process with pid: 3307 exits with status 256 2018-04-18 15:58:02: pid 10737: LOG: fork a new child process with pid: 5895 2018-04-18 15:58:02: pid 3308: FATAL: unable to read data from DB node 0 2018-04-18 15:58:02: pid 3308: DETAIL: EOF encountered with backend 2018-04-18 15:58:02: pid 3308: WARNING: write on backend 0 failed with error :"Success" 2018-04-18 15:58:02: pid 3308: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-04-18 15:58:02: pid 10737: LOG: child process with pid: 3308 exits with status 256 2018-04-18 15:58:02: pid 10737: LOG: fork a new child process with pid: 5896 2018-04-18 15:58:02: pid 3315: FATAL: unable to read data from DB node 0 2018-04-18 15:58:02: pid 3315: DETAIL: EOF encountered with backend 2018-04-18 15:58:02: pid 3315: WARNING: write on backend 0 failed with error :"Success" 2018-04-18 15:58:02: pid 3315: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-04-18 15:58:02: pid 3315: WARNING: write on backend 1 failed with error :"Success" 2018-04-18 15:58:02: pid 3315: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-04-18 15:58:02: pid 10737: LOG: child process with pid: 3315 exits with status 256 2018-04-18 15:58:02: pid 10737: LOG: fork a new child process with pid: 5897 2018-04-18 15:58:02: pid 3321: FATAL: unable to read data from DB node 0 2018-04-18 15:58:02: pid 3321: DETAIL: EOF encountered with backend 2018-04-18 15:58:02: pid 3321: WARNING: write on backend 0 failed with error :"Success" 2018-04-18 15:58:02: pid 3321: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-04-18 15:58:02: pid 10737: LOG: child process with pid: 3321 exits with status 256 2018-04-18 15:58:02: pid 10737: LOG: fork a new child process with pid: 5898 2018-04-18 15:58:02: pid 3324: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3324: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3329: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3329: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3331: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3331: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3334: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3334: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3333: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3333: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3339: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3339: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3337: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3337: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3341: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3341: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3342: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3342: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3373: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3373: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3486: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3486: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3527: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3527: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3559: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3559: DETAIL: EOF encountered with frontend 2018-04-18 15:58:02: pid 3562: ERROR: unable to read data from frontend 2018-04-18 15:58:02: pid 3562: DETAIL: EOF encountered with frontend ============================================================================= 2018-06-01 11:13:46: pid 13796: LOG: new connection received 2018-06-01 11:13:46: pid 13796: DETAIL: connecting host=XXX.XXX.XXX.8 port=XXXX 2018-06-01 11:13:46: pid 13796: LOG: new connection received 2018-06-01 11:13:46: pid 13796: DETAIL: connecting host=XXX.XXX.XXX.XXX port=XXXX 2018-06-01 11:13:46: pid 10737: LOG: fork a new child process with pid: 13797 2018-06-01 11:13:46: pid 13797: LOG: new connection received 2018-06-01 11:13:46: pid 13797: DETAIL: connecting host=XXX.XXX.XXX.XXX port=XXXX 2018-06-01 11:13:48: pid 7433: FATAL: unable to read data from DB node 0 2018-06-01 11:13:48: pid 7433: DETAIL: EOF encountered with backend 2018-06-01 11:13:48: pid 7433: WARNING: write on backend 0 failed with error :"Success" 2018-06-01 11:13:48: pid 7433: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-06-01 11:13:48: pid 10737: LOG: child process with pid: 7433 exits with status 256 2018-06-01 11:13:48: pid 10737: LOG: fork a new child process with pid: 13800 2018-06-01 11:13:48: pid 13800: LOG: new connection received 2018-06-01 11:13:48: pid 13800: DETAIL: connecting host=XXX.XXX.XXX.XXX port=XXXX 2018-06-01 11:13:48: pid 10902: FATAL: unable to read data from DB node 0 2018-06-01 11:13:48: pid 10902: DETAIL: EOF encountered with backend 2018-06-01 11:13:48: pid 10902: WARNING: write on backend 0 failed with error :"Success" 2018-06-01 11:13:48: pid 10902: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-06-01 11:13:48: pid 10737: LOG: child process with pid: 10902 exits with status 256 2018-06-01 11:13:48: pid 10737: LOG: fork a new child process with pid: 13802 2018-06-01 11:13:48: pid 13802: LOG: new connection received 2018-06-01 11:13:48: pid 13802: DETAIL: connecting host=XXX.XXX.XXX.XXX port=XXXX 2018-06-01 11:13:49: pid 4042: FATAL: unable to read data from DB node 0 2018-06-01 11:13:49: pid 4042: DETAIL: EOF encountered with backend 2018-06-01 11:13:49: pid 4042: WARNING: write on backend 0 failed with error :"Success" 2018-06-01 11:13:49: pid 4042: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-06-01 11:13:49: pid 10737: LOG: child process with pid: 4042 exits with status 256 2018-06-01 11:13:49: pid 10737: LOG: fork a new child process with pid: 13805 2018-06-01 11:13:49: pid 13805: LOG: new connection received 2018-06-01 11:13:49: pid 13805: DETAIL: connecting host=XXX.XXX.XXX.XXX port=XXXX 2018-06-01 11:13:51: pid 9303: FATAL: unable to read data from DB node 0 2018-06-01 11:13:51: pid 9303: DETAIL: EOF encountered with backend 2018-06-01 11:13:51: pid 9303: WARNING: write on backend 0 failed with error :"Success" 2018-06-01 11:13:51: pid 9303: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-06-01 11:13:51: pid 10737: LOG: child process with pid: 9303 exits with status 256 2018-06-01 11:13:51: pid 10737: LOG: fork a new child process with pid: 13808 2018-06-01 11:13:51: pid 13808: LOG: new connection received 2018-06-01 11:13:51: pid 13808: DETAIL: connecting host=XXX.XXX.XXX.XXX port=XXXX 2018-06-01 11:13:52: pid 9318: FATAL: unable to read data from DB node 0 2018-06-01 11:13:52: pid 9318: DETAIL: EOF encountered with backend 2018-06-01 11:13:52: pid 9318: WARNING: write on backend 0 failed with error :"Success" 2018-06-01 11:13:52: pid 9318: DETAIL: while trying to write data from offset: 0 wlen: 5 2018-06-01 11:13:52: pid 10737: LOG: child process with pid: 9318 exits with status 256 2018-06-01 11:13:52: pid 10737: LOG: fork a new child process with pid: 13811 2018-06-01 11:13:52: pid 13811: LOG: new connection received 2018-06-01 11:13:52: pid 13811: DETAIL: connecting host=XXX.XXX.XXX.XXX port=XXXX 2018-06-01 11:13:54: pid 25112: FATAL: unable to read data from DB node 0 2018-06-01 11:13:54: pid 25112: DETAIL: EOF encountered with backend 2018-06-01 11:13:54: pid 25112: WARNING: write on backend 0 failed with error :"Success" 2018-06-01 11:13:54: pid 25112: DETAIL: while trying to write data from offset: 0 wlen: 5 ============================================================================= | ||||
| Tags | pgpool-II | ||||
|
|
|
|
|
I am not sure the latest Pgpool-II 3.6 still has the problem as 3.6.5 is pretty old (released 1 year ago). Anyway the symptom seems to be similar to bug 395, so I recommend you to apply the patch I uploaded and see if the issue is solved. |
|
|
how can in download what you uoload? where is it?? |
|
|
https://www.pgpool.net/mantisbt/file_download.php?file_id=449&type=bug |
|
|
Thanks a quick answer. I have one more question... Do you know how to apply the file you uploaded? Is there a link that says that? |
|
|
Download the file and save it as /tmp/write.diff cd pgpool-II-3.6.5 patch -b -p1 < /tmp/write.diff ./configure .... make make install |
|
|
1. I installed pg-poll with rpm Is it possible that way?? 2. If i use your way Do I need to restart the database service? |
|
|
> 1. I installed pg-poll with rpm > Is it possible that way?? No. |
|
|
So, is there any solution you can tell? |
|
|
Technically I you could create a new rpm including the fix. If you are not sure how to do it, you could look for someone who is familiar with rpm packaging. |
|
|
I have committed/pushed the patch. Will appear in the next minor releases. |
|
|
Thank you.. you are my...Messiah... I have a question. 1. Is the patch you are talking about rpm patching? 2. If so, when is the next release coming out? 3. I will report to my boss that you will fix the problem by patching. But the boss is not exactly aware of your existence. Can I report to my boss that you are a developer of pg-poolII? If so, can you tell me the proof that you are a developer of pg-poolII? You may feel like I'm asking you straight and cheeky, but I know I need to be polite when I ask you these questions to you. My English translations are of low quality. I'm sorry to ask you an rude question. If possible, I hope you will give me the answer. |
|
|
> 1. Is the patch you are talking about rpm patching? No. It's against the source code. New rpm including my fix will come with the next release. > 2. If so, when is the next release coming out? Not decided yet but we are thinking about early August. > 3. I will report to my boss that you will fix the problem by patching. But the boss is not exactly aware of your existence. Can I report to my boss that you are a developer of pg-poolII? If so, can you tell me the proof that you are a developer of pg-poolII? Interesting question. I was never asked such a question:-) My name is Tatsuo Ishii and here is the fix I pushed into the Pgpool-II git repository. https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=ec7c84ee88a74e74a316b764a9c8ef2a7d0c49be If you look into this, you will notice that the commit is exactly same as I already let you know. Also remember that only developers can push fixes into the repository. BTW, I would like to tell you that I am the original author of Pgpool if you are not aware of the fact. > You may feel like I'm asking you straight and cheeky, but I know I need to be polite when I ask you these questions to you. My English translations are of low quality. I'm sorry to ask you an rude question. If possible, I hope you will give me the answer. No problem. I am not an English speaker either. |
|
|
Hi, t-ishii Long time no see.. 1. Because of this issue I updated the pg-pool. (3.6.6 -> 3.6.12) But the same problem is happening. Could there be a cause or solution to this problem? 2. I set the connection setting for pg-pool to 2000 users. However, 760 users are connected, but the connection time exceeds the connection time. Do you know why or why there is no connection in the pg-pool after the connection time has passed? If you know, please reply. 3. here is my pg-pool & DB configuration and if you need another information please reply PG-pool config listen_addresses = '*' port = XXXX socket_dir = '/tmp' pcp_listen_addresses = '*' pcp_port = XXXX pcp_socket_dir = '/tmp' listen_backlog_multiplier = 2 serialize_accept = off backend_hostname0 = 'xx.xx.xxx.xxx' backend_port0 = XXXX backend_weight0 = 0.5 backend_data_directory0 = '/data/data' backend_flag0 = 'ALLOW_TO_FAILOVER' backend_hostname1 = 'xx.xx.xxx.xxx' backend_port1 = XXXX backend_weight1 = 0.5 backend_data_directory1 = '/data/data' backend_flag1 = 'ALLOW_TO_FAILOVER' enable_pool_hba = off pool_passwd = 'pool_passwd' authentication_timeout = 60 ssl = off num_init_children = 2050 max_pool = 1 child_life_time = 300 child_max_connections = 0 connection_life_time = 0 client_idle_limit = 0 log_destination = 'stderr' log_connections = on log_hostname = off log_statement = off log_per_node_statement = off log_standby_delay = 'if_over_threshold' syslog_facility = 'LOCAL0' logdir = '/tmp' syslog_ident = 'pgpool' pid_file_name = '/var/run/pgpool/pgpool.pid' connection_cache = on reset_query_list = 'ABORT; DISCARD ALL' replication_mode = off replicate_select = off insert_lock = off lobj_lock_table = '' replication_stop_on_mismatch = off failover_if_affected_tuples_mismatch = off load_balance_mode = on ignore_leading_white_space = on white_function_list = '' black_function_list = 'currval,lastval,nextval,setval' database_redirect_preference_list = '' app_name_redirect_preference_list = '' allow_sql_comments = off master_slave_mode = on master_slave_sub_mode = 'stream' sr_check_period = 10 sr_check_user = 'XXXXXX' sr_check_password = 'XXXXXX' sr_check_database = 'postgres' delay_threshold = 10000000 follow_master_command = '' health_check_period = 0 health_check_timeout = 20 health_check_user = 'nobody' health_check_password = '' health_check_database = '' health_check_max_retries = 0 health_check_retry_delay = 1 connect_timeout = 10000 failover_command = '/etc/pgpool-II/failover.sh %d %P %H %R' failback_command = '' fail_over_on_backend_error = on search_primary_node_timeout = 300 recovery_user = 'nobody' recovery_password = '' recovery_1st_stage_command = '' recovery_2nd_stage_command = '' recovery_timeout = 90 client_idle_limit_in_recovery = 0 use_watchdog = off trusted_servers = '' ping_path = '/bin' wd_hostname = '' wd_port = 9000 wd_priority = 1 wd_authkey = '' wd_ipc_socket_dir = '/tmp' delegate_IP = '' if_cmd_path = '/sbin' if_up_cmd = 'ip addr add $_IP_$/24 dev eth0 label eth0:0' if_down_cmd = 'ip addr del $_IP_$/24 dev eth0' arping_path = '/usr/sbin' arping_cmd = 'arping -U $_IP_$ -w 1' clear_memqcache_on_escalation = on wd_escalation_command = '' wd_de_escalation_command = '' wd_lifecheck_method = 'heartbeat' wd_interval = 2 wd_heartbeat_port = XXXX wd_heartbeat_keepalive = 2 wd_heartbeat_deadtime = 30 heartbeat_destination0 = 'host0_ip1' heartbeat_destination_port0 = XXXX heartbeat_device0 = '' wd_life_point = 3 wd_lifecheck_query = 'SELECT 1' wd_lifecheck_dbname = 'template1' wd_lifecheck_user = 'nobody' wd_lifecheck_password = '' relcache_expire = 0 relcache_size = 256 check_temp_table = on check_unlogged_table = on memory_cache_enabled = off memqcache_method = 'shmem' memqcache_memcached_host = 'localhost' memqcache_memcached_port = 11211 memqcache_total_size = 67108864 memqcache_max_num_cache = 1000000 memqcache_expire = 0 memqcache_auto_cache_invalidation = on memqcache_maxcache = 409600 memqcache_cache_block_size = 1048576 memqcache_oiddir = '/var/log/pgpool/oiddir' white_memqcache_table_list = '' black_memqcache_table_list = '' DB config listen_addresses ='*' port = xxxx max_connections = 2100 shared_buffers = 32GB work_mem = 16MB maintenance_work_mem = 1GB wal_level = replica checkpoint_timeout = 10min max_wal_size = 4GB min_wal_size = 2GB checkpoint_completion_target = 0.9 archive_mode = on archive_command = 'cp %p /data/archive/%f' max_wal_senders = 5 random_page_cost = 2.0 effective_cache_size = 96GB log_destination = 'stderr' logging_collector = on log_min_duration_statement = 3s track_activity_query_size = 65536 pg_stat_statements.track = all pg_stat_statements.max = 10000 shared_preload_libraries = '$libdir/pg_stat_statements' |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2018-06-04 13:31 | Parkbyunggyu | New Issue | |
| 2018-06-04 13:31 | Parkbyunggyu | File Added: postgresql-2018-04-18_000000_1.log | |
| 2018-06-04 13:31 | Parkbyunggyu | File Added: messages_1.txt | |
| 2018-06-04 13:31 | Parkbyunggyu | Tag Attached: pgpool-II | |
| 2018-06-15 09:18 | pengbo | Assigned To | => t-ishii |
| 2018-06-15 09:18 | pengbo | Status | new => assigned |
| 2018-06-15 09:25 | t-ishii | Note Added: 0002046 | |
| 2018-06-15 09:26 | t-ishii | Status | assigned => feedback |
| 2018-06-21 13:56 | Parkbyunggyu | Note Added: 0002057 | |
| 2018-06-21 13:56 | Parkbyunggyu | Status | feedback => assigned |
| 2018-06-21 14:13 | t-ishii | Note Added: 0002058 | |
| 2018-06-21 14:18 | Parkbyunggyu | Note Added: 0002059 | |
| 2018-06-21 15:00 | t-ishii | Note Added: 0002063 | |
| 2018-06-21 15:22 | Parkbyunggyu | Note Added: 0002064 | |
| 2018-06-21 15:30 | t-ishii | Note Added: 0002065 | |
| 2018-06-21 15:35 | Parkbyunggyu | Note Added: 0002066 | |
| 2018-06-22 15:35 | t-ishii | Note Added: 0002069 | |
| 2018-06-22 15:35 | t-ishii | Status | assigned => feedback |
| 2018-07-04 15:29 | t-ishii | Target Version | => 3.6.12 |
| 2018-07-04 15:30 | t-ishii | Note Added: 0002085 | |
| 2018-07-04 17:48 | Parkbyunggyu | Note Added: 0002086 | |
| 2018-07-04 17:48 | Parkbyunggyu | Status | feedback => assigned |
| 2018-07-04 22:56 | t-ishii | Note Added: 0002090 | |
| 2018-11-15 11:30 | Parkbyunggyu | Note Added: 0002257 |
