View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000700 | Pgpool-II | Bug | public | 2021-03-29 22:38 | 2021-05-20 12:13 |
| Reporter | G79189 | Assigned To | t-ishii | ||
| Priority | high | Severity | major | Reproducibility | always |
| Status | closed | Resolution | open | ||
| Product Version | 4.2.2 | ||||
| Target Version | 4.2.3 | Fixed in Version | 4.2.3 | ||
| Summary | 0000700: Pgpool child process with pid: XXXXX was terminated by segmentation fault | ||||
| Description | Hi, I am using Pgpool-II 4.2.2 with single instance of PostgreSQL as a backend node and been running into segmentation fault errors with pgpool only when using in-memory caching (memory_cache_enabled = on). When memory_cache_enabled is off, not really seeing any such issues. Pgpool errors: 2021-03-26 03:41:23: pid 17816: WARNING: child process with pid: 18034 was terminated by segmentation fault 2021-03-26 03:41:23: pid 17816: LOG: fork a new child process with pid: 19635 2021-03-26 03:43:19: pid 17816: WARNING: child process with pid: 18720 was terminated by segmentation fault 2021-03-26 03:43:19: pid 17816: LOG: fork a new child process with pid: 19683 Also, after a series of segmentation fault errors pgpool going into a bad state with errors "ERROR: Sorry, too many clients already". 2021-03-26 04:06:56: pid 17816: LOG: fork a new child process with pid: 24196 2021-03-26 04:07:35: pid 17816: WARNING: child process with pid: 23965 was terminated by segmentation fault 2021-03-26 04:07:35: pid 17816: LOG: fork a new child process with pid: 24257 2021-03-26 04:08:20: pid 17816: WARNING: child process with pid: 24189 was terminated by segmentation fault 2021-03-26 04:08:20: pid 17816: LOG: fork a new child process with pid: 24314 2021-03-26 04:08:23: pid 17816: WARNING: child process with pid: 24196 was terminated by segmentation fault 2021-03-26 04:08:23: pid 17816: LOG: fork a new child process with pid: 24325 2021-03-26 04:08:43: pid 17816: WARNING: child process with pid: 24181 was terminated by segmentation fault 2021-03-26 04:08:43: pid 17816: LOG: fork a new child process with pid: 24348 2021-03-26 04:08:47: pid 24314: LOG: failed to delete cache on memcached, error:"NOT FOUND" 2021-03-26 04:09:00: pid 24325: ERROR: Sorry, too many clients already 2021-03-26 04:09:00: pid 24325: ERROR: Sorry, too many clients already 2021-03-26 04:09:00: pid 24325: ERROR: Sorry, too many clients already 2021-03-26 04:09:01: pid 20135: ERROR: Sorry, too many clients already 2021-03-26 04:09:03: pid 17816: WARNING: child process with pid: 24314 was terminated by segmentation fault 2021-03-26 04:09:03: pid 17816: LOG: fork a new child process with pid: 24506 2021-03-26 04:09:04: pid 24506: ERROR: Sorry, too many clients already 2021-03-26 04:09:07: pid 23723: ERROR: Sorry, too many clients already | ||||
| Steps To Reproduce | - Configure pgpool Pgpool-II 4.2.2 with on backend PostgreSQL (here backed is Azure PostgreSQL 11.6) - Configured memcached with 128 MB - Java JDBC connections from client application running fine before enabled in-memory caching - Enable in-memory caching with memcached - Started seeing segmentation faluts, - After around 43-45 such errors (core dumps). seeing "ERROR: Sorry, too many clients already" pgpool.conf configuration ( 1 backend node) backend_clustering_mode = 'raw' listen_addresses = '127.0.0.1' port = 5432 backend_flag0 = 'ALWAYS_PRIMARY' enable_pool_hba = on pool_passwd = '' authentication_timeout = 1min allow_clear_text_frontend_auth = on ssl = on num_init_children = 48 max_pool = 6 listen_backlog_multiplier = 4 serialize_accept = on memory_cache_enabled = on memqcache_method = 'memcached' memqcache_memcached_host = '127.0.0.1' memqcache_memcached_port = 11211 memqcache_total_size = 128MB memqcache_expire = 3600 memqcache_auto_cache_invalidation = on memqcache_maxcache = 512kB memqcache_cache_block_size = 1MB | ||||
| Additional Information | Tried to collect core-dumps (attached on the bug), but backtrace is not loading any symbols. Made some attampts to install some -dbgsym packages to address symbols issue with no luck. Core was generated by `pgpool: master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres fdb'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x000056268c219687 in ?? () (gdb) bt #0 0x000056268c219687 in ?? () 0000001 0x000056268c1f539b in ?? () 0000002 0x0000000000000042 in ?? () 0000003 0x000056268c1ecb45 in ?? () 0000004 0x000056268d04bac8 in ?? () 0000005 0x00007f6ba8e1c180 in ?? () 0000006 0x000056268d04bac8 in ?? () 0000007 0x00007ffff994b85c in ?? () 0000008 0x00007ffff994b85a in ?? () 0000009 0x000056268c1f27ed in ?? () 0000010 0x0000000000000000 in ?? () (gdb) bt full #0 0x000056268c219687 in ?? () No symbol table info available. 0000001 0x000056268c1f539b in ?? () No symbol table info available. 0000002 0x0000000000000042 in ?? () No symbol table info available. 0000003 0x000056268c1ecb45 in ?? () No symbol table info available. 0000004 0x000056268d04bac8 in ?? () No symbol table info available. 0000005 0x00007f6ba8e1c180 in ?? () No symbol table info available. 0000006 0x000056268d04bac8 in ?? () No symbol table info available. 0000007 0x00007ffff994b85c in ?? () No symbol table info available. 0000008 0x00007ffff994b85a in ?? () No symbol table info available. 0000009 0x000056268c1f27ed in ?? () No symbol table info available. 0000010 0x0000000000000000 in ?? () No symbol table info available. | ||||
| Tags | No tags attached. | ||||
|
|
|
|
|
Unfortunately the stack trace has not any useful information because of lacking symbol tables. Can you add install symbol tables (in case you install from binary package) or recompile with enable debug symbol? |
|
|
> ERROR: Sorry, too many clients already This is a side effect of the segfault. Pgpool-II process was terminated by the segfault before counting down the connection counter. If the segfault problem is solved, the problem should be gone as well. |
|
|
Managed to install debug symbols successfully this time. Below is backtrace from one of the core dump. Also attached the traces from few other processes. gdb /usr/local/pgpool/bin/pgpool core-pgpool.31874.vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2.1617120616 GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/pgpool/bin/pgpool...done. [New LWP 31874] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `pgpool: master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres fdb'. Program terminated with signal SIGSEGV, Segmentation fault. #0 GetMemoryChunkContext (pointer=0xffffffff00000113) at ../../src/include/utils/memutils.h:118 118 context = *(MemoryContext *) (((char *) pointer) - sizeof(void *)); (gdb) bt full #0 GetMemoryChunkContext (pointer=0xffffffff00000113) at ../../src/include/utils/memutils.h:118 context = 0xffffffff00000113 context = <optimized out> 0000001 pfree (pointer=0xffffffff00000113) at ../../src/utils/mmgr/mcxt.c:954 context = <optimized out> 0000002 0x0000561f1f4d9ed7 in pool_discard_buffer (buffer=0x561f1fd1efa8) at query_cache/pool_memqcache.c:3270 No locals. 0000003 0x0000561f1f4dc3b7 in pool_discard_temp_query_cache (temp_cache=0x561f1fd5cf28) at query_cache/pool_memqcache.c:3152 No locals. 0000004 0x0000561f1f4d3b45 in CloseComplete (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2981c0) at protocol/pool_proto_modules.c:2352 session_context = <optimized out> status = POOL_CONTINUE kind = <optimized out> name = 0x561f1fd19650 "C_284" 0000005 0x0000561f1f4d97ed in ProcessBackendResponse (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2981c0, state=state@entry=0x7ffc56abf1bc, num_fields=num_fields@entry=0x7ffc56abf1ba) at protocol/pool_proto_modules.c:2948 status = 0 kind = 51 '3' 0000006 0x0000561f1f4ca3d2 in pool_process_query (frontend=0x561f1faf3ac8, backend=0x7f14cd2981c0, reset_request=reset_request@entry=0) at protocol/pool_process_query.c:326 num_fields = 0 status = <optimized out> qcnt = <optimized out> i = <optimized out> ProcessQueryContext = 0x7f14cd29d140 state = 0 cont = <optimized out> 0000007 0x0000561f1f4c35ac in do_child (fds=fds@entry=0x561f1fc28960) at protocol/child.c:445 status = <optimized out> sp = 0x7f14cd2957f8 front_end_fd = <optimized out> saddr = {addr = {ss_family = 2, __ss_padding = "\335\316\177\000\000\001", '\000' <repeats 111 times>, __ss_align = 0}, salen = 16} con_count = <optimized out> local_sigjmp_buf = {{__jmpbuf = {94691676817768, 8749050718718474587, 47, 140721762598992, 4, 94691676829152, 3074917343839074651, 8749050176441629019}, __mask_was_saved = 1, __saved_mask = {__val = {0, 140721762596256, 1056, 94691669751903, 0, 3432, 94691669751904, 94691669751902, 0, 140721762596704, 94691669877194, 140721762597568, 140721762598048, 3, 140721762597584, 3}}}} backend = 0x7f14cd2981c0 now = {tv_sec = 1617120489, tv_usec = 315755} tz = {tz_minuteswest = 0, tz_dsttime = 0} timeout = {tv_sec = 297, tv_usec = 60989} connected = 1 connections_count = 0 psbuf = "master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres fdb 127.0.0.1(56782) idle\000\000\000\240\373\253V\374\177\000\000\001\000\000\000\000\000\000\000\346\003\000\000\000\000\000\000\371\377\240\313\024\177\000\000\001\200\255\373\000\000\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\326\032\257\037\037V\000\000\267\036\257\037\037V\000\000\321\032\257\037\037V\000\000\267\036\257\037\037V", '\000' <repeats 42 times>... walk = <optimized out> 0000008 0x0000561f1f49b217 in fork_a_child (fds=0x561f1fc28960, id=47) at main/pgpool_main.c:675 pid = 0 elevel_ = <optimized out> 0000009 0x0000561f1f4a1b19 in PgpoolMain (discard_status=<optimized out>, clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:406 i = 47 local_sigjmp_buf = {{__jmpbuf = {140721762599456, 2147483649, 140721762598816, 140721762598832, 94691676706168, 0, 5, 140721762649736}, __mask_was_saved = -1, __saved_mask = {__val = {3871410228, 139727276641000, 94691676705264, 140721762599304, 140721762599296, 140721762599392, 16, 94691676790368, 139727292613309, 94691675532912, 140721762599392, 140721762599384, 140721762599376, 94691676790368, 139727278813772, 94691676829744}}}} first = 1 '\001' 0000010 0x0000561f1f499380 in main (argc=<optimized out>, argv=0x7ffc56acc688) at main/main.c:364 opt = <optimized out> debug_level = 0 optindex = 0 discard_status = 0 '\000' ---Type <return> to continue, or q <return> to quit--- clear_memcache_oidmaps = 0 '\000' pcp_conf_file_path = "/usr/local/pgpool/etc/pcp.conf", '\000' <repeats 8162 times> conf_file_path = "/usr/local/pgpool/etc/pgpool.conf", '\000' <repeats 8159 times> hba_file_path = "/usr/local/pgpool/etc/pool_hba.conf", '\000' <repeats 8157 times> pool_passwd_key_file_path = "/root/.pgpoolkey", '\000' <repeats 4104 times>... long_options = {{name = 0x561f1f56a97b "hba-file", has_arg = 1, flag = 0x0, val = 97}, {name = 0x561f1f56a984 "debug", has_arg = 0, flag = 0x0, val = 100}, {name = 0x561f1f56a98a "config-file", has_arg = 1, flag = 0x0, val = 102}, {name = 0x561f1f56a996 "key-file", has_arg = 1, flag = 0x0, val = 107}, {name = 0x561f1f56a99f "pcp-file", has_arg = 1, flag = 0x0, val = 70}, {name = 0x561f1f56a9a8 "help", has_arg = 0, flag = 0x0, val = 104}, { name = 0x561f1f570407 "mode", has_arg = 1, flag = 0x0, val = 109}, {name = 0x561f1f56a9ad "dont-detach", has_arg = 0, flag = 0x0, val = 110}, {name = 0x561f1f56a9b9 "discard-status", has_arg = 0, flag = 0x0, val = 68}, { name = 0x561f1f56a9c8 "clear-oidmaps", has_arg = 0, flag = 0x0, val = 67}, {name = 0x561f1f56a9d6 "debug-assertions", has_arg = 0, flag = 0x0, val = 120}, {name = 0x561f1f57df6c "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} bt_full_core-pgpool.31830 (4,881 bytes)
root@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2:/data/tmp/pgpool-coredumps# gdb /usr/local/pgpool/bin/pgpool core-pgpool.31830.vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2.1617120676
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/pgpool/bin/pgpool...done.
[New LWP 31830]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `pgpool: master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres fdb'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 GetMemoryChunkContext (pointer=0xffffffff00000113) at ../../src/include/utils/memutils.h:118
118 context = *(MemoryContext *) (((char *) pointer) - sizeof(void *));
(gdb) bt full
#0 GetMemoryChunkContext (pointer=0xffffffff00000113) at ../../src/include/utils/memutils.h:118
context = 0xffffffff00000113
context = <optimized out>
#1 pfree (pointer=0xffffffff00000113) at ../../src/utils/mmgr/mcxt.c:954
context = <optimized out>
#2 0x0000561f1f4d9ed7 in pool_discard_buffer (buffer=0x561f1fd1fde8) at query_cache/pool_memqcache.c:3270
No locals.
#3 0x0000561f1f4dc3b7 in pool_discard_temp_query_cache (temp_cache=0x561f1fd21938) at query_cache/pool_memqcache.c:3152
No locals.
#4 0x0000561f1f4d3b45 in CloseComplete (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2981c0) at protocol/pool_proto_modules.c:2352
session_context = <optimized out>
status = POOL_CONTINUE
kind = <optimized out>
name = 0x561f1fd21380 "C_38"
#5 0x0000561f1f4d97ed in ProcessBackendResponse (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2981c0, state=state@entry=0x7ffc56abf1bc, num_fields=num_fields@entry=0x7ffc56abf1ba)
at protocol/pool_proto_modules.c:2948
status = 0
kind = 51 '3'
#6 0x0000561f1f4ca3d2 in pool_process_query (frontend=0x561f1faf3ac8, backend=0x7f14cd2981c0, reset_request=reset_request@entry=0) at protocol/pool_process_query.c:326
num_fields = 0
status = <optimized out>
qcnt = <optimized out>
i = <optimized out>
ProcessQueryContext = 0x7f14cd29d140
state = 0
cont = <optimized out>
#7 0x0000561f1f4c35ac in do_child (fds=fds@entry=0x561f1fc28960) at protocol/child.c:445
status = <optimized out>
sp = 0x7f14cd2957f8
front_end_fd = <optimized out>
saddr = {addr = {ss_family = 2, __ss_padding = "\341\b\177\000\000\001", '\000' <repeats 111 times>, __ss_align = 0}, salen = 16}
con_count = <optimized out>
local_sigjmp_buf = {{__jmpbuf = {94691676817768, 8749050718718474587, 3, 140721762598992, 4, 94691676829152, 3074917343839074651, 8749050176441629019}, __mask_was_saved = 1, __saved_mask = {__val = {
0, 140721762596256, 1056, 94691669751903, 0, 3432, 94691669751904, 94691669751902, 0, 140721762596704, 94691669877194, 140721762597568, 140721762598048, 3, 140721762597584, 3}}}}
backend = 0x7f14cd2981c0
now = {tv_sec = 1617120489, tv_usec = 306716}
tz = {tz_minuteswest = 0, tz_dsttime = 0}
timeout = {tv_sec = 152, tv_usec = 567970}
connected = 1
connections_count = 0
psbuf = "master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres fdb 127.0.0.1(57608) idle\000\000\000\240\373\253V\374\177\000\000\001\000\000\000\000\000\000\000\346\003\000\000\000\000\000\000\371\377\240\313\024\177\000\000\001\200\255\373\000\000\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\326\032\257\037\037V\000\000\267\036\257\037\037V\000\000\321\032\257\037\037V\000\000\267\036\257\037\037V", '\000' <repeats 42 times>...
walk = <optimized out>
#8 0x0000561f1f49b217 in fork_a_child (fds=0x561f1fc28960, id=3) at main/pgpool_main.c:675
pid = 0
elevel_ = <optimized out>
#9 0x0000561f1f4a1b19 in PgpoolMain (discard_status=<optimized out>, clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:406
---Type <return> to continue, or q <return> to quit--- bt_full_core-pgpool.711 (7,351 bytes)
root@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2:/data/tmp/pgpool-coredumps# gdb /usr/local/pgpool/bin/pgpool core-pgpool.711.vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2.1617120982
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/pgpool/bin/pgpool...done.
[New LWP 711]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `pgpool: master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres edb'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 GetMemoryChunkContext (pointer=0xffffffff00000114) at ../../src/include/utils/memutils.h:118
118 context = *(MemoryContext *) (((char *) pointer) - sizeof(void *));
(gdb) bt full
#0 GetMemoryChunkContext (pointer=0xffffffff00000114) at ../../src/include/utils/memutils.h:118
context = 0xffffffff00000114
context = <optimized out>
#1 pfree (pointer=0xffffffff00000114) at ../../src/utils/mmgr/mcxt.c:954
context = <optimized out>
#2 0x0000561f1f4d9ed7 in pool_discard_buffer (buffer=0x561f1fd14f28) at query_cache/pool_memqcache.c:3270
No locals.
#3 0x0000561f1f4dc3b7 in pool_discard_temp_query_cache (temp_cache=0x561f1fd0ff80) at query_cache/pool_memqcache.c:3152
No locals.
#4 0x0000561f1f4d3b45 in CloseComplete (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2981c0) at protocol/pool_proto_modules.c:2352
session_context = <optimized out>
status = POOL_CONTINUE
kind = <optimized out>
name = 0x561f1fd108e8 "C_22"
#5 0x0000561f1f4d97ed in ProcessBackendResponse (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2981c0, state=state@entry=0x7ffc56abf15c, num_fields=num_fields@entry=0x7ffc56abf15a)
at protocol/pool_proto_modules.c:2948
status = 0
kind = 51 '3'
#6 0x0000561f1f4ca3d2 in pool_process_query (frontend=0x561f1faf3ac8, backend=0x7f14cd2981c0, reset_request=reset_request@entry=0) at protocol/pool_process_query.c:326
num_fields = 0
status = <optimized out>
qcnt = <optimized out>
i = <optimized out>
ProcessQueryContext = 0x7f14cd29d140
state = 0
cont = <optimized out>
#7 0x0000561f1f4c35ac in do_child (fds=fds@entry=0x561f1fc28960) at protocol/child.c:445
status = <optimized out>
sp = 0x7f14cd2957f8
front_end_fd = <optimized out>
saddr = {addr = {ss_family = 2, __ss_padding = "\344\266\177\000\000\001", '\000' <repeats 111 times>, __ss_align = 0}, salen = 16}
con_count = <optimized out>
local_sigjmp_buf = {{__jmpbuf = {94691676817768, 8749050718718474587, 15, 140721762598384, 94691669751097, 31842, 3074917343927155035, 8749050176441629019}, __mask_was_saved = 1, __saved_mask = {__val = {0, 140721762596160,
1056, 94691669751903, 0, 3432, 94691669751904, 94691669751902, 0, 0, 94691669877194, 140721762597472, 140721762597952, 7, 94691669319912, 3}}}}
backend = 0x7f14cd2981c0
now = {tv_sec = 1617120848, tv_usec = 33569}
tz = {tz_minuteswest = 0, tz_dsttime = 0}
timeout = {tv_sec = 195, tv_usec = 215186}
connected = 1
connections_count = 0
psbuf = "master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres edb 127.0.0.1(58550) idle\000\000\000@\373\253V\374\177\000\000\001\000\000\000\000\000\000\000\346\003\000\000\000\000\000\000\371\377\240\313\024\177\000\000\001\200\255\373\000\000\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\326\032\257\037\037V\000\000\267\036\257\037\037V\000\000\321\032\257\037\037V\000\000\267\036\257\037\037V", '\000' <repeats 46 times>...
walk = <optimized out>
#8 0x0000561f1f49b217 in fork_a_child (fds=0x561f1fc28960, id=15) at main/pgpool_main.c:675
pid = 0
elevel_ = <optimized out>
#9 0x0000561f1f49bbd7 in reaper () at main/pgpool_main.c:2417
exiting_process_name = <optimized out>
new_pid = 0
shutdown_system = <optimized out>
restart_child = <optimized out>
found = 1 '\001'
process_health_check = 0 '\000'
pid = <optimized out>
status = 139
i = 15
---Type <return> to continue, or q <return> to quit---
#10 0x0000561f1f4a220f in PgpoolMain (discard_status=<optimized out>, clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:469
i = 1
local_sigjmp_buf = {{__jmpbuf = {94691673121984, 3074917343449004379, 94691673198784, 0, 4, 94691676829152, 3074917343442712923, 8749050193819941211}, __mask_was_saved = 1, __saved_mask = {__val = {18446744066192964103,
139727276641000, 94691676705264, 140721762599304, 140721762599296, 140721762599392, 16, 94691676790368, 139727292613309, 94691675532912, 140721762599392, 140721762599384, 140721762599376, 94691676790368, 139727278813772,
94691676829744}}}}
first = 0 '\000'
#11 0x0000561f1f499380 in main (argc=<optimized out>, argv=0x7ffc56acc688) at main/main.c:364
opt = <optimized out>
debug_level = 0
optindex = 0
discard_status = 0 '\000'
clear_memcache_oidmaps = 0 '\000'
pcp_conf_file_path = "/usr/local/pgpool/etc/pcp.conf", '\000' <repeats 8162 times>
conf_file_path = "/usr/local/pgpool/etc/pgpool.conf", '\000' <repeats 8159 times>
hba_file_path = "/usr/local/pgpool/etc/pool_hba.conf", '\000' <repeats 8157 times>
pool_passwd_key_file_path = "/root/.pgpoolkey", '\000' <repeats 4104 times>...
long_options = {{name = 0x561f1f56a97b "hba-file", has_arg = 1, flag = 0x0, val = 97}, {name = 0x561f1f56a984 "debug", has_arg = 0, flag = 0x0, val = 100}, {name = 0x561f1f56a98a "config-file", has_arg = 1, flag = 0x0,
val = 102}, {name = 0x561f1f56a996 "key-file", has_arg = 1, flag = 0x0, val = 107}, {name = 0x561f1f56a99f "pcp-file", has_arg = 1, flag = 0x0, val = 70}, {name = 0x561f1f56a9a8 "help", has_arg = 0, flag = 0x0, val = 104}, {
name = 0x561f1f570407 "mode", has_arg = 1, flag = 0x0, val = 109}, {name = 0x561f1f56a9ad "dont-detach", has_arg = 0, flag = 0x0, val = 110}, {name = 0x561f1f56a9b9 "discard-status", has_arg = 0, flag = 0x0, val = 68}, {
name = 0x561f1f56a9c8 "clear-oidmaps", has_arg = 0, flag = 0x0, val = 67}, {name = 0x561f1f56a9d6 "debug-assertions", has_arg = 0, flag = 0x0, val = 120}, {name = 0x561f1f57df6c "version", has_arg = 0, flag = 0x0,
val = 118}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} bt_full_core-pgpool.32086 (26,202 bytes)
root@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2:/data/tmp/pgpool-coredumps# gdb /usr/local/pgpool/bin/pgpool core-pgpool.32086.vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2.1617120857
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/pgpool/bin/pgpool...done.
[New LWP 32086]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `pgpool: master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres edb'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:316
316 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt full
#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:316
No locals.
#1 0x0000561f1f4da392 in memcpy (__len=4, __src=0x7ffc56abc964, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
No locals.
#2 pool_add_buffer (buffer=0x561f1feae4d8, data=data@entry=0x7ffc56abc964, len=len@entry=4) at query_cache/pool_memqcache.c:3302
session_context = <optimized out>
old_context = <optimized out>
#3 0x0000561f1f4dfab8 in pool_add_buffer (len=4, data=0x7ffc56abc964, buffer=<optimized out>) at query_cache/pool_memqcache.c:3234
session_context = <optimized out>
old_context = <optimized out>
session_context = <optimized out>
old_context = <optimized out>
allocate_size = <optimized out>
elevel_ = <optimized out>
elevel_ = <optimized out>
#4 pool_add_oids_temp_query_cache (oids=0x7ffc56abc964, num_oids=1, temp_cache=0x561f1fea8818) at query_cache/pool_memqcache.c:3234
buffer = <optimized out>
buffer = <optimized out>
#5 pool_handle_query_cache (backend=backend@entry=0x7f14cd2989d0,
query=0x561f1fec3be8 "select farmpowers0_.farm_id as farm_id2_24_0_, farmpowers0_.ID as id1_24_0_, farmpowers0_.ID as id1_24_1_, farmpowers0_.farm_id as farm_id2_24_1_, farmpowers0_.name as name3_24_1_, farmpowers0_.start_"...,
node=node@entry=0x561f1fec14f8, state=<optimized out>) at query_cache/pool_memqcache.c:3550
cache = <optimized out>
ctx = {has_system_catalog = false, has_temp_table = false, has_unlogged_table = false, has_view = false, has_function_call = false, pg_terminate_backend_pid = 0, has_non_immutable_function_call = false,
has_insertinto_or_locking_clause = false, num_oids = 1, table_oids = {24955, 24233, 0 <repeats 107 times>, 535410056, 22047, 525260128, 22047, 1454099792, 32764, 1, 0, 0, 0, 529211136, 22047, 525246368, 22047, 0, 0, 535410216,
22047, 525260128}, table_names = {"\"farm_power_schedule\"\000lancing_configuration\"\000\000\000\000\000\000\000\000\070\276\352\037\037V\000\000`\325N\037",
"\037V\000\000PͫV\374\177", '\000' <repeats 19 times>, "\037\213\037\037V\000\000\240\237N\037\037V\000\000\000\000\000\000\000\000\000\000ؾ\352\037\037V\000\000`\325N\037",
"\037V\000\000PͫV\374\177", '\000' <repeats 19 times>, "\037\213\037\037V\000\000`\241N\037\037V\000\000\000\000\000\000\000\000\000\000\320\363\354\037\037V\000\000`\325N\037",
"\037V\000\000PͫV\374\177", '\000' <repeats 19 times>, "\037\213\037\037V\000\000\240\237N\037\037V\000\000\200/\213\037\037V\000\000`0\354\037\037V\000\000`\325N\037",
"\037V\000\000PͫV\374\177\000\000\016", '\000' <repeats 16 times>, "\037\213\037\037V\000\000\240\237N\037\037V\000\000\000\000\000\000\000\000\000\000\310\062\354\037\037V\000\000`\325N\037",
"\037V\000\000PͫV\374\177\000\000\310\063\354\037\037V", '\000' <repeats 11 times>, "\037\213\037\037V\000\000\240\237N\037\037V\000\000\000\000\000\000\000\000\000\000\370\024\354\037\037V\000\000`\325N\037",
"\037V\000\000PͫV\374\177\000\000\310\063\354\037\037V", '\000' <repeats 11 times>, "\037\213\037\037V\000\000\360\243N\037\037V\000\000\000\000\000\000\000\000\000\000Љ)\315\024\177\000\000\310:\257\037",
"\037V\000\000\370\024\354\037\037V\000\000\310\063\354\037\037V", '\000' <repeats 11 times>, "\037\213\037\037V\000\000\374\302N\037\037V\000\000\000\000\000\000\000\000\000\000\377\377\377\377\000\000\000\000\000\000\000",
'\000' <repeats 63 times>, '\000' <repeats 36 times>, "\333FB\314\024\177\000\000x\247\354\037\037V\000\000{WB\314\024\177\000\000 n", <incomplete sequence \314>,
"\024\177\000\000h\022\347\037\037V\000\000\020\000\000\000\024\177\000\000@\021\347\037\037V\000\000\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\323\334\347\037\037V\000\000\340\020\347\037\037V\000\000\340\020\347\037",
"\037V\000\000\340\017\347\037\037V\000\000\300\334\347\037\037V\000\000\206XB\314\024\177\000\000\023\000\000\000\000\000\000\000\260@\350\037\037V\000\000\300\334\347\037\037V\000\000\206\070@\314\024\177\000\000\000\000\000", "\000\000\000\000\350@\350\037\037V\000\000\260@\350\037\037V\000\000\260@\350\037\037V\000\000\001\000\000\000\000\000\000\000\240-\347\037\037V\000\000\035ѫV\374\177\000\000\364٫V\374\177\000\000\030ϫV",
"\374\177\000\000\\gy\314\024\177\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\035ѫV\374\177\000\000\240٫V\374\177\000\000\000\000\000\000\020\000\000\000\277ҫV",
"\374\177\000\000PE\347\037\037V\000\000`\220t\314\024\177\000\000PE\347\037\037V\000\000OE\347\037\037V\000\000\020ϫV\374\177\000\000+\000\000\000\000\000\000\000H\376\304\363\356\324Bm\234\345\001\262",
"\373\212L3_φ\024\061\020\222\001\267\320\003\t|ܺ\345,\210\035\t7\327\277\311", '\000' <repeats 24 times>, "\333FB\314\024\177\000\000\000\000\000",
"\000\000\000\000{WB\314\024\177\000\000 nB\314\024\177\000\000bRB\314\024\177\000\000 nB\314\024\177\000\000\340\017\347\037\037V\000\000\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\320\334\347\037",
"\037V\000\000\340\020\347\037\037V\000\000\340\020\347\037\037V\000\000\340\017\347\037\037V\000\000\300\334\347\037\037V\000\000\206XB\314\024\177\000\000\020\000\000\000\000\000\000\000\260@\350\037\037V\000\000\300\334\347\037", "\037V\000\000\206\070@\314\024\177\000\000\322\334\347\037\037V\000\000\350@\350\037\037V\000\000\260@\350\037\037V\000\000\260@\350\037\037V\000\000\001\000\000\000\000\000\000\000\240-\347\037\037V\000\000\255ҫV",
"\374\177\000\000\204۫V\374\177\000\000\250ЫV\374\177\000\000\\gy\314\024\177\000\000\340\020\347\037\037V\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\255ҫV\374\177\000\000\060۫V",
"\374\177\000\000\260@\350\037\020\000\000\000OԫV\374\177\000\000PE\347\037\037V\000\000`\220t\314\024\177\000\000PE\347\037\037V\000\000OE\347\037\037V\000\000\240ЫV\374\177\000\000(\000\000",
"\000\000\000\000\240-\347\037\037V\000\000=ӫV\374\177\000\000\024ܫV\374\177\000\000\070ѫV\374\177\000\000\\gy\314\024\177\000\000`\220t\314\024\177\000\000\001\000\000\000\000\000\000\000\001\000\000",
"\000\000\000\000=ӫV\374\177\000\000\300۫V\374\177\000\000\370\260\300\037\020\000\000\000\337ԫV\374\177\000\000PE\347\037\037V\000\000\000\000\000\000\000\000\003\210\027\003\003\000\033V\000\000OE\347\037",
"\037V\000\000\060ѫV\374\177\000\000.\000\000\000\000\000\000\000@٫V\374\177\000\000\000\000\000\000\000\000\000\000\206XB\314\024\177\000\000\000\000\000\000\000\000\000\000\200\363\302\037\037V\000\000\222\322\317\037",
"\037V\000\000\004\243\062\314\024\177", '\000' <repeats 53 times>, '\000' <repeats 63 times>, '\000' <repeats 63 times>,
"\000\000\000\000{WB\314\024\177\000\000 nB\314\024\177\000\000\205l5\314\024\177\000\000\240ҫV\374\177\000\000\263\334\347\037\037V\000\000\060\000\000\000\000\000\000\000\n 5\314\024\177\000\000\020\330\346\037",
"\037V\000\000e\020\065\314\024\177\000\000\340\020\347\037\037V\000\000\370\065\347\037\037V\000\000\370\065\347\037\037V\000\000\300\266*\315\024\177\000\000X\344\253V\374\177\000\000\003\025\065\314\024\177\000\000\060\000\000", "\000\000\000\000\000\246\002Rϯ\216Y\000\000\000\000\000\000\000\000\267,y\314\024\177\000\000\000\000\000\000\000\000\000\000\060\000\000\000\000\000\000\000\240ӫV\374\177\000\000\240-\347\037\037V\000\000(\000\000",
"\000\000\000\000\240٫V\374\177\000\000\320ӫV\374\177\000\000\001\000\000\000\000\000\000\000\220ӫV\374\177\000\000\325;y\314\024\177\000\000\001\000\000\000\027\000\000\000\210ӫV\374\177\000\000\000\035", <incomplete sequence \315>, "\024\177\000\000X\344\253V\374\177\000\000\023\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\230ӫV\374\177", '\000' <repeats 18 times>, "\001\000\000",
"\000\000\000\000\220ӫV\374\177\000\000\240٫V\374\177\000\000\230ӫV\374\177\000\000\000\000\000\000\000\000\000\000+\000\000\000\000\000\000\000\033\000\000\000\000\000\000\000\325\343\332)\215>\320Z\000\000\000",
"\000\000\000\000\260\334\347\037\037V\000\000\063\000\000\000\000\000\000\000\063\000\000\000\000\000\000\000\260@", '\000' <repeats 14 times>, "\060ԫV\374\177\000\000\263\334\347\037\037V\000\000-\000\000",
"\000\000\000\000\n 5\314\024\177\000\000\020\330\346\037\037V\000\000e\020\065\314\024\177\000\000\000\000\000\000\000\000\000\000\370\065\347\037\037V\000\000\370\065\347\037\037V\000\000\300\266*\315\024\177\000\000\350\345---Type <return> to continue, or q <return> to quit---
\253V", "\374\177\000\000\003\025\065\314\024\177\000\000-\000\000\000\000\000\000\000\000\246\002Rϯ\216Y\000\000\000\000\000\000\000\000\267,y\314\024\177\000\000\000\000\000\000\000\000\000\000-\000\000\000\000\000\000\000\060իV",
"\374\177\000\000\240-\347\037\037V\000\000(\000\000\000\000\000\000\000\060۫V\374\177\000\000`իV\374\177\000\000\001\000\000\000\000\000\000\000 իV\374\177\000\000\325;y\314\024\177\000\000p\256\300\037",
"\027\000\000\000\030իV\374\177\000\000\000\035*\315\024\177\000\000\350\345\253V\374\177\000\000\020\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000(իV\374\177\000\000\000\000\000", '\000' <repeats 12 times>, "\001\000\000\000\000\000\000\000 իV\374\177\000\000\060۫V\374\177\000\000(իV\374\177\000\000\000\000\000\000\000\000\000\000(\000\000\000\000\000\000\000\030\000\000",
"\000\000\000\000\001", '\000' <repeats 15 times>, "\260\334\347\037\037V\000\000\060\000\000\000\000\000\000\000\060\000\000\000\000\000\000\000\260@", '\000' <repeats 17 times>,
"\000\000\003\207\027\003\003\000N\177\000\000\360H\301\037\037V\000\000e\020\065\314\024\177\000\000\000\000\000\000\000\000\000\000\350\024\260\037\037V\000\000\350\024\260\037\037V\000\000\300\266*\315\024\177\000\000h\347\253V", "\374\177\000\000\003\025\065\314\024\177\000\000\v\002\000\000\000\000\000\000\000\246\002Rϯ\216Y\000\000\000\000\000\000\000\000\267,y\314\024\177\000\000\000\000\000\000\000\000\000\000\v\002\000\000\000\000\000\000\260֫V",
"\374\177\000\000\220\f\260\037\037V\000\000(\000\000\000\000\000\000\000\260ܫV\374\177\000\000\340֫V\374\177\000\000\001\000\000\000\000\000\000\000\240֫V\374\177\000\000\325;y\314\024\177\000\000&\000\000",
"\027\000\000\000\230֫V\374\177\000\000\370\024\303\037\037V\000\000h\347\253V\374\177\000\000\365\001\000\000\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\250֫V\374\177\000\000\000\000\000", '\000' <repeats 12 times>, "\001\000\000\000\000\000\000\000\240֫V\374\177\000\000\260ܫV\374\177\000\000\205l5\314\024\177\000\000\360֫V\374\177\000\000\263\334\347\037\037V\000\000c\000\000",
"\000\000\000\000\n 5\314\024\177\000\000\020\330\346\037\037V\000\000e\020\065\314\024\177\000\000\016\002\000\000\000\000\000\000\370\065\347\037\037V\000\000\370\065\347\037\037V\000\000\300\266*\315\024\177\000\000\250\350\253V",
"\374\177\000\000\003\025\065\314\024\177\000\000c\000\000\000\000\000\000\000\000\246\002Rϯ\216Y\000\000\000\000\000\000\000\000\267,y\314\024\177\000\000\000\000\000\000\000\000\000\000c\000\000\000\000\000\000\000\360\327\253V", "\374\177\000\000\240-\347\037\037V\000\000(\000\000\000\000\000\000\000\333FB\314\024\177\000\000 ثV\374\177\000\000{WB\314\024\177\000\000 nB\314\024\177\000\000h\022\347\037\037V\000\000 \000\000",
"\024\177\000\000@\021\347\037\037V\000\000\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\305\334\347\037\037V\000\000\340\020\347\037\037V\000\000\340\020\347\037\037V\000\000\340\017\347\037\037V\000\000\300\334\347\037",
"\037V\000\000\206XB\314\024\177\000\000\005\000\000\000\000\000\000\000\260@\350\037\037V\000\000\300\334\347\037\037V\000\000\206\070@\314\024\177\000\000\000\000\000\000\000\000\000\000\350@\350\037\037V\000\000\260@\350\037", "\037V\000\000\260@\350\037\037V\000\000\001\000\000\000\000\000\000\000\240-\347\037\037V\000\000\215ګV\374\177\000\000d\343\253V\374\177\000\000\210ثV\374\177\000\000\\gy\314\024\177\000\000\000\000\000",
"\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\215ګV\374\177\000\000\020\343\253V\374\177\000\000\000\000\000\000\020\000\000\000/ܫV\374\177\000\000PE\347\037\037V\000\000`\220", <incomplete sequence \314>, "\024\177\000\000PE\347\037\037V\000\000OE\347\037\037V\000\000\200ثV\374\177\000\000\035", '\000' <repeats 34 times>, '\000' <repeats 63 times>,
'\000' <repeats 19 times>, "\002]\033B\314\024\177", '\000' <repeats 18 times>, "\345\205\312\177\274\251EcC%\356Uc\231\257\307\067\021i\037",
"\215;z\032K猪WL\363\201]F\270\256\240\275\333\027\353\365k\225}]7\031~\371%<d\t\302\024\372\273q\347,b(o\270̚\005(X\240\243\063\334\022\273J\370\235\227\346}",
"\ta\200\347[rD\255#4\224\262\341\320.q\342\226:\347B\270j\265\065X\024\255\222\000\000\000\000\000\000\000\201RB\314\024\177\000\000 nB\314\024\177\000\000p\256\300\037\037V\000\000\020\000\000",
"\000\000\000\000{WB\314\024\177\000\000 nB\314\024\177\000\000\370\260\300\037\037V\000\000\020\000\000\000\037V\000\000Я\300\037\037V\000\000\200\363\302\037\037V\000\000\201RB\314\024\177\000\000 n", <incomplete sequence \314>, "\024\177\000\000p\256\300\037\037V\000\000\000\000\000\000\000\000\000\000{WB\314\024\177\000\000 nB\314\024\177\000\000\370\260\300\037\037V\000\000 \000\000\000\000\000\000\000Я\300\037\037V\000\000\200\363\302\037",
"\037V\000\000\020\000\000\000\000\000\000\000\270\363\302\037\037V\000\000p\257\300\037\037V\000\000p\256\300\037\037V\000\000P\342\253V\374\177\000\000\000\000\000\000\000\000\000\000\206XB\314\024\177\000\000\000\000\000", "\000\000\000\000\200\363\302\037\037V\000\000\217\322\317\037\037V\000\000\035\070@\314\024\177\000\000\210\322\317\037\037V\000\000\020۫V\374\177\000\000\001\000\000\000\000\000\000\000\024۫V\374\177\000\000\220\f\260\037",
"\037V\000\000\070/@\314\024\177\000\000\001\000\000\000\000\000\000\000\220\f\260\037\037V\000\000\200\363\302\037\037V\000\000\001\000\000\000\000\000\000\000P\342\253V\374\177\000\000\227\032A\314\024\177\000\000\220\f\260\037", "\037V\000\000\205l5\314\024\177\000\000P۫V\374\177\000\000\203\322\317\037\037V\000\000\034\000\000\000\000\000\000\000\n 5\314\024\177\000\000\360H\301\037\037V\000\000e\020\065\314\024\177\000\000\000\000\000",
"\000\000\000\000\350\024\260\037\037V\000\000\350\024\260\037\037V\000\000\300\266*\315\024\177\000\000\b\355\253V\374\177\000\000\003\025\065\314\024\177\000\000\034\000\000\000\000\000\000\000\000\246\002Rϯ\216Y\000\000\000", "\000\000\000\000\267,y\314\024\177\000\000\000\000\000\000\000\000\000\000\034\000\000\000\000\000\000\000PܫV\374\177\000\000\220\f\260\037\037V\000\000(\000\000\000\000\000\000\000P\342\253V\374\177\000\000\200ܫV",
"\374\177\000\000\001\000\000\000\000\000\000\000@ܫV\374\177\000\000\325;y\314\024\177\000\000\"\000\000\000\027\000\000\000\070ܫV\374\177\000\000\370\024\303\037\037V\000\000\b\355\253V\374\177\000\000\006\000\000",
"\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000HܫV\374\177", '\000' <repeats 18 times>, "\001\000\000\000\000\000\000\000@ܫV\374\177\000\000P\342\253V",
"\374\177\000\000HܫV\374\177\000\000\000\000\000\000\000\000\000\000\027\000\000\000\000\000\000\000\a\000\000\000\000\000\000\000\240-\347\037\037V\000\000\000\000\000\000\000\000\000\000\200\322\317\037\037V\000\000\037\000\000", "\000\000\000\000\037\000\000\000\000\000\000\000\260@", '\000' <repeats 18 times>, "\027\000\000\000\370ܫV\374\177\000\000\000\035*\315\024\177\000\000\310\355\253V\374\177\000\000\005\000\000",
"\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\bݫV\374\177", '\000' <repeats 18 times>, "\001\000\000\000\000\000\000\000\000ݫV\374\177\000\000\020\343\253V",
"\374\177\000\000\bݫV\374\177\000\000\000\000\000\000\000\000\000\000\035\000\000\000\000\000\000\000\r", '\000' <repeats 23 times>, "\260\334\347\037\037V\000\000%\000\000",
"\000\000\000\000%\000\000\000\000\000\000\000\260@", '\000' <repeats 49 times>, '\000' <repeats 63 times>, '\000' <repeats 63 times>, '\000' <repeats 63 times>, '\000' <repeats 63 times>,
'\000' <repeats 20 times>, "\333FB\314\024\177\000\000\000\000\000\000\000\000\000\000{WB\314\024\177\000\000 nB\314\024\177\000\000XL\350\037\037V\000\000 \000\000",
"\024\177\000\000\060K\350\037\037V\000\000\020K\350\037\037V\000\000\020\000\000\000\000\000\000\000X\214\346\037\037V\000\000\320J\350\037\037V\000\000\320J\350\037\037V\000\000\320I\350\037\037V\000\000p\233\347\037",
"\037V\000\000\206XB\314\024\177\000\000\016\000\000\000\000\000\000\000X\214\346\037\037V\000\000p\233\347\037\037V\000\000\323\067@\314\024\177\000\000\000\000\000\000\000\000\000\000X\214\346\037\037V\000\000 \214\346\037", "\037V\000\000 \214\346\037\037V\000\000\000\000\000\000\000\000\000\000\240-\347\037\037V\000\000\315\341\253V\374\177\000\000L;\347\037\037V\000\000\310߫V\374\177\000\000>iy\314\024\177\000\000\000\000\000",
"\000\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\315\341\253V\374\177\000\000\370:\347\037\037V\000\000\000\000\000\000\020\000\000\000o\343\253V\374\177\000\000HE\347\037\037V\000\000`\220", <incomplete sequence \314>, "\024\177\000\000PE\347\037\037V\000\000GE\347\037\037V\000\000\300߫V\374\177\000\000&", '\000' <repeats 34 times>, '\000' <repeats 63 times>, '\000' <repeats 63 times>, '\000' <repeats 63 times>,
'\000' <repeats 63 times>, '\000' <repeats 63 times>, '\000' <repeats 63 times>, '\000' <repeats 63 times>, '\000' <repeats 34 times>, "\003\210\027\003\003\000&", '\000' <repeats 22 times>,
'\000' <repeats 28 times>, "\003\003\000\000\027\000\000\000.", '\000' <repeats 23 times>, "\210\322\317\037",
"\037V\000\000\210\322\317\037\037V", '\000' <repeats 34 times>, "\003\003\000\000\027\000\000\000\034\000\000\000\000\000\000\000\000\000\000",
'\000' <repeats 12 times>, "\210\322\317\037\037V\000\000\210\322\317\037\037V", '\000' <repeats 37 times>, '\000' <repeats 63 times> <repeats 35 times>}}
old_context = <optimized out>
session_context = 0x561f1f8b1f00 <session_context_d>
oldmask = {__val = {0, 662054688, 0 <repeats 14 times>}}
cache_buffer = <optimized out>
len = 67
num_oids = 1
oids = 0x7ffc56abc964
i = <optimized out>
#6 0x0000561f1f4d7141 in ReadyForQuery (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2989d0, send_ready=send_ready@entry=1 '\001', cache_commit=cache_commit@entry=1 '\001') at protocol/pool_proto_modules.c:2159
---Type <return> to continue, or q <return> to quit---
i = <optimized out>
len = 83886080
kind = 84 'T'
state = 84 'T'
session_context = <optimized out>
node = 0x561f1fec14f8
query = <optimized out>
got_estate = <optimized out>
#7 0x0000561f1f4d92f0 in ProcessBackendResponse (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2989d0, state=state@entry=0x7ffc56abf15c, num_fields=num_fields@entry=0x7ffc56abf15a)
at protocol/pool_proto_modules.c:3080
status = 0
kind = 90 'Z'
#8 0x0000561f1f4ca3d2 in pool_process_query (frontend=0x561f1faf3ac8, backend=0x7f14cd2989d0, reset_request=reset_request@entry=0) at protocol/pool_process_query.c:326
num_fields = 0
status = <optimized out>
qcnt = <optimized out>
i = <optimized out>
ProcessQueryContext = 0x7f14cd29cf20
state = 0
cont = <optimized out>
#9 0x0000561f1f4c35ac in do_child (fds=fds@entry=0x561f1fc28960) at protocol/child.c:445
status = <optimized out>
sp = 0x7f14cd294ac0
front_end_fd = <optimized out>
saddr = {addr = {ss_family = 2, __ss_padding = "\342\242\177\000\000\001", '\000' <repeats 111 times>, __ss_align = 0}, salen = 16}
con_count = <optimized out>
local_sigjmp_buf = {{__jmpbuf = {94691676817768, 8749050718718474587, 26, 140721762598384, 94691669751097, 31853, 3074917343927155035, 8749050176441629019}, __mask_was_saved = 1, __saved_mask = {__val = {0, 140721762596160,
1056, 94691669751903, 0, 3432, 94691669751904, 94691669751902, 0, 0, 94691669877194, 140721762597472, 140721762597952, 7, 94691669319912, 3}}}}
backend = 0x7f14cd2989d0
now = {tv_sec = 1617120596, tv_usec = 751542}
tz = {tz_minuteswest = 0, tz_dsttime = 0}
timeout = {tv_sec = 123, tv_usec = 178334}
connected = 1
connections_count = 0
psbuf = "master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres edb 127.0.0.1(58018) idle\000\000\000@\373\253V\374\177\000\000\001\000\000\000\000\000\000\000\346\003\000\000\000\000\000\000\371\377\240\313\024\177\000\000\001\200\255\373\000\000\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\326\032\257\037\037V\000\000\267\036\257\037\037V\000\000\321\032\257\037\037V\000\000\267\036\257\037\037V", '\000' <repeats 46 times>...
walk = <optimized out>
#10 0x0000561f1f49b217 in fork_a_child (fds=0x561f1fc28960, id=26) at main/pgpool_main.c:675
pid = 0
elevel_ = <optimized out>
#11 0x0000561f1f49bbd7 in reaper () at main/pgpool_main.c:2417
exiting_process_name = <optimized out>
new_pid = 0
shutdown_system = <optimized out>
restart_child = <optimized out>
found = 1 '\001'
process_health_check = 0 '\000'
pid = <optimized out>
status = 139
i = 26
#12 0x0000561f1f4a220f in PgpoolMain (discard_status=<optimized out>, clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:469
i = 1
local_sigjmp_buf = {{__jmpbuf = {94691673121984, 3074917343449004379, 94691673198784, 0, 4, 94691676829152, 3074917343442712923, 8749050193819941211}, __mask_was_saved = 1, __saved_mask = {__val = {18446744066192964103,
139727276641000, 94691676705264, 140721762599304, 140721762599296, 140721762599392, 16, 94691676790368, 139727292613309, 94691675532912, 140721762599392, 140721762599384, 140721762599376, 94691676790368, 139727278813772,
94691676829744}}}}
first = 0 '\000'
---Type <return> to continue, or q <return> to quit---
#13 0x0000561f1f499380 in main (argc=<optimized out>, argv=0x7ffc56acc688) at main/main.c:364
opt = <optimized out>
debug_level = 0
optindex = 0
discard_status = 0 '\000'
clear_memcache_oidmaps = 0 '\000'
pcp_conf_file_path = "/usr/local/pgpool/etc/pcp.conf", '\000' <repeats 8162 times>
conf_file_path = "/usr/local/pgpool/etc/pgpool.conf", '\000' <repeats 8159 times>
hba_file_path = "/usr/local/pgpool/etc/pool_hba.conf", '\000' <repeats 8157 times>
pool_passwd_key_file_path = "/root/.pgpoolkey", '\000' <repeats 4104 times>...
long_options = {{name = 0x561f1f56a97b "hba-file", has_arg = 1, flag = 0x0, val = 97}, {name = 0x561f1f56a984 "debug", has_arg = 0, flag = 0x0, val = 100}, {name = 0x561f1f56a98a "config-file", has_arg = 1, flag = 0x0,
val = 102}, {name = 0x561f1f56a996 "key-file", has_arg = 1, flag = 0x0, val = 107}, {name = 0x561f1f56a99f "pcp-file", has_arg = 1, flag = 0x0, val = 70}, {name = 0x561f1f56a9a8 "help", has_arg = 0, flag = 0x0, val = 104}, {
name = 0x561f1f570407 "mode", has_arg = 1, flag = 0x0, val = 109}, {name = 0x561f1f56a9ad "dont-detach", has_arg = 0, flag = 0x0, val = 110}, {name = 0x561f1f56a9b9 "discard-status", has_arg = 0, flag = 0x0, val = 68}, {
name = 0x561f1f56a9c8 "clear-oidmaps", has_arg = 0, flag = 0x0, val = 67}, {name = 0x561f1f56a9d6 "debug-assertions", has_arg = 0, flag = 0x0, val = 120}, {name = 0x561f1f57df6c "version", has_arg = 0, flag = 0x0,
val = 118}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
bt_full_core-pgpool.31857 (6,981 bytes)
root@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2:/data/tmp/pgpool-coredumps# gdb /usr/local/pgpool/bin/pgpool core-pgpool.31857.vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-2.1617120882
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/pgpool/bin/pgpool...done.
[New LWP 31857]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `pgpool: master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres edb'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 GetMemoryChunkContext (pointer=0xffffffff00000009) at ../../src/include/utils/memutils.h:118
118 context = *(MemoryContext *) (((char *) pointer) - sizeof(void *));
(gdb) bt full
#0 GetMemoryChunkContext (pointer=0xffffffff00000009) at ../../src/include/utils/memutils.h:118
context = 0xffffffff00000009
context = <optimized out>
#1 pfree (pointer=0xffffffff00000009) at ../../src/utils/mmgr/mcxt.c:954
context = <optimized out>
#2 0x0000561f1f4d9ed7 in pool_discard_buffer (buffer=0x561f1fd0f6f8) at query_cache/pool_memqcache.c:3270
No locals.
#3 0x0000561f1f4dc3b7 in pool_discard_temp_query_cache (temp_cache=0x561f1fcfda28) at query_cache/pool_memqcache.c:3152
No locals.
#4 0x0000561f1f4d3b45 in CloseComplete (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2981c0) at protocol/pool_proto_modules.c:2352
session_context = <optimized out>
status = POOL_CONTINUE
kind = <optimized out>
name = 0x561f1fd131a8 "C_19"
#5 0x0000561f1f4d97ed in ProcessBackendResponse (frontend=frontend@entry=0x561f1faf3ac8, backend=backend@entry=0x7f14cd2981c0, state=state@entry=0x7ffc56abf1bc, num_fields=num_fields@entry=0x7ffc56abf1ba)
at protocol/pool_proto_modules.c:2948
status = 0
kind = 51 '3'
#6 0x0000561f1f4ca3d2 in pool_process_query (frontend=0x561f1faf3ac8, backend=0x7f14cd2981c0, reset_request=reset_request@entry=0) at protocol/pool_process_query.c:326
num_fields = 0
status = <optimized out>
qcnt = <optimized out>
i = <optimized out>
ProcessQueryContext = 0x7f14cd29d140
state = 0
cont = <optimized out>
#7 0x0000561f1f4c35ac in do_child (fds=fds@entry=0x561f1fc28960) at protocol/child.c:445
status = <optimized out>
sp = 0x7f14cd2957f8
front_end_fd = <optimized out>
saddr = {addr = {ss_family = 2, __ss_padding = "\343\036\177\000\000\001", '\000' <repeats 111 times>, __ss_align = 0}, salen = 16}
con_count = <optimized out>
local_sigjmp_buf = {{__jmpbuf = {94691676817768, 8749050718718474587, 30, 140721762598992, 4, 94691676829152, 3074917343839074651, 8749050176441629019}, __mask_was_saved = 1, __saved_mask = {__val = {0, 140721762596256, 1056,
94691669751903, 0, 3432, 94691669751904, 94691669751902, 0, 140721762596704, 94691669877194, 140721762597568, 140721762598048, 3, 140721762597584, 3}}}}
backend = 0x7f14cd2981c0
now = {tv_sec = 1617120489, tv_usec = 311988}
tz = {tz_minuteswest = 0, tz_dsttime = 0}
timeout = {tv_sec = 0, tv_usec = 0}
connected = 1
connections_count = 0
psbuf = "master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres edb 127.0.0.1(58142) idle\000\000\000\240\373\253V\374\177\000\000\001\000\000\000\000\000\000\000\346\003\000\000\000\000\000\000\371\377\240\313\024\177\000\000\001\200\255\373\000\000\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\321\032\257\037\037V\000\000\326\032\257\037\037V\000\000\267\036\257\037\037V\000\000\321\032\257\037\037V\000\000\267\036\257\037\037V", '\000' <repeats 42 times>...
walk = <optimized out>
#8 0x0000561f1f49b217 in fork_a_child (fds=0x561f1fc28960, id=30) at main/pgpool_main.c:675
pid = 0
elevel_ = <optimized out>
#9 0x0000561f1f4a1b19 in PgpoolMain (discard_status=<optimized out>, clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:406
i = 30
local_sigjmp_buf = {{__jmpbuf = {140721762599456, 2147483649, 140721762598816, 140721762598832, 94691676706168, 0, 5, 140721762649736}, __mask_was_saved = -1, __saved_mask = {__val = {3871410228, 139727276641000, 94691676705264,
140721762599304, 140721762599296, 140721762599392, 16, 94691676790368, 139727292613309, 94691675532912, 140721762599392, 140721762599384, 140721762599376, 94691676790368, 139727278813772, 94691676829744}}}}
first = 1 '\001'
#10 0x0000561f1f499380 in main (argc=<optimized out>, argv=0x7ffc56acc688) at main/main.c:364
opt = <optimized out>
debug_level = 0
optindex = 0
discard_status = 0 '\000'
---Type <return> to continue, or q <return> to quit---
clear_memcache_oidmaps = 0 '\000'
pcp_conf_file_path = "/usr/local/pgpool/etc/pcp.conf", '\000' <repeats 8162 times>
conf_file_path = "/usr/local/pgpool/etc/pgpool.conf", '\000' <repeats 8159 times>
hba_file_path = "/usr/local/pgpool/etc/pool_hba.conf", '\000' <repeats 8157 times>
pool_passwd_key_file_path = "/root/.pgpoolkey", '\000' <repeats 4104 times>...
long_options = {{name = 0x561f1f56a97b "hba-file", has_arg = 1, flag = 0x0, val = 97}, {name = 0x561f1f56a984 "debug", has_arg = 0, flag = 0x0, val = 100}, {name = 0x561f1f56a98a "config-file", has_arg = 1, flag = 0x0,
val = 102}, {name = 0x561f1f56a996 "key-file", has_arg = 1, flag = 0x0, val = 107}, {name = 0x561f1f56a99f "pcp-file", has_arg = 1, flag = 0x0, val = 70}, {name = 0x561f1f56a9a8 "help", has_arg = 0, flag = 0x0, val = 104}, {
name = 0x561f1f570407 "mode", has_arg = 1, flag = 0x0, val = 109}, {name = 0x561f1f56a9ad "dont-detach", has_arg = 0, flag = 0x0, val = 110}, {name = 0x561f1f56a9b9 "discard-status", has_arg = 0, flag = 0x0, val = 68}, {
name = 0x561f1f56a9c8 "clear-oidmaps", has_arg = 0, flag = 0x0, val = 67}, {name = 0x561f1f56a9d6 "debug-assertions", has_arg = 0, flag = 0x0, val = 120}, {name = 0x561f1f57df6c "version", has_arg = 0, flag = 0x0,
val = 118}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} |
|
|
I have looked in to the gdb back trace. I think the direct cause of the segfault is kind of "double free" (different from standard meaning of it since Pgpool-II uses its own memory management system which was imported from PostgreSQL). However I was not able to reproduce the problem (segfault). Is it possible for you to provide a minimal test case for this? Or can you share pgpool log with log_client_message is on while you are seeing the segfault? |
|
|
I also found a bug with query cache in raw mode. The symptom is not segfault but it just does not create a query cache. I am not sure if that is related to this though. https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=7d54c1b53cee3950e8ad1ebcda85b9a0acf405e8 |
|
|
I have looked in to the gdb back trace. I think the direct cause of the segfault is kind of "double free" (different from standard meaning of it since Pgpool-II uses its own memory management system which was imported from PostgreSQL). However I was not able to reproduce the problem (segfault). Is it possible for you to provide a minimal test case for this? Or can you share pgpool log with log_client_message is on while you are seeing the segfault? As requested above, please find attached pgpool logs with log_client_message enabled. Also, attaching a back trace from one of the crashed process. bt_full_core-pgpool.18208_ log_client_message (6,734 bytes)
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/pgpool/bin/pgpool...done.
[New LWP 18208]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `pgpool: master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres fdb'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 GetMemoryChunkContext (pointer=0xffffffff00000113) at ../../src/include/utils/memutils.h:118
118 context = *(MemoryContext *) (((char *) pointer) - sizeof(void *));
(gdb) bt full
#0 GetMemoryChunkContext (pointer=0xffffffff00000113) at ../../src/include/utils/memutils.h:118
context = 0xffffffff00000113
context = <optimized out>
#1 pfree (pointer=0xffffffff00000113) at ../../src/utils/mmgr/mcxt.c:954
context = <optimized out>
#2 0x000055bdadf4ded7 in pool_discard_buffer (buffer=0x55bdb03a6da8) at query_cache/pool_memqcache.c:3270
No locals.
#3 0x000055bdadf503b7 in pool_discard_temp_query_cache (temp_cache=0x55bdb0396008) at query_cache/pool_memqcache.c:3152
No locals.
#4 0x000055bdadf47b45 in CloseComplete (frontend=frontend@entry=0x55bdb018aab8, backend=backend@entry=0x7f579effd1c0) at protocol/pool_proto_modules.c:2352
session_context = <optimized out>
status = POOL_CONTINUE
kind = <optimized out>
name = 0x55bdb0397240 "C_17"
#5 0x000055bdadf4d7ed in ProcessBackendResponse (frontend=frontend@entry=0x55bdb018aab8, backend=backend@entry=0x7f579effd1c0, state=state@entry=0x7ffd21bc06ec, num_fields=num_fields@entry=0x7ffd21bc06ea)
at protocol/pool_proto_modules.c:2948
status = 0
kind = 51 '3'
#6 0x000055bdadf3e3d2 in pool_process_query (frontend=0x55bdb018aab8, backend=0x7f579effd1c0, reset_request=reset_request@entry=0) at protocol/pool_process_query.c:326
num_fields = 0
status = <optimized out>
qcnt = <optimized out>
i = <optimized out>
ProcessQueryContext = 0x7f579f002140
state = 0
cont = <optimized out>
#7 0x000055bdadf375ac in do_child (fds=fds@entry=0x55bdb02c3e90) at protocol/child.c:445
status = <optimized out>
sp = 0x7f579effa7f8
front_end_fd = <optimized out>
saddr = {addr = {ss_family = 2, __ss_padding = "\215\356\177\000\000\001", '\000' <repeats 111 times>, __ss_align = 0}, salen = 16}
con_count = <optimized out>
local_sigjmp_buf = {{__jmpbuf = {94273192869528, -7831768791790272525, 44, 140725169427840, 4, 94273192769744, -4049043003722317837, -7831777028476911629}, __mask_was_saved = 1, __saved_mask = {
__val = {0, 140725169425104, 1056, 94273156282463, 0, 3432, 94273156282464, 94273156282462, 0, 140725169425552, 94273156407754, 140725169426416, 140725169426896, 3, 140725169426432, 3}}}}
backend = 0x7f579effd1c0
now = {tv_sec = 1617635563, tv_usec = 954044}
tz = {tz_minuteswest = 0, tz_dsttime = 0}
timeout = {tv_sec = 284, tv_usec = 541299}
connected = 1
connections_count = 0
psbuf = "master@vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-postgres fdb 127.0.0.1(36334) idle\000\000\000\320\020\274!\375\177\000\000\001\000\000\000\000\000\000\000\346\003\000\000\000\000\000\000\371Ow\235W\177\000\000\001\200\255\373\000\000\000\000\301\212\030\260\275U\000\000\301\212\030\260\275U\000\000\301\212\030\260\275U\000\000\301\212\030\260\275U\000\000Ɗ\030\260\275U\000\000\247\216\030\260\275U\000\000\301\212\030\260\275U\000\000\247\216\030\260\275U", '\000' <repeats 42 times>...
walk = <optimized out>
#8 0x000055bdadf0f217 in fork_a_child (fds=0x55bdb02c3e90, id=44) at main/pgpool_main.c:675
pid = 0
elevel_ = <optimized out>
#9 0x000055bdadf15b19 in PgpoolMain (discard_status=<optimized out>, clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:406
i = 44
---Type <return> to continue, or q <return> to quit---
local_sigjmp_buf = {{__jmpbuf = {140725169427856, 3, 140725169427840, 140724603453440, 140014306537864, 0, 14, 140014306546912}, __mask_was_saved = -1, __saved_mask = {__val = {1508362478,
140014306592840, 140725169428008, 140014306538960, 140014306591984, 0, 140014304379807, 8, 140014306538960, 1, 0, 1, 140014306591984, 140014299737590, 7926244294, 140725169427840}}}}
first = 1 '\001'
#10 0x000055bdadf0d380 in main (argc=<optimized out>, argv=0x7ffd21bcdbb8) at main/main.c:364
opt = <optimized out>
debug_level = 0
optindex = 0
discard_status = 0 '\000'
clear_memcache_oidmaps = 0 '\000'
pcp_conf_file_path = "/usr/local/pgpool/etc/pcp.conf", '\000' <repeats 8162 times>
conf_file_path = "/usr/local/pgpool/etc/pgpool.conf", '\000' <repeats 8159 times>
hba_file_path = "/usr/local/pgpool/etc/pool_hba.conf", '\000' <repeats 8157 times>
pool_passwd_key_file_path = "/root/.pgpoolkey", '\000' <repeats 4104 times>...
long_options = {{name = 0x55bdadfde97b "hba-file", has_arg = 1, flag = 0x0, val = 97}, {name = 0x55bdadfde984 "debug", has_arg = 0, flag = 0x0, val = 100}, {name = 0x55bdadfde98a "config-file",
has_arg = 1, flag = 0x0, val = 102}, {name = 0x55bdadfde996 "key-file", has_arg = 1, flag = 0x0, val = 107}, {name = 0x55bdadfde99f "pcp-file", has_arg = 1, flag = 0x0, val = 70}, {
name = 0x55bdadfde9a8 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x55bdadfe4407 "mode", has_arg = 1, flag = 0x0, val = 109}, {name = 0x55bdadfde9ad "dont-detach", has_arg = 0,
flag = 0x0, val = 110}, {name = 0x55bdadfde9b9 "discard-status", has_arg = 0, flag = 0x0, val = 68}, {name = 0x55bdadfde9c8 "clear-oidmaps", has_arg = 0, flag = 0x0, val = 67}, {
name = 0x55bdadfde9d6 "debug-assertions", has_arg = 0, flag = 0x0, val = 120}, {name = 0x55bdadff1f6c "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x0, has_arg = 0, flag = 0x0,
val = 0}}
(gdb)
(gdb)
(gdb) quit |
|
|
About your response "However I was not able to reproduce the problem (segfault). Is it possible for you to provide a minimal test case for this? Or can you share pgpool log with log_client_message is on while you are seeing the segfault?" I am bit confused about your inputs, this in-memory caching is not working with 'shmem' option either (Attaching pgpool logs with shmem option ). Also, i tried this with couple of older release (4.1.6 and 4.0.13) builds. It isn't working there either. |
|
|
I assume "not working" means segfault. I think the segfault could happen either shmem and memcached option because it s likely the segfault is related to the module which is common with both shmen and memcached. I will look into the pgpool log to find out if there's particular pattern to cause the segfault. |
|
|
I found a minimal test case. A statement S1 is parsed and bound to portal P1. Then P1 is closed but S1 still remains. Next S1 is bound to P2. The segfault happens when P2 is closed because when P1 is closed, a buffer is freed. But second close message also tries to free the same memory, which causes crash. Attached is a patch to fix the bug. query_cache.diff (769 bytes)
diff --git a/src/protocol/pool_proto_modules.c b/src/protocol/pool_proto_modules.c
index f412c498..ec64abef 100644
--- a/src/protocol/pool_proto_modules.c
+++ b/src/protocol/pool_proto_modules.c
@@ -2349,7 +2349,19 @@ CloseComplete(POOL_CONNECTION * frontend, POOL_CONNECTION_POOL * backend)
kind, name)));
if (pool_config->memory_cache_enabled)
{
- pool_discard_temp_query_cache(pool_get_current_cache());
+ POOL_QUERY_CONTEXT *query_context;
+ POOL_TEMP_QUERY_CACHE *temp_cache;
+
+ query_context = session_context->query_context;
+ if (query_context)
+ {
+ temp_cache = query_context->temp_cache;
+ if (temp_cache)
+ {
+ pool_discard_temp_query_cache(temp_cache);
+ query_context->temp_cache = NULL;
+ }
+ }
}
}
|
|
|
With the patch in-place not seeing segmentaion fault errors, but in-memory caching with 'memcached' is not capturing any *_cache_entries data. All data columns is showing up as 0. memcached: fdb=> fdb=> show pool_cache; num_cache_hits | num_selects | cache_hit_ratio | num_hash_entries | used_hash_entries | num_cache_entries | used_cache_entries_size | free_cache_entries_size | fragment_cache_entries_size ----------------+-------------+-----------------+------------------+-------------------+-------------------+-------------------------+-------------------------+----------------------------- 86796 | 77685 | 0.53 | 0 | 0 | 0 | 0 | 0 | 0 (1 row) With 'shmem' type, data is being logged as expected. Wondering if this is side effect from the patch changes? shmem: fdb=> show pool_cache; num_cache_hits | num_selects | cache_hit_ratio | num_hash_entries | used_hash_entries | num_cache_entries | used_cache_entries_size | free_cache_entries_size | fragment_cache_entries_size ----------------+-------------+-----------------+------------------+-------------------+-------------------+-------------------------+-------------------------+----------------------------- 146 | 201 | 0.42 | 1048576 | 449 | 449 | 60092 | 536810820 | 0 (1 row) |
|
|
If the cache storage is memcached, show pool_cache shows all 0 except num_cache_hits, num_selects and cache_hit_ratio. Sorry that the document is not clear about this. I will fix the document. |
|
|
Done. https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=b0fa4bc2bc9d253897f787dfa788fb7cbb431e30 |
|
|
If you don't have further issues, I would like to close this issue. May I? |
|
|
Yes, I was able to validate the patch change. Please feel free to close this bug. Thanks for your help. BTW, i see this is targeted for 4.2.3. Do we have an ETA date for this patch release? |
|
|
Yes, May 20th, 2021. https://pgpool.net/mediawiki/index.php/Roadmap |
|
|
4.2.3 is released. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2021-03-29 22:38 | G79189 | New Issue | |
| 2021-03-29 22:38 | G79189 | File Added: core-pgpool.13778.vmw-hcs-59eca46d-26c1-4cc2-80c7-6ede3bd34ae5-2743-0-node-1.1617020942 | |
| 2021-03-30 11:41 | t-ishii | Note Added: 0003787 | |
| 2021-03-30 11:41 | t-ishii | Assigned To | => t-ishii |
| 2021-03-30 11:41 | t-ishii | Status | new => assigned |
| 2021-03-30 11:41 | t-ishii | Status | assigned => feedback |
| 2021-03-30 11:41 | t-ishii | Description Updated | |
| 2021-03-30 11:41 | t-ishii | Steps to Reproduce Updated | |
| 2021-03-30 11:41 | t-ishii | Additional Information Updated | |
| 2021-03-30 11:47 | t-ishii | Note Added: 0003788 | |
| 2021-03-31 01:32 | G79189 | Note Added: 0003790 | |
| 2021-03-31 01:32 | G79189 | File Added: bt_full_core-pgpool.711 | |
| 2021-03-31 01:32 | G79189 | File Added: bt_full_core-pgpool.31830 | |
| 2021-03-31 01:32 | G79189 | File Added: bt_full_core-pgpool.31857 | |
| 2021-03-31 01:32 | G79189 | File Added: bt_full_core-pgpool.32086 | |
| 2021-03-31 01:32 | G79189 | Status | feedback => assigned |
| 2021-04-05 11:50 | t-ishii | Note Added: 0003791 | |
| 2021-04-05 11:54 | t-ishii | Note Added: 0003792 | |
| 2021-04-05 11:54 | t-ishii | Status | assigned => feedback |
| 2021-04-06 00:37 | G79189 | Note Added: 0003794 | |
| 2021-04-06 00:37 | G79189 | File Added: pgpool.log_ log_client_message.log | |
| 2021-04-06 00:37 | G79189 | File Added: bt_full_core-pgpool.18208_ log_client_message | |
| 2021-04-06 00:37 | G79189 | Status | feedback => assigned |
| 2021-04-06 00:43 | G79189 | Note Added: 0003795 | |
| 2021-04-06 00:43 | G79189 | File Added: pgpool.log_shmem | |
| 2021-04-06 06:55 | t-ishii | Note Added: 0003796 | |
| 2021-04-09 19:55 | t-ishii | Note Added: 0003800 | |
| 2021-04-09 19:55 | t-ishii | File Added: query_cache.diff | |
| 2021-04-09 19:55 | t-ishii | Status | assigned => feedback |
| 2021-04-09 19:56 | t-ishii | Target Version | => 4.2.3 |
| 2021-04-13 08:16 | G79189 | Note Added: 0003801 | |
| 2021-04-13 08:16 | G79189 | Status | feedback => assigned |
| 2021-04-13 13:14 | t-ishii | Note Added: 0003802 | |
| 2021-04-13 13:14 | t-ishii | Status | assigned => feedback |
| 2021-04-19 09:19 | t-ishii | Note Added: 0003816 | |
| 2021-04-19 13:00 | t-ishii | Note Added: 0003817 | |
| 2021-04-20 06:27 | G79189 | Note Added: 0003819 | |
| 2021-04-20 06:27 | G79189 | Status | feedback => assigned |
| 2021-04-20 06:47 | t-ishii | Note Added: 0003820 | |
| 2021-04-20 08:36 | t-ishii | Status | assigned => feedback |
| 2021-05-20 12:13 | administrator | Note Added: 0003834 | |
| 2021-05-20 12:13 | administrator | Status | feedback => closed |
| 2021-05-20 12:13 | administrator | Fixed in Version | => 4.2.3 |
