View Issue Details

IDProjectCategoryView StatusLast Update
0000212Pgpool-IIBugpublic2016-09-28 10:07
ReporterlassetuominenAssigned ToMuhammad Usama 
PrioritynormalSeveritycrashReproducibilityalways
Status closedResolutionopen 
PlatformLinuxOSCentOSOS Version7 64-bit
Product Version 
Target VersionFixed in Version 
Summary0000212: pgpool runs out of memory
DescriptionOur setup has four db servers with one pgpool in replication mode with no loadbalancing or caching. pgpool server has 8 GB memory and db servers have 4 GB memory.

After a couple of hours running our frontend application, pgpool has eaten all of the memory and crashes. Occasionally one of the backend databases might crash as well and there is a mismatch between databases.

I ran pgpool using valgrind:

valgrind --tool=memcheck --track-origins=yes --undef-value-errors=yes --trace-children=yes --leak-check=full pgpool -n -f /etc/pgpool-II-95/pgpool.conf > pgpool_valgrind.log 2>&1 &

Please let me know if you need more information.
Steps To ReproduceCould be possible to reproduce with pgbench.
Additional InformationPostgreSQL 9.5.3
pgpool-II 3.5.2

Somewhat relevant parameters of pgpool.conf:
listen_backlog_multiplier = 2
serialize_accept = off
authentication_timeout = 60
num_init_children = 4
max_pool = 4
child_life_time = 0
child_max_connections = 0
connection_life_time = 0
client_idle_limit = 0
connection_cache = off
reset_query_list = 'ABORT; DISCARD ALL'
replication_mode = on
replicate_select = on
insert_lock = on
lobj_lock_table = ''
replication_stop_on_mismatch = off
failover_if_affected_tuples_mismatch = off
relcache_expire = 300
relcache_size = 256
check_temp_table = off
check_unlogged_table = off
memory_cache_enabled = off
TagsNo tags attached.

Activities

lassetuominen

2016-06-29 21:49

reporter  

valgrind_pgpool.txt (69,465 bytes)
==9598== Memcheck, a memory error detector
==9598== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==9598== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==9598== Command: pgpool -n -f /etc/pgpool-II-95/pgpool.conf
==9598== 
2016-06-29 14:19:36: pid 9598: LOG:  Setting up socket for 0.0.0.0:9999
2016-06-29 14:19:36: pid 9598: LOCATION:  pgpool_main.c:797
2016-06-29 14:19:36: pid 9598: LOG:  Setting up socket for :::9999
2016-06-29 14:19:36: pid 9598: LOCATION:  pgpool_main.c:797
2016-06-29 14:19:36: pid 9598: LOG:  pgpool-II successfully started. version 3.5.2 (ekieboshi)
2016-06-29 14:19:36: pid 9598: LOCATION:  pgpool_main.c:370
2016-06-29 14:21:08: pid 9603: LOG:  forked new pcp worker, pid=9671 socket=7
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:275
2016-06-29 14:21:08: pid 9603: LOG:  forked new pcp worker, pid=9674 socket=7
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:275
==9671== Invalid free() / delete / delete[] / realloc()
==9671==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9671==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9671==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9671==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9671==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9671==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9671==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9671==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9671==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9671==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9671==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9671==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9671== 
==9671== 
==9671== HEAP SUMMARY:
==9671==     in use at exit: 225,874 bytes in 3,060 blocks
==9671==   total heap usage: 3,144 allocs, 85 frees, 268,531 bytes allocated
==9671== 
==9671== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9671==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9671==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9671==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9671==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9671== 
==9671== LEAK SUMMARY:
==9671==    definitely lost: 192 bytes in 1 blocks
==9671==    indirectly lost: 2,037 bytes in 23 blocks
==9671==      possibly lost: 0 bytes in 0 blocks
==9671==    still reachable: 223,645 bytes in 3,036 blocks
==9671==         suppressed: 0 bytes in 0 blocks
==9671== Reachable blocks (those to which a pointer was found) are not shown.
==9671== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9671== 
==9671== For counts of detected and suppressed errors, rerun with: -v
==9671== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9671 exit with SUCCESS.
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9671 exits with status 0
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:21:08: pid 9603: LOG:  forked new pcp worker, pid=9676 socket=7
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:275
==9674== Invalid free() / delete / delete[] / realloc()
==9674==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9674==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9674==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9674==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9674==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9674==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9674==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9674==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9674==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9674==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9674==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9674==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9674== 
==9674== 
==9674== HEAP SUMMARY:
==9674==     in use at exit: 225,874 bytes in 3,060 blocks
==9674==   total heap usage: 3,146 allocs, 87 frees, 268,561 bytes allocated
==9674== 
==9674== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9674==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9674==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9674==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9674==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9674== 
==9674== LEAK SUMMARY:
==9674==    definitely lost: 192 bytes in 1 blocks
==9674==    indirectly lost: 2,037 bytes in 23 blocks
==9674==      possibly lost: 0 bytes in 0 blocks
==9674==    still reachable: 223,645 bytes in 3,036 blocks
==9674==         suppressed: 0 bytes in 0 blocks
==9674== Reachable blocks (those to which a pointer was found) are not shown.
==9674== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9674== 
==9674== For counts of detected and suppressed errors, rerun with: -v
==9674== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9674 exit with SUCCESS.
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9674 exits with status 0
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:21:08: pid 9603: LOG:  forked new pcp worker, pid=9678 socket=7
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:275
==9676== Invalid free() / delete / delete[] / realloc()
==9676==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9676==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9676==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9676==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9676==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9676==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9676==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9676==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9676==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9676==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9676==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9676==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9676== 
==9676== 
==9676== HEAP SUMMARY:
==9676==     in use at exit: 225,874 bytes in 3,060 blocks
==9676==   total heap usage: 3,152 allocs, 93 frees, 268,651 bytes allocated
==9676== 
==9676== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9676==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9676==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9676==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9676==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9676== 
==9676== LEAK SUMMARY:
==9676==    definitely lost: 192 bytes in 1 blocks
==9676==    indirectly lost: 2,037 bytes in 23 blocks
==9676==      possibly lost: 0 bytes in 0 blocks
==9676==    still reachable: 223,645 bytes in 3,036 blocks
==9676==         suppressed: 0 bytes in 0 blocks
==9676== Reachable blocks (those to which a pointer was found) are not shown.
==9676== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9676== 
==9676== For counts of detected and suppressed errors, rerun with: -v
==9676== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9676 exit with SUCCESS.
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9676 exits with status 0
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:21:08: pid 9603: LOG:  forked new pcp worker, pid=9680 socket=7
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:275
==9678== Invalid free() / delete / delete[] / realloc()
==9678==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9678==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9678==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9678==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9678==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9678==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9678==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9678==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9678==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9678==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9678==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9678==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9678== 
==9678== 
==9678== HEAP SUMMARY:
==9678==     in use at exit: 225,874 bytes in 3,060 blocks
==9678==   total heap usage: 3,158 allocs, 99 frees, 268,741 bytes allocated
==9678== 
==9678== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9678==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9678==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9678==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9678==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9678== 
==9678== LEAK SUMMARY:
==9678==    definitely lost: 192 bytes in 1 blocks
==9678==    indirectly lost: 2,037 bytes in 23 blocks
==9678==      possibly lost: 0 bytes in 0 blocks
==9678==    still reachable: 223,645 bytes in 3,036 blocks
==9678==         suppressed: 0 bytes in 0 blocks
==9678== Reachable blocks (those to which a pointer was found) are not shown.
==9678== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9678== 
==9678== For counts of detected and suppressed errors, rerun with: -v
==9678== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9678 exit with SUCCESS.
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9678 exits with status 0
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:341
==9680== Invalid free() / delete / delete[] / realloc()
==9680==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9680==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9680==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9680==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9680==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9680==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9680==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9680==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9680==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9680==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9680==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9680==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9680== 
==9680== 
==9680== HEAP SUMMARY:
==9680==     in use at exit: 225,874 bytes in 3,060 blocks
==9680==   total heap usage: 3,164 allocs, 105 frees, 268,831 bytes allocated
==9680== 
==9680== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9680==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9680==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9680==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9680==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9680== 
==9680== LEAK SUMMARY:
==9680==    definitely lost: 192 bytes in 1 blocks
==9680==    indirectly lost: 2,037 bytes in 23 blocks
==9680==      possibly lost: 0 bytes in 0 blocks
==9680==    still reachable: 223,645 bytes in 3,036 blocks
==9680==         suppressed: 0 bytes in 0 blocks
==9680== Reachable blocks (those to which a pointer was found) are not shown.
==9680== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9680== 
==9680== For counts of detected and suppressed errors, rerun with: -v
==9680== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9680 exit with SUCCESS.
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:08: pid 9603: LOG:  PCP process with pid: 9680 exits with status 0
2016-06-29 14:21:08: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:21:22: pid 9603: LOG:  forked new pcp worker, pid=9695 socket=7
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:275
2016-06-29 14:21:22: pid 9603: LOG:  forked new pcp worker, pid=9698 socket=7
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:275
==9695== Invalid free() / delete / delete[] / realloc()
==9695==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9695==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9695==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9695==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9695==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9695==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9695==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9695==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9695==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9695==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9695==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9695==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9695== 
==9695== 
==9695== HEAP SUMMARY:
==9695==     in use at exit: 225,874 bytes in 3,060 blocks
==9695==   total heap usage: 3,174 allocs, 115 frees, 268,981 bytes allocated
==9695== 
==9695== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9695==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9695==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9695==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9695==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9695== 
==9695== LEAK SUMMARY:
==9695==    definitely lost: 192 bytes in 1 blocks
==9695==    indirectly lost: 2,037 bytes in 23 blocks
==9695==      possibly lost: 0 bytes in 0 blocks
==9695==    still reachable: 223,645 bytes in 3,036 blocks
==9695==         suppressed: 0 bytes in 0 blocks
==9695== Reachable blocks (those to which a pointer was found) are not shown.
==9695== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9695== 
==9695== For counts of detected and suppressed errors, rerun with: -v
==9695== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9695 exit with SUCCESS.
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9695 exits with status 0
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:21:22: pid 9603: LOG:  forked new pcp worker, pid=9700 socket=7
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:275
==9698== Invalid free() / delete / delete[] / realloc()
==9698==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9698==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9698==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9698==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9698==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9698==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9698==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9698==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9698==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9698==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9698==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9698==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9698== 
==9698== 
==9698== HEAP SUMMARY:
==9698==     in use at exit: 225,874 bytes in 3,060 blocks
==9698==   total heap usage: 3,176 allocs, 117 frees, 269,011 bytes allocated
==9698== 
==9698== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9698==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9698==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9698==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9698==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9698== 
==9698== LEAK SUMMARY:
==9698==    definitely lost: 192 bytes in 1 blocks
==9698==    indirectly lost: 2,037 bytes in 23 blocks
==9698==      possibly lost: 0 bytes in 0 blocks
==9698==    still reachable: 223,645 bytes in 3,036 blocks
==9698==         suppressed: 0 bytes in 0 blocks
==9698== Reachable blocks (those to which a pointer was found) are not shown.
==9698== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9698== 
==9698== For counts of detected and suppressed errors, rerun with: -v
==9698== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9698 exit with SUCCESS.
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9698 exits with status 0
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:21:22: pid 9603: LOG:  forked new pcp worker, pid=9702 socket=7
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:275
==9700== Invalid free() / delete / delete[] / realloc()
==9700==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9700==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9700==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9700==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9700==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9700==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9700==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9700==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9700==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9700==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9700==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9700==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9700== 
==9700== 
==9700== HEAP SUMMARY:
==9700==     in use at exit: 225,874 bytes in 3,060 blocks
==9700==   total heap usage: 3,182 allocs, 123 frees, 269,101 bytes allocated
==9700== 
==9700== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9700==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9700==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9700==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9700==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9700== 
==9700== LEAK SUMMARY:
==9700==    definitely lost: 192 bytes in 1 blocks
==9700==    indirectly lost: 2,037 bytes in 23 blocks
==9700==      possibly lost: 0 bytes in 0 blocks
==9700==    still reachable: 223,645 bytes in 3,036 blocks
==9700==         suppressed: 0 bytes in 0 blocks
==9700== Reachable blocks (those to which a pointer was found) are not shown.
==9700== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9700== 
==9700== For counts of detected and suppressed errors, rerun with: -v
==9700== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9700 exit with SUCCESS.
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9700 exits with status 0
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:21:22: pid 9603: LOG:  forked new pcp worker, pid=9704 socket=7
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:275
==9702== Invalid free() / delete / delete[] / realloc()
==9702==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9702==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9702==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9702==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9702==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9702==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9702==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9702==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9702==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9702==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9702==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9702==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9702== 
==9702== 
==9702== HEAP SUMMARY:
==9702==     in use at exit: 225,874 bytes in 3,060 blocks
==9702==   total heap usage: 3,188 allocs, 129 frees, 269,191 bytes allocated
==9702== 
==9702== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9702==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9702==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9702==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9702==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9702== 
==9702== LEAK SUMMARY:
==9702==    definitely lost: 192 bytes in 1 blocks
==9702==    indirectly lost: 2,037 bytes in 23 blocks
==9702==      possibly lost: 0 bytes in 0 blocks
==9702==    still reachable: 223,645 bytes in 3,036 blocks
==9702==         suppressed: 0 bytes in 0 blocks
==9702== Reachable blocks (those to which a pointer was found) are not shown.
==9702== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9702== 
==9702== For counts of detected and suppressed errors, rerun with: -v
==9702== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9702 exit with SUCCESS.
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9702 exits with status 0
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:341
==9704== Invalid free() / delete / delete[] / realloc()
==9704==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9704==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9704==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9704==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9704==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9704==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9704==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9704==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9704==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9704==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9704==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9704==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9704== 
==9704== 
==9704== HEAP SUMMARY:
==9704==     in use at exit: 225,874 bytes in 3,060 blocks
==9704==   total heap usage: 3,194 allocs, 135 frees, 269,281 bytes allocated
==9704== 
==9704== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==9704==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9704==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9704==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9704==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9704== 
==9704== LEAK SUMMARY:
==9704==    definitely lost: 192 bytes in 1 blocks
==9704==    indirectly lost: 2,037 bytes in 23 blocks
==9704==      possibly lost: 0 bytes in 0 blocks
==9704==    still reachable: 223,645 bytes in 3,036 blocks
==9704==         suppressed: 0 bytes in 0 blocks
==9704== Reachable blocks (those to which a pointer was found) are not shown.
==9704== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9704== 
==9704== For counts of detected and suppressed errors, rerun with: -v
==9704== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9704 exit with SUCCESS.
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:21:22: pid 9603: LOG:  PCP process with pid: 9704 exits with status 0
2016-06-29 14:21:22: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:27:48: pid 9603: LOG:  forked new pcp worker, pid=10023 socket=7
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:275
2016-06-29 14:27:48: pid 9603: LOG:  forked new pcp worker, pid=10027 socket=7
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:275
==10023== Invalid free() / delete / delete[] / realloc()
==10023==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10023==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==10023==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==10023==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==10023==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==10023==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10023==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10023==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10023==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10023==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10023==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10023==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==10023== 
==10023== 
==10023== HEAP SUMMARY:
==10023==     in use at exit: 225,874 bytes in 3,060 blocks
==10023==   total heap usage: 3,204 allocs, 145 frees, 269,431 bytes allocated
==10023== 
==10023== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==10023==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10023==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10023==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10023==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10023== 
==10023== LEAK SUMMARY:
==10023==    definitely lost: 192 bytes in 1 blocks
==10023==    indirectly lost: 2,037 bytes in 23 blocks
==10023==      possibly lost: 0 bytes in 0 blocks
==10023==    still reachable: 223,645 bytes in 3,036 blocks
==10023==         suppressed: 0 bytes in 0 blocks
==10023== Reachable blocks (those to which a pointer was found) are not shown.
==10023== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==10023== 
==10023== For counts of detected and suppressed errors, rerun with: -v
==10023== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:27:48: pid 9603: LOG:  PCP process with pid: 10023 exit with SUCCESS.
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:27:48: pid 9603: LOG:  PCP process with pid: 10023 exits with status 0
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:27:48: pid 9603: LOG:  forked new pcp worker, pid=10032 socket=7
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:275
==10027== Invalid free() / delete / delete[] / realloc()
==10027==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10027==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==10027==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==10027==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==10027==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==10027==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10027==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10027==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10027==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10027==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10027==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10027==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==10027== 
==10027== 
==10027== HEAP SUMMARY:
==10027==     in use at exit: 225,874 bytes in 3,060 blocks
==10027==   total heap usage: 3,206 allocs, 147 frees, 269,461 bytes allocated
==10027== 
==10027== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==10027==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10027==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10027==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10027==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10027== 
==10027== LEAK SUMMARY:
==10027==    definitely lost: 192 bytes in 1 blocks
==10027==    indirectly lost: 2,037 bytes in 23 blocks
==10027==      possibly lost: 0 bytes in 0 blocks
==10027==    still reachable: 223,645 bytes in 3,036 blocks
==10027==         suppressed: 0 bytes in 0 blocks
==10027== Reachable blocks (those to which a pointer was found) are not shown.
==10027== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==10027== 
==10027== For counts of detected and suppressed errors, rerun with: -v
==10027== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:27:48: pid 9603: LOG:  PCP process with pid: 10027 exit with SUCCESS.
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:27:48: pid 9603: LOG:  PCP process with pid: 10027 exits with status 0
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:27:48: pid 9603: LOG:  forked new pcp worker, pid=10034 socket=7
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:275
==10032== Invalid free() / delete / delete[] / realloc()
==10032==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10032==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==10032==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==10032==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==10032==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==10032==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10032==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10032==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10032==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10032==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10032==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10032==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==10032== 
==10032== 
==10032== HEAP SUMMARY:
==10032==     in use at exit: 225,874 bytes in 3,060 blocks
==10032==   total heap usage: 3,212 allocs, 153 frees, 269,551 bytes allocated
==10032== 
==10032== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==10032==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10032==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10032==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10032==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10032== 
==10032== LEAK SUMMARY:
==10032==    definitely lost: 192 bytes in 1 blocks
==10032==    indirectly lost: 2,037 bytes in 23 blocks
==10032==      possibly lost: 0 bytes in 0 blocks
==10032==    still reachable: 223,645 bytes in 3,036 blocks
==10032==         suppressed: 0 bytes in 0 blocks
==10032== Reachable blocks (those to which a pointer was found) are not shown.
==10032== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==10032== 
==10032== For counts of detected and suppressed errors, rerun with: -v
==10032== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:27:48: pid 9603: LOG:  PCP process with pid: 10032 exit with SUCCESS.
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:27:48: pid 9603: LOG:  PCP process with pid: 10032 exits with status 0
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 14:27:48: pid 9603: LOG:  forked new pcp worker, pid=10036 socket=7
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:275
==10034== Invalid free() / delete / delete[] / realloc()
==10034==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10034==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==10034==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==10034==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==10034==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==10034==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10034==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10034==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10034==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10034==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10034==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10034==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==10034== 
==10034== 
==10034== HEAP SUMMARY:
==10034==     in use at exit: 225,874 bytes in 3,060 blocks
==10034==   total heap usage: 3,218 allocs, 159 frees, 269,641 bytes allocated
==10034== 
==10034== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==10034==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10034==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10034==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10034==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10034== 
==10034== LEAK SUMMARY:
==10034==    definitely lost: 192 bytes in 1 blocks
==10034==    indirectly lost: 2,037 bytes in 23 blocks
==10034==      possibly lost: 0 bytes in 0 blocks
==10034==    still reachable: 223,645 bytes in 3,036 blocks
==10034==         suppressed: 0 bytes in 0 blocks
==10034== Reachable blocks (those to which a pointer was found) are not shown.
==10034== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==10034== 
==10034== For counts of detected and suppressed errors, rerun with: -v
==10034== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:27:48: pid 9603: LOG:  PCP process with pid: 10034 exit with SUCCESS.
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:27:48: pid 9603: LOG:  PCP process with pid: 10034 exits with status 0
2016-06-29 14:27:48: pid 9603: LOCATION:  pcp_child.c:341
==10036== Invalid free() / delete / delete[] / realloc()
==10036==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10036==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==10036==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==10036==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==10036==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==10036==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10036==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10036==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10036==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10036==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10036==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10036==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==10036== 
==10036== 
==10036== HEAP SUMMARY:
==10036==     in use at exit: 225,874 bytes in 3,060 blocks
==10036==   total heap usage: 3,224 allocs, 165 frees, 269,731 bytes allocated
==10036== 
==10036== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==10036==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==10036==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10036==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==10036==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==10036== 
==10036== LEAK SUMMARY:
==10036==    definitely lost: 192 bytes in 1 blocks
==10036==    indirectly lost: 2,037 bytes in 23 blocks
==10036==      possibly lost: 0 bytes in 0 blocks
==10036==    still reachable: 223,645 bytes in 3,036 blocks
==10036==         suppressed: 0 bytes in 0 blocks
==10036== Reachable blocks (those to which a pointer was found) are not shown.
==10036== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==10036== 
==10036== For counts of detected and suppressed errors, rerun with: -v
==10036== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 14:27:49: pid 9603: LOG:  PCP process with pid: 10036 exit with SUCCESS.
2016-06-29 14:27:49: pid 9603: LOCATION:  pcp_child.c:327
2016-06-29 14:27:49: pid 9603: LOG:  PCP process with pid: 10036 exits with status 0
2016-06-29 14:27:49: pid 9603: LOCATION:  pcp_child.c:341
2016-06-29 15:30:14: pid 9598: LOG:  failed to connect to PostgreSQL server on "10.50.1.217:5432", timed out
2016-06-29 15:30:14: pid 9598: LOCATION:  pool_connection_pool.c:630
2016-06-29 15:30:14: pid 9598: ERROR:  failed to make persistent db connection
2016-06-29 15:30:14: pid 9598: DETAIL:  connection to host:"10.50.1.217:5432" failed
2016-06-29 15:30:14: pid 9598: LOCATION:  child.c:1180
2016-06-29 15:30:29: pid 9598: LOG:  failed to connect to PostgreSQL server on "10.50.1.217:5432", timed out
2016-06-29 15:30:29: pid 9598: LOCATION:  pool_connection_pool.c:630
2016-06-29 15:30:29: pid 9598: ERROR:  failed to make persistent db connection
2016-06-29 15:30:29: pid 9598: DETAIL:  connection to host:"10.50.1.217:5432" failed
2016-06-29 15:30:29: pid 9598: LOCATION:  child.c:1180
2016-06-29 15:30:29: pid 9598: LOG:  setting backend node 3 status to NODE DOWN
2016-06-29 15:30:29: pid 9598: LOCATION:  pgpool_main.c:543
2016-06-29 15:30:30: pid 9598: LOG:  received degenerate backend request for node_id: 3 from pid [9598]
2016-06-29 15:30:30: pid 9598: LOCATION:  pgpool_main.c:1110
2016-06-29 15:30:30: pid 9598: LOG:  starting degeneration. shutdown host 10.50.1.217(5432)
2016-06-29 15:30:30: pid 9598: LOCATION:  pgpool_main.c:1606
2016-06-29 15:30:30: pid 9598: LOG:  Restart all children
2016-06-29 15:30:30: pid 9598: LOCATION:  pgpool_main.c:1705
2016-06-29 15:30:30: pid 9600: LOG:  child process received shutdown request signal 3
2016-06-29 15:30:30: pid 9600: LOCATION:  child.c:911
2016-06-29 15:30:30: pid 9598: LOG:  failover: set new primary node: -1
2016-06-29 15:30:30: pid 9598: LOCATION:  pgpool_main.c:1848
2016-06-29 15:30:30: pid 9598: LOG:  failover: set new master node: 0
2016-06-29 15:30:30: pid 9598: LOCATION:  pgpool_main.c:1854
2016-06-29 15:30:30: pid 9601: LOG:  child process received shutdown request signal 3
2016-06-29 15:30:30: pid 9601: LOCATION:  child.c:911
2016-06-29 15:30:30: pid 9602: LOG:  child process received shutdown request signal 3
2016-06-29 15:30:30: pid 9602: LOCATION:  child.c:911
2016-06-29 15:30:30: pid 9599: LOG:  child process received shutdown request signal 3
2016-06-29 15:30:30: pid 9599: LOCATION:  child.c:911
failover done. shutdown host 10.50.1.217(5432)2016-06-29 15:30:39: pid 9598: LOG:  failover done. shutdown host 10.50.1.217(5432)
2016-06-29 15:30:41: pid 9598: LOCATION:  pgpool_main.c:1926
==9601== Invalid free() / delete / delete[] / realloc()
2016-06-29 15:30:43: pid 9604: LOG:  worker process received restart request
2016-06-29 15:30:43: pid 9604: LOCATION:  pool_worker_child.c:149
==9601==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9601==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9601==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9599== Invalid free() / delete / delete[] / realloc()
==9599==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9599==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9599==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9601==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9599==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9599==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9599==    by 0x425B6D: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9599==    by 0x66B866F: ??? (in /usr/lib64/libc-2.17.so)
==9601==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9601==    by 0x425B6D: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9601==    by 0x66B866F: ??? (in /usr/lib64/libc-2.17.so)
==9599==    by 0x67713F2: __select_nocancel (in /usr/lib64/libc-2.17.so)
==9601==    by 0x67713F2: __select_nocancel (in /usr/lib64/libc-2.17.so)
==9599==    by 0x426FD0: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9599==    by 0x408094: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9601==    by 0x426FD0: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9599==    by 0x40C145: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9601==    by 0x408094: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9599==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9601==    by 0x40C145: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9601==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9599==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9599== 
==9601==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9601== 
==9599== 
==9601== 
==9599== HEAP SUMMARY:
==9599==     in use at exit: 213,666 bytes in 3,059 blocks
==9599==   total heap usage: 3,140 allocs, 82 frees, 247,909 bytes allocated
==9599== 
==9601== HEAP SUMMARY:
==9601==     in use at exit: 246,434 bytes in 3,063 blocks
==9601==   total heap usage: 3,263 allocs, 201 frees, 519,627 bytes allocated
==9601== 
==9599== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 264
==9601== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 268
==9599==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9599==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9599==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9599==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9599== 
==9599== LEAK SUMMARY:
==9599==    definitely lost: 192 bytes in 1 blocks
==9599==    indirectly lost: 2,037 bytes in 23 blocks
==9599==      possibly lost: 0 bytes in 0 blocks
==9599==    still reachable: 211,437 bytes in 3,035 blocks
==9599==         suppressed: 0 bytes in 0 blocks
==9599== Reachable blocks (those to which a pointer was found) are not shown.
==9599== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9599== 
==9599== For counts of detected and suppressed errors, rerun with: -v
==9599== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
==9601==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9601==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9601==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9601==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9601== 
==9601== LEAK SUMMARY:
==9601==    definitely lost: 192 bytes in 1 blocks
==9601==    indirectly lost: 2,037 bytes in 23 blocks
==9601==      possibly lost: 0 bytes in 0 blocks
==9601==    still reachable: 244,205 bytes in 3,039 blocks
==9601==         suppressed: 0 bytes in 0 blocks
==9601== Reachable blocks (those to which a pointer was found) are not shown.
==9601== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9601== 
==9601== For counts of detected and suppressed errors, rerun with: -v
==9601== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
==9602== Invalid free() / delete / delete[] / realloc()
==9602==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9602==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9602==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9602==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9602==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9602==    by 0x425B6D: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9602==    by 0x66B866F: ??? (in /usr/lib64/libc-2.17.so)
==9602==    by 0x67713F2: __select_nocancel (in /usr/lib64/libc-2.17.so)
==9602==    by 0x426FD0: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9602==    by 0x408094: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9602==    by 0x40C145: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9602==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9602==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9602== 
==9602== 
==9602== HEAP SUMMARY:
==9602==     in use at exit: 246,434 bytes in 3,063 blocks
==9602==   total heap usage: 3,200 allocs, 138 frees, 391,960 bytes allocated
==9602== 
==9602== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 268
==9602==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9602==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9602==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9602==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9602== 
==9602== LEAK SUMMARY:
==9602==    definitely lost: 192 bytes in 1 blocks
==9602==    indirectly lost: 2,037 bytes in 23 blocks
==9602==      possibly lost: 0 bytes in 0 blocks
==9602==    still reachable: 244,205 bytes in 3,039 blocks
==9602==         suppressed: 0 bytes in 0 blocks
==9602== Reachable blocks (those to which a pointer was found) are not shown.
==9602== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9602== 
==9602== For counts of detected and suppressed errors, rerun with: -v
==9602== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
==9600== Invalid free() / delete / delete[] / realloc()
==9600==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9600==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9600==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9600==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9600==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9600==    by 0x425B6D: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x66B866F: ??? (in /usr/lib64/libc-2.17.so)
==9600==    by 0x67713F2: __select_nocancel (in /usr/lib64/libc-2.17.so)
==9600==    by 0x428919: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x428BD4: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x428D76: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x441419: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9600== 
==9600== 
==9600== HEAP SUMMARY:
==9600==     in use at exit: 1,869,801,752 bytes in 656,297 blocks
==9600==   total heap usage: 2,178,715 allocs, 1,522,419 frees, 4,784,540,318 bytes allocated
==9600== 
==9604== Invalid free() / delete / delete[] / realloc()
==9604==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9604==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9604==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9604==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9604==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9604==    by 0x4427D2: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9604==    by 0x40823A: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9604==    by 0x40C3D0: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9604==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9604==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9604==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9604== 
==9604== 
==9604== HEAP SUMMARY:
==9604==     in use at exit: 209,490 bytes in 3,058 blocks
==9604==   total heap usage: 3,138 allocs, 81 frees, 243,165 bytes allocated
==9604== 
==9604== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 263
==9604==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9604==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9604==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9604==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9604== 
==9604== LEAK SUMMARY:
==9604==    definitely lost: 192 bytes in 1 blocks
==9604==    indirectly lost: 2,037 bytes in 23 blocks
==9604==      possibly lost: 0 bytes in 0 blocks
==9604==    still reachable: 207,261 bytes in 3,034 blocks
==9604==         suppressed: 0 bytes in 0 blocks
==9604== Reachable blocks (those to which a pointer was found) are not shown.
==9604== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9604== 
==9604== For counts of detected and suppressed errors, rerun with: -v
==9604== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
==9600== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 259 of 305
==9600==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9600==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9600== 
==9600== 6,045 bytes in 494 blocks are definitely lost in loss record 267 of 305
==9600==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9600==    by 0x6709AB9: strdup (in /usr/lib64/libc-2.17.so)
==9600==    by 0x442B68: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x448417: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x443A53: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x444312: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x436D14: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x437820: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x42C13C: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x42CBA2: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x427578: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600==    by 0x408094: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9600== 
==9600== LEAK SUMMARY:
==9600==    definitely lost: 6,237 bytes in 495 blocks
==9600==    indirectly lost: 2,037 bytes in 23 blocks
==9600==      possibly lost: 0 bytes in 0 blocks
==9600==    still reachable: 1,869,793,478 bytes in 655,779 blocks
==9600==         suppressed: 0 bytes in 0 blocks
==9600== Reachable blocks (those to which a pointer was found) are not shown.
==9600== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9600== 
==9600== For counts of detected and suppressed errors, rerun with: -v
==9600== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 1 from 1)
2016-06-29 15:31:01: pid 13028: LOG:  pgpool detected difference of the number of inserted, updated or deleted tuples. Possible last query was: "update ssh_ca_certificates set c_publish_status = $1  where c_id = $2 "
2016-06-29 15:31:01: pid 13028: LOCATION:  CommandComplete.c:341
2016-06-29 15:31:01: pid 13028: LOG:  processing command complete
2016-06-29 15:31:01: pid 13028: DETAIL:  CommandComplete: Number of affected tuples are: 0 1 1 -1
2016-06-29 15:31:01: pid 13028: LOCATION:  CommandComplete.c:353
2016-06-29 15:38:37: pid 9603: LOG:  forked new pcp worker, pid=13364 socket=7
2016-06-29 15:38:37: pid 9603: LOCATION:  pcp_child.c:275
2016-06-29 15:38:37: pid 9603: LOG:  restart request received in pcp child process
2016-06-29 15:38:37: pid 9603: LOCATION:  pcp_child.c:150
==13364== Invalid free() / delete / delete[] / realloc()
==13364==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13364==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==13364==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==13364==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==13364==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==13364==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13364==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13364==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13364==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13364==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13364==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==13364==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==13364== 
==13364== 
==13364== HEAP SUMMARY:
==13364==     in use at exit: 225,874 bytes in 3,060 blocks
==13364==   total heap usage: 3,234 allocs, 175 frees, 269,881 bytes allocated
==13364== 
==13364== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 265
==13364==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13364==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13364==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13364==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==13364== 
==13364== LEAK SUMMARY:
==13364==    definitely lost: 192 bytes in 1 blocks
==13364==    indirectly lost: 2,037 bytes in 23 blocks
==13364==      possibly lost: 0 bytes in 0 blocks
==13364==    still reachable: 223,645 bytes in 3,036 blocks
==13364==         suppressed: 0 bytes in 0 blocks
==13364== Reachable blocks (those to which a pointer was found) are not shown.
==13364== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==13364== 
==13364== For counts of detected and suppressed errors, rerun with: -v
==13364== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
==9603== Invalid free() / delete / delete[] / realloc()
==9603==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9603==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==9603==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==9603==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==9603==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==9603==    by 0x417EBD: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9603==    by 0x418804: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9603==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9603==    by 0x40C3C5: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9603==    by 0x406741: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9603==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9603==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==9603== 
==9603== 
==9603== HEAP SUMMARY:
==9603==     in use at exit: 209,490 bytes in 3,058 blocks
==9603==   total heap usage: 3,230 allocs, 173 frees, 244,545 bytes allocated
==9603== 
==9603== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 263
==9603==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==9603==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9603==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==9603==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==9603== 
==9603== LEAK SUMMARY:
==9603==    definitely lost: 192 bytes in 1 blocks
==9603==    indirectly lost: 2,037 bytes in 23 blocks
==9603==      possibly lost: 0 bytes in 0 blocks
==9603==    still reachable: 207,261 bytes in 3,034 blocks
==9603==         suppressed: 0 bytes in 0 blocks
==9603== Reachable blocks (those to which a pointer was found) are not shown.
==9603== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9603== 
==9603== For counts of detected and suppressed errors, rerun with: -v
==9603== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 15:38:38: pid 9598: LOG:  PCP child 9603 exits with status 0 in failover()
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:1968
2016-06-29 15:38:38: pid 9598: LOG:  fork a new PCP child pid 13367 in failover()
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:1972
2016-06-29 15:38:38: pid 9598: LOG:  child process with pid: 9599 exits with status 0
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2195
2016-06-29 15:38:38: pid 9598: LOG:  child process with pid: 9599 exited with success and will not be restarted
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2287
2016-06-29 15:38:38: pid 9598: LOG:  child process with pid: 9600 exits with status 0
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2195
2016-06-29 15:38:38: pid 9598: LOG:  child process with pid: 9600 exited with success and will not be restarted
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2287
2016-06-29 15:38:38: pid 9598: LOG:  child process with pid: 9601 exits with status 0
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2195
2016-06-29 15:38:38: pid 9598: LOG:  child process with pid: 9601 exited with success and will not be restarted
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2287
2016-06-29 15:38:38: pid 9598: LOG:  child process with pid: 9602 exits with status 0
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2195
2016-06-29 15:38:38: pid 9598: LOG:  child process with pid: 9602 exited with success and will not be restarted
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2287
2016-06-29 15:38:38: pid 9598: LOG:  worker child process with pid: 9604 exits with status 256
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2195
2016-06-29 15:38:38: pid 9598: LOG:  fork a new worker child process with pid: 13369
2016-06-29 15:38:38: pid 9598: LOCATION:  pgpool_main.c:2281
2016-06-29 15:38:38: pid 13367: LOG:  forked new pcp worker, pid=13370 socket=7
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:275
2016-06-29 15:38:38: pid 13367: LOG:  forked new pcp worker, pid=13372 socket=7
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:275
==13370== Invalid free() / delete / delete[] / realloc()
==13370==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13370==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==13370==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==13370==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==13370==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==13370==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==    by 0x40A68F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==    by 0x40B15C: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==    by 0x40B2AA: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==    by 0x40D28F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==13370== 
==13370== 
==13370== HEAP SUMMARY:
==13370==     in use at exit: 234,066 bytes in 3,061 blocks
==13370==   total heap usage: 17,099 allocs, 14,039 frees, 17,099,798 bytes allocated
==13370== 
==13370== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 266
==13370==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13370==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13370==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==13370== 
==13370== LEAK SUMMARY:
==13370==    definitely lost: 192 bytes in 1 blocks
==13370==    indirectly lost: 2,037 bytes in 23 blocks
==13370==      possibly lost: 0 bytes in 0 blocks
==13370==    still reachable: 231,837 bytes in 3,037 blocks
==13370==         suppressed: 0 bytes in 0 blocks
==13370== Reachable blocks (those to which a pointer was found) are not shown.
==13370== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==13370== 
==13370== For counts of detected and suppressed errors, rerun with: -v
==13370== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 15:38:38: pid 13367: LOG:  PCP process with pid: 13370 exit with SUCCESS.
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:327
2016-06-29 15:38:38: pid 13367: LOG:  PCP process with pid: 13370 exits with status 0
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:341
2016-06-29 15:38:38: pid 13367: LOG:  forked new pcp worker, pid=13374 socket=7
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:275
==13372== Invalid free() / delete / delete[] / realloc()
==13372==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13372==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==13372==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==13372==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==13372==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==13372==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==    by 0x40A68F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==    by 0x40B15C: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==    by 0x40B2AA: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==    by 0x40D28F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==13372== 
==13372== 
==13372== HEAP SUMMARY:
==13372==     in use at exit: 234,066 bytes in 3,061 blocks
==13372==   total heap usage: 17,101 allocs, 14,041 frees, 17,099,828 bytes allocated
==13372== 
==13372== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 266
==13372==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13372==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13372==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==13372== 
==13372== LEAK SUMMARY:
==13372==    definitely lost: 192 bytes in 1 blocks
==13372==    indirectly lost: 2,037 bytes in 23 blocks
==13372==      possibly lost: 0 bytes in 0 blocks
==13372==    still reachable: 231,837 bytes in 3,037 blocks
==13372==         suppressed: 0 bytes in 0 blocks
==13372== Reachable blocks (those to which a pointer was found) are not shown.
==13372== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==13372== 
==13372== For counts of detected and suppressed errors, rerun with: -v
==13372== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 15:38:38: pid 13367: LOG:  PCP process with pid: 13372 exit with SUCCESS.
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:327
2016-06-29 15:38:38: pid 13367: LOG:  PCP process with pid: 13372 exits with status 0
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:341
2016-06-29 15:38:38: pid 13367: LOG:  forked new pcp worker, pid=13376 socket=7
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:275
==13374== Invalid free() / delete / delete[] / realloc()
==13374==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13374==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==13374==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==13374==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==13374==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==13374==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==    by 0x40A68F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==    by 0x40B15C: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==    by 0x40B2AA: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==    by 0x40D28F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==13374== 
==13374== 
==13374== HEAP SUMMARY:
==13374==     in use at exit: 234,066 bytes in 3,061 blocks
==13374==   total heap usage: 17,107 allocs, 14,047 frees, 17,099,918 bytes allocated
==13374== 
==13374== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 266
==13374==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13374==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13374==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==13374== 
==13374== LEAK SUMMARY:
==13374==    definitely lost: 192 bytes in 1 blocks
==13374==    indirectly lost: 2,037 bytes in 23 blocks
==13374==      possibly lost: 0 bytes in 0 blocks
==13374==    still reachable: 231,837 bytes in 3,037 blocks
==13374==         suppressed: 0 bytes in 0 blocks
==13374== Reachable blocks (those to which a pointer was found) are not shown.
==13374== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==13374== 
==13374== For counts of detected and suppressed errors, rerun with: -v
==13374== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 15:38:38: pid 13367: LOG:  PCP process with pid: 13374 exit with SUCCESS.
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:327
2016-06-29 15:38:38: pid 13367: LOG:  PCP process with pid: 13374 exits with status 0
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:341
==13376== Invalid free() / delete / delete[] / realloc()
==13376==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13376==    by 0x67E763B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==13376==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==13376==    by 0x66BBE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so)
==13376==    by 0x66BBEB4: exit (in /usr/lib64/libc-2.17.so)
==13376==    by 0x41930F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==    by 0x4188F6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==    by 0x408173: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==    by 0x40A68F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==    by 0x40B15C: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==    by 0x40B2AA: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==    by 0x40D28F: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==  Address 0x6a3e3d0 is 0 bytes inside data symbol "noai6ai_cached"
==13376== 
==13376== 
==13376== HEAP SUMMARY:
==13376==     in use at exit: 234,066 bytes in 3,061 blocks
==13376==   total heap usage: 17,113 allocs, 14,053 frees, 17,100,008 bytes allocated
==13376== 
==13376== 2,229 (192 direct, 2,037 indirect) bytes in 1 blocks are definitely lost in loss record 251 of 266
==13376==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==13376==    by 0x446B59: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==    by 0x4064C6: ??? (in /usr/pgpool-9.5/bin/pgpool)
==13376==    by 0x66A4B14: (below main) (in /usr/lib64/libc-2.17.so)
==13376== 
==13376== LEAK SUMMARY:
==13376==    definitely lost: 192 bytes in 1 blocks
==13376==    indirectly lost: 2,037 bytes in 23 blocks
==13376==      possibly lost: 0 bytes in 0 blocks
==13376==    still reachable: 231,837 bytes in 3,037 blocks
==13376==         suppressed: 0 bytes in 0 blocks
==13376== Reachable blocks (those to which a pointer was found) are not shown.
==13376== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==13376== 
==13376== For counts of detected and suppressed errors, rerun with: -v
==13376== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 1 from 1)
2016-06-29 15:38:38: pid 13367: LOG:  PCP process with pid: 13376 exit with SUCCESS.
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:327
2016-06-29 15:38:38: pid 13367: LOG:  PCP process with pid: 13376 exits with status 0
2016-06-29 15:38:38: pid 13367: LOCATION:  pcp_child.c:341
valgrind_pgpool.txt (69,465 bytes)

lassetuominen

2016-06-29 21:49

reporter  

kernelmessages.txt (12,640 bytes)
Jun 28 14:01:14 server kernel: memcheck-amd64- invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0
Jun 28 14:01:14 server kernel: memcheck-amd64- cpuset=/ mems_allowed=0
Jun 28 14:01:14 server kernel: CPU: 0 PID: 18000 Comm: memcheck-amd64- Not tainted 3.10.0-327.18.2.el7.x86_64 #1
Jun 28 14:01:14 server kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
Jun 28 14:01:15 server kernel: ffff8800b7505080 00000000df9ea10b ffff8801d2dfba10 ffffffff81635a0c
Jun 28 14:01:15 server kernel: ffff8801d2dfbaa0 ffffffff816309ac ffff880230babde0 ffff880230babdf8
Jun 28 14:01:15 server kernel: 0000000000000202 ffff8800b7505080 ffff8801d2dfba88 ffffffff81128b1f
Jun 28 14:01:15 server kernel: Call Trace:
Jun 28 14:01:15 server kernel: [<ffffffff81635a0c>] dump_stack+0x19/0x1b
Jun 28 14:01:15 server kernel: [<ffffffff816309ac>] dump_header+0x8e/0x214
Jun 28 14:01:15 server kernel: [<ffffffff81128b1f>] ? delayacct_end+0x8f/0xb0
Jun 28 14:01:15 server kernel: [<ffffffff8116d0be>] oom_kill_process+0x24e/0x3b0
Jun 28 14:01:15 server kernel: [<ffffffff8116cc26>] ? find_lock_task_mm+0x56/0xc0
Jun 28 14:01:15 server kernel: [<ffffffff81088dae>] ? has_capability_noaudit+0x1e/0x30
Jun 28 14:01:15 server kernel: [<ffffffff8116d8e6>] out_of_memory+0x4b6/0x4f0
Jun 28 14:01:15 server kernel: [<ffffffff81173ac5>] __alloc_pages_nodemask+0xa95/0xb90
Jun 28 14:01:15 server kernel: [<ffffffff811b7b8a>] alloc_pages_vma+0x9a/0x140
Jun 28 14:01:15 server kernel: [<ffffffff811a9b4b>] read_swap_cache_async+0xeb/0x160
Jun 28 14:01:15 server kernel: [<ffffffff811a9c68>] swapin_readahead+0xa8/0x110
Jun 28 14:01:15 server kernel: [<ffffffff811977e2>] handle_mm_fault+0xa42/0xf50
Jun 28 14:01:15 server kernel: [<ffffffff81516870>] ? sock_getsockopt+0x140/0x890
Jun 28 14:01:15 server kernel: [<ffffffff816416c0>] __do_page_fault+0x150/0x450
Jun 28 14:01:15 server kernel: [<ffffffff816419e3>] do_page_fault+0x23/0x80
Jun 28 14:01:15 server kernel: [<ffffffff8163dc48>] page_fault+0x28/0x30
Jun 28 14:01:15 server kernel: Mem-Info:
Jun 28 14:01:15 server kernel: Node 0 DMA per-cpu:
Jun 28 14:01:15 server kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jun 28 14:01:15 server kernel: CPU    1: hi:    0, btch:   1 usd:   0
Jun 28 14:01:15 server kernel: Node 0 DMA32 per-cpu:
Jun 28 14:01:15 server kernel: CPU    0: hi:  186, btch:  31 usd:  61
Jun 28 14:01:15 server kernel: CPU    1: hi:  186, btch:  31 usd:   9
Jun 28 14:01:15 server kernel: Node 0 Normal per-cpu:
Jun 28 14:01:15 server kernel: CPU    0: hi:  186, btch:  31 usd:  58
Jun 28 14:01:15 server kernel: CPU    1: hi:  186, btch:  31 usd: 138
Jun 28 14:01:15 server kernel: active_anon:1611779 inactive_anon:315400 isolated_anon:0#012 active_file:730 inactive_file:3272 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 free:25796 slab_reclaimable:6472 slab_unreclaimable:6762#012 mapped:28980 shmem:84924 pagetables:8987 bounce:0#012 free_cma:0
Jun 28 14:01:15 server kernel: Node 0 DMA free:15892kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jun 28 14:01:15 server kernel: lowmem_reserve[]: 0 2817 7800 7800
Jun 28 14:01:15 server kernel: Node 0 DMA32 free:44276kB min:24348kB low:30432kB high:36520kB active_anon:2234332kB inactive_anon:559416kB active_file:32kB inactive_file:6348kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129216kB managed:2884848kB mlocked:0kB dirty:0kB writeback:0kB mapped:113272kB shmem:125832kB slab_reclaimable:8640kB slab_unreclaimable:4836kB kernel_stack:800kB pagetables:15276kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:271 all_unreclaimable? yes
Jun 28 14:01:15 server kernel: lowmem_reserve[]: 0 0 4983 4983
Jun 28 14:01:15 server kernel: Node 0 Normal free:43016kB min:43096kB low:53868kB high:64644kB active_anon:4212784kB inactive_anon:702184kB active_file:2888kB inactive_file:6740kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5242880kB managed:5102956kB mlocked:0kB dirty:0kB writeback:0kB mapped:2648kB shmem:213864kB slab_reclaimable:17248kB slab_unreclaimable:22196kB kernel_stack:4592kB pagetables:20672kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:8504 all_unreclaimable? yes
Jun 28 14:01:15 server kernel: lowmem_reserve[]: 0 0 0 0
Jun 28 14:01:15 server kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15892kB
Jun 28 14:01:15 server kernel: Node 0 DMA32: 91*4kB (UE) 71*8kB (UE) 129*16kB (UE) 162*32kB (UE) 82*64kB (UE) 55*128kB (UE) 33*256kB (UE) 6*512kB (UE) 12*1024kB (EM) 0*2048kB 0*4096kB = 44276kB
Jun 28 14:01:15 server kernel: Node 0 Normal: 44*4kB (UEM) 513*8kB (UE) 178*16kB (UEM) 157*32kB (UEM) 94*64kB (E) 60*128kB (UEM) 21*256kB (UE) 3*512kB (EM) 10*1024kB (EM) 0*2048kB 0*4096kB = 43000kB
Jun 28 14:01:15 server kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 28 14:01:15 server kernel: 91715 total pagecache pages
Jun 28 14:01:16 server kernel: 2756 pages in swap cache
Jun 28 14:01:16 server kernel: Swap cache stats: add 16169243, delete 16166487, find 31899756/32473092
Jun 28 14:01:16 server kernel: Free swap  = 0kB
Jun 28 14:01:16 server kernel: Total swap = 4194300kB
Jun 28 14:01:16 server kernel: 2097022 pages RAM
Jun 28 14:01:16 server kernel: 0 pages HighMem/MovableOnly
Jun 28 14:01:16 server kernel: 96094 pages reserved
Jun 28 14:01:16 server kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Jun 28 14:01:16 server kernel: [  466]     0   466    30855      960      63     1082             0 systemd-journal
Jun 28 14:01:16 server kernel: [  492]     0   492    10840        2      21      192         -1000 systemd-udevd
Jun 28 14:01:16 server kernel: [  502]     0   502    12797       30      25       78         -1000 auditd
Jun 28 14:01:16 server kernel: [  595]     0   595     4828       39      14       36             0 irqbalance
Jun 28 14:01:16 server kernel: [  606]   996   606    28961       22      26       69             0 chronyd
Jun 28 14:01:16 server kernel: [  608]    28   608   282269       99      55      137             0 nscd
Jun 28 14:01:16 server kernel: [  616]     0   616    62188      124      66      773             0 vmtoolsd
Jun 28 14:01:16 server kernel: [  624]     0   624     6155       34      14       68             0 systemd-logind
Jun 28 14:01:16 server kernel: [  625]     0   625    52191        1      53      438             0 abrtd
Jun 28 14:01:16 server kernel: [  626]     0   626    51993       10      51      329             0 abrt-watch-log
Jun 28 14:01:16 server kernel: [  629]    81   629     6225       58      15       43          -900 dbus-daemon
Jun 28 14:01:16 server kernel: [  664]     0   664    31104       18      16      131             0 crond
Jun 28 14:01:16 server kernel: [  665]     0   665    27031        1       9       27             0 agetty
Jun 28 14:01:16 server kernel: [  668]     0   668     6066        6      16       42             0 atd
Jun 28 14:01:16 server kernel: [  850]     0   850    83235      192     161     1330             0 httpd
Jun 28 14:01:16 server kernel: [  852]     0   852   137782      133      85     2519             0 tuned
Jun 28 14:01:16 server kernel: [  853]     0   853    20215        1      42      210         -1000 sshd
Jun 28 14:01:16 server kernel: [  915]    65   915   109694       89      47      159             0 nslcd
Jun 28 14:01:16 server kernel: [ 2055]   997  2055   131468      106      53     1558             0 polkitd
Jun 28 14:01:16 server kernel: [11811]     0 11811    35904       85      73      248             0 sshd
Jun 28 14:01:16 server kernel: [11814]     0 11814    28699       83      12      347             0 bash
Jun 28 14:01:16 server kernel: [11914]     0 11914   114089      118     110      146             0 rsyslogd
Jun 28 14:01:16 server kernel: [36651]    26 36651   188515     2630      66      229         -1000 postgres
Jun 28 14:01:16 server kernel: [36652]    26 36652    48099       42      44      227         -1000 postgres
Jun 28 14:01:16 server kernel: [36654]    26 36654   188545    26254     250      233         -1000 postgres
Jun 28 14:01:16 server kernel: [36655]    26 36655   188515      518      50      241         -1000 postgres
Jun 28 14:01:16 server kernel: [36656]    26 36656   188515     1541      56      246         -1000 postgres
Jun 28 14:01:16 server kernel: [36657]    26 36657   188621       82      53      270         -1000 postgres
Jun 28 14:01:16 server kernel: [36658]    26 36658    48227       82      44      237         -1000 postgres
Jun 28 14:01:16 server kernel: [39615]   995 39615    21256        9      41      201             0 zabbix_agentd
Jun 28 14:01:16 server kernel: [39616]   995 39616    21256       71      41      190             0 zabbix_agentd
Jun 28 14:01:16 server kernel: [39617]   995 39617    21285       31      42      212             0 zabbix_agentd
Jun 28 14:01:16 server kernel: [39618]   995 39618    21285       30      42      213             0 zabbix_agentd
Jun 28 14:01:16 server kernel: [39619]   995 39619    21285       30      42      213             0 zabbix_agentd
Jun 28 14:01:16 server kernel: [39620]   995 39620    21354      124      42      212             0 zabbix_agentd
Jun 28 14:01:16 server kernel: [51497]     0 51497    35906       87      69      246             0 sshd
Jun 28 14:01:16 server kernel: [51500]     0 51500    28533        0      12      271             0 bash
Jun 28 14:01:16 server kernel: [ 5427]    48  5427    83269      170     158     1356             0 httpd
Jun 28 14:01:16 server kernel: [ 5428]    48  5428    83269      169     158     1357             0 httpd
Jun 28 14:01:16 server kernel: [ 5429]    48  5429    83269      169     158     1357             0 httpd
Jun 28 14:01:16 server kernel: [ 5430]    48  5430    83269      169     158     1357             0 httpd
Jun 28 14:01:16 server kernel: [ 5431]    48  5431    83269      169     158     1357             0 httpd
Jun 28 14:01:16 server kernel: [17956]     0 17956    38975      190      32       65             0 top
Jun 28 14:01:16 server kernel: [18000]     0 18000    52896     9783      94     7663             0 memcheck-amd64-
Jun 28 14:01:16 server kernel: [18001]     0 18001    46485     1642      79    10787             0 memcheck-amd64-
Jun 28 14:01:16 server kernel: [18002]     0 18002  2731416  1824563    5326   862488             0 memcheck-amd64-
Jun 28 14:01:16 server kernel: [18003]     0 18003    46515     1643      80    10860             0 memcheck-amd64-
Jun 28 14:01:16 server kernel: [18004]     0 18004    46511     1651      80    10844             0 memcheck-amd64-
Jun 28 14:01:16 server kernel: [18005]     0 18005    46486     1641      79    10782             0 memcheck-amd64-
Jun 28 14:01:16 server kernel: [18006]     0 18006    46485     4728      79     7695             0 memcheck-amd64-
Jun 28 14:01:16 server kernel: [18019]    26 18019   188993    19446     255      304         -1000 postgres
Jun 28 14:01:16 server kernel: [18055]     0 18055    26984        0      10       24             0 tail
Jun 28 14:01:16 server kernel: [24781]     0 24781    45144      130      42      100             0 crond
Jun 28 14:01:16 server kernel: [24782]     0 24782    28280       53      11        0             0 run-parts
Jun 28 14:01:16 server kernel: [24783]    26 24783   188748      323      54      184         -1000 postgres
Jun 28 14:01:16 server kernel: [24787]     0 24787    28280      144      11        0             0 0anacron
Jun 28 14:01:16 server kernel: [24788]     0 24788    28371       34      10        0             0 awk
Jun 28 14:01:16 server kernel: [24790]     0 24790    28280       93      11        0             0 0anacron
Jun 28 14:01:16 server kernel: Out of memory: Kill process 18002 (memcheck-amd64-) score 856 or sacrifice child
Jun 28 14:01:16 server kernel: Killed process 18002 (memcheck-amd64-) total-vm:10925664kB, anon-rss:7298252kB, file-rss:0kB
kernelmessages.txt (12,640 bytes)

lassetuominen

2016-06-29 21:50

reporter  

memoryusage.png (41,777 bytes)
memoryusage.png (41,777 bytes)

Muhammad Usama

2016-07-13 03:15

developer   ~0000900

Hi

I am not able to reproduce the memory issue using the pgbench and also the valgrind report you shared only shows a couple of KB memory unreachable.
Can you please share some more information on how to reproduce a memory issue. Did you get this with pgbench or some other specific type of load? Also if it is possible can you plese share the output of following just before pgpool runs out of memory

1- free -m
2- vmstat -s
3- top
4- cat /proc/meminfo



t-ishii

2016-09-28 10:07

developer   ~0001097

No response over 2 month. Issue closed.

Issue History

Date Modified Username Field Change
2016-06-29 21:49 lassetuominen New Issue
2016-06-29 21:49 lassetuominen File Added: valgrind_pgpool.txt
2016-06-29 21:49 lassetuominen File Added: kernelmessages.txt
2016-06-29 21:50 lassetuominen File Added: memoryusage.png
2016-07-01 15:35 t-ishii Assigned To => Muhammad Usama
2016-07-01 15:35 t-ishii Status new => assigned
2016-07-13 03:15 Muhammad Usama Note Added: 0000900
2016-07-13 03:15 Muhammad Usama Status assigned => feedback
2016-09-28 10:07 t-ishii Note Added: 0001097
2016-09-28 10:07 t-ishii Status feedback => closed