View Issue Details

IDProjectCategoryView StatusLast Update
0000263Pgpool-IIBugpublic2016-12-20 11:46
Reportertomc797Assigned Tot-ishii 
PrioritynormalSeveritycrashReproducibilityalways
Status resolvedResolutionopen 
PlatformAMD64OSUbuntuOS Version16.10
Product Version3.5.4 
Target Version3.5.5Fixed in Version 
Summary0000263: Segfault when query cache enabled
DescriptionI experience children crashing when the in memory cache is enabled:

memory_cache_enabled = on
memqcache_method = 'shmem'

Once the the cache is disabled, crashing stops.

I've experienced this problem with both pgpool2 3.4.8 and 3.5.4 and the packaged versions shipped with ubuntu (3.4.3 and 3.5.3).
Steps To ReproduceEnable caching and let openstack query the databases.
Additional InformationA crash produces:

[15520.701615] pgpool[9120]: segfault at fffffffffffffff0 ip 000055ffb940c2e3 sp 00007ffc32531668 error 5 in pgpool (deleted)[55ffb93aa000+13b000]

Using Yakkety's packaged pgpool2 3.5.3, I've obtained:

(gdb) bt
#0 pfree (pointer=0x0) at ../../src/utils/mmgr/mcxt.c:709
0000001 0x000055ffb93f0ded in pool_check_and_discard_cache_buffer (oids=0x55ffba21d0a0, num_oids=1) at query_cache/pool_memqcache.c:3084
0000002 pool_handle_query_cache (backend=backend@entry=0x55ffba1ce1e0,
    query=query@entry=0x55ffba225d30 "INSERT INTO token (id, expires, extra, valid, user_id, trust_id) VALUES ('8eae5f91b861499f96657fc0c8da9d5f', '2016-11-19T02:05:42'::timestamp, '{\"is_domain\": false, \"token_data\": {\"token\": {\"is_domain"...,
    node=node@entry=0x55ffba2178c0, state=<optimized out>) at query_cache/pool_memqcache.c:3346
0000003 0x000055ffb93e9d31 in ReadyForQuery (frontend=frontend@entry=0x55ffba1cf240, backend=backend@entry=0x55ffba1ce1e0,
    send_ready=send_ready@entry=1 '\001', cache_commit=cache_commit@entry=1 '\001') at protocol/pool_proto_modules.c:1768
0000004 0x000055ffb93ea115 in ProcessBackendResponse (frontend=frontend@entry=0x55ffba1cf240, backend=backend@entry=0x55ffba1ce1e0,
    state=state@entry=0x7ffc32533f2c, num_fields=num_fields@entry=0x7ffc32533f2a) at protocol/pool_proto_modules.c:2588
0000005 0x000055ffb93df8de in pool_process_query (frontend=0x55ffba1cf240, backend=0x55ffba1ce1e0, reset_request=reset_request@entry=0)
    at protocol/pool_process_query.c:304
0000006 0x000055ffb93da046 in do_child (fds=fds@entry=0x55ffba1c8f30) at protocol/child.c:370
0000007 0x000055ffb93b83a7 in fork_a_child (fds=0x55ffba1c8f30, id=135) at main/pgpool_main.c:678
0000008 0x000055ffb93b8d92 in reaper () at main/pgpool_main.c:2263
0000009 0x000055ffb93bd4c6 in PgpoolMain (discard_status=<optimized out>, clear_memcache_oidmaps=<optimized out>)
    at main/pgpool_main.c:429
0000010 0x000055ffb93b6a51 in main (argc=<optimized out>, argv=0x7ffc325393d8) at main/main.c:310(gdb) frame 1
0000001 0x000055ffb93f0ded in pool_check_and_discard_cache_buffer (oids=0x55ffba21d0a0, num_oids=1) at query_cache/pool_memqcache.c:3084
3084 query_cache/pool_memqcache.c: No such file or directory.
(gdb) p soids
$1 = (int *) 0x0
(gdb) p *cache->oids
$2 = {bufsize = 0, buflen = 0, buf = 0x0}
(gdb) p len
$3 = 0
(gdb) p num_oids
$4 = 1
(gdb) p oids
$5 = (int *) 0x55ffba21d0a0
(gdb) p *oids
$6 = 42557

I'm attaching two patches that remedy the problem on my system.
Tagsquery cache

Activities

tomc797

2016-11-19 10:31

reporter  

check_for_null_buffer.patch (446 bytes)
Index: pgpool-II-3.5.3/src/query_cache/pool_memqcache.c
===================================================================
--- pgpool-II-3.5.3.orig/src/query_cache/pool_memqcache.c
+++ pgpool-II-3.5.3/src/query_cache/pool_memqcache.c
@@ -3064,6 +3064,8 @@ static void pool_check_and_discard_cache
 			continue;
 
 		soids = (int *)pool_get_buffer(cache->oids, &len);
+		if (!soids || !len)
+			continue;
 
 		for(j=0;j<cache->num_oids;j++)
 		{

tomc797

2016-11-19 10:32

reporter  

pfree_check_for_null_pointer.patch (425 bytes)
Index: pgpool-II-3.5.3/src/utils/mmgr/mcxt.c
===================================================================
--- pgpool-II-3.5.3.orig/src/utils/mmgr/mcxt.c
+++ pgpool-II-3.5.3/src/utils/mmgr/mcxt.c
@@ -703,6 +703,9 @@ pfree(void *pointer)
 	Assert(pointer != NULL);
 	Assert(pointer == (void *) MAXALIGN(pointer));
 
+	if (pointer == NULL)
+		return;
+
 	/*
 	 * OK, it's probably safe to look at the chunk header.
 	 */

tomc797

2016-11-19 10:33

reporter   ~0001175

A 150 MB core dump can be provided.

t-ishii

2016-11-19 11:42

developer   ~0001176

What is the query exactly? It is omitted in the middle of the query string in the gdb stack trace.

tomc797

2016-11-21 04:03

reporter  

gdb.txt.1 (10,521 bytes)

tomc797

2016-11-21 04:03

reporter  

gdb.txt.2 (10,779 bytes)

tomc797

2016-11-21 04:05

reporter   ~0001179

I've submitted two versions of the same query in gdb.txt.1. and gdb.txt.2. The query is long, about 8k.

tomc797

2016-11-22 03:26

reporter   ~0001185

I'm attaching a debug log.

shortened.log (4,837,191 bytes)

t-ishii

2016-12-06 10:41

developer   ~0001203

Your patches look good to me. Will be included in the next minor releases. Thanks!

niekb

2016-12-09 19:28

reporter   ~0001220

Is 3.6.0 also affected by this bug? If so, what is the timeline for the next minor releases?

t-ishii

2016-12-20 11:45

developer   ~0001237

Yes. The next minor releases will be out by the end of this month.

Issue History

Date Modified Username Field Change
2016-11-19 10:31 tomc797 New Issue
2016-11-19 10:31 tomc797 File Added: check_for_null_buffer.patch
2016-11-19 10:31 tomc797 Tag Attached: query cache
2016-11-19 10:32 tomc797 File Added: pfree_check_for_null_pointer.patch
2016-11-19 10:33 tomc797 Note Added: 0001175
2016-11-19 11:10 t-ishii Assigned To => t-ishii
2016-11-19 11:10 t-ishii Status new => assigned
2016-11-19 11:42 t-ishii Note Added: 0001176
2016-11-19 11:42 t-ishii Status assigned => feedback
2016-11-21 04:03 tomc797 File Added: gdb.txt.1
2016-11-21 04:03 tomc797 File Added: gdb.txt.2
2016-11-21 04:05 tomc797 Note Added: 0001179
2016-11-21 04:05 tomc797 Status feedback => assigned
2016-11-22 03:26 tomc797 File Added: shortened.log
2016-11-22 03:26 tomc797 Note Added: 0001185
2016-12-06 10:41 t-ishii Note Added: 0001203
2016-12-09 19:28 niekb Note Added: 0001220
2016-12-20 11:45 t-ishii Note Added: 0001237
2016-12-20 11:46 t-ishii Target Version => 3.5.5
2016-12-20 11:46 t-ishii Status assigned => resolved