View Issue Details

IDProjectCategoryView StatusLast Update
0000031Pgpool-IIBugpublic2013-02-09 11:15
Reporterjgentsch Assigned Tot-ishii  
PrioritynormalSeveritycrashReproducibilitysometimes
Status resolvedResolutionopen 
OSCentOSOS Version5.8 
Summary0000031: pgpool V3_2_STABLE - segfault in pool_memqcache.c:2529
DescriptionI'm using shmem and master-slave streaming replication. We're getting segmentation faults occasionally. I'm not sure what it is about the SQL that may make a difference, but I'll include the back trace for the specific child process that died.
TagsNo tags attached.

Activities

jgentsch

2012-10-23 06:25

reporter  

jgentsch

2012-10-23 06:28

reporter   ~0000120

I would suspect that it may be the newline characters.

jgentsch

2012-10-23 06:54

reporter   ~0000121

I get the same result without the \n and the \".

t-ishii

2012-10-27 06:25

developer   ~0000126

Can you print the value of variables?

cache_array
cache_array->caches
cache_array->num_caches

jgentsch

2012-11-03 00:56

reporter   ~0000137

I apologize for the delayed reply.
Is this what you're looking for?

(gdb) print cache_array
$1 = (POOL_QUERY_CACHE_ARRAY *) 0x2ad527c0b010
(gdb) print cache_array->caches
$2 = {0x6d64204d4f52460a}
(gdb) print cache_array->num_caches
$3 = 1162626387

If it helps, here are my cache settings:
memory_cache_enabled = on
memqcache_method = 'shmem'
memqcache_total_size = 1073741824
memqcache_max_num_cache = 1000000
memqcache_expire = 0
memqcache_auto_cache_invalidation = on
memqcache_maxcache = 104857600
memqcache_cache_block_size = 104857600

It's a pretty large shmem allocation, but it shouldn't be an issue.

Thanks for looking at this.

t-ishii

2012-11-09 09:28

developer   ~0000142

Thanks for the back trace. I found a nasty bug with the module in charge of query cache array handling. The cache arrary is used to keep temporary cache results in a transaction. If there are more than 128 SELECTs in a transaction, the module expands cache_arrary by using realloc. However it does not record the new pointer returned by realloc. So the module keeps on using the old pointer which is absoleted. Attached patch fix the problem. Could you please try it?

t-ishii

2012-11-09 09:29

developer  

query_cache.patch (2,931 bytes)   
query_cache.patch (2,931 bytes)   

a.mizurov

2012-11-13 22:08

reporter   ~0000144

bug fixed after applying the patch

a.mizurov

2012-11-14 06:12

reporter   ~0000145

sorry, bug appears again

t-ishii

2012-11-14 07:40

developer   ~0000146

Can you please provide back trace again?

a.mizurov

2012-11-15 00:29

reporter  

error_pgpool.txt (15,664 bytes)   
error_pgpool.txt (15,664 bytes)   

a.mizurov

2012-11-15 00:30

reporter   ~0000150

Last edited: 2012-11-15 00:34

Maybe attached file <error_pgpool> help, if not, what i need do to create useful backtrace for you. We use OS Centos 6.3, postgresql-9.2 and pgpool-3.2.1 with your patch

t-ishii

2012-11-19 16:56

developer   ~0000157

error_pgpool doesn't help too much. Can you please create usefull backtrace for me?

jgentsch

2013-02-09 02:49

reporter   ~0000224

I can confirm that it fixes my particular crash as of 3.2.2. Thanks for fixing it!

t-ishii

2013-02-09 11:15

developer   ~0000225

You are welcome!

Issue History

Date Modified Username Field Change
2012-10-23 06:25 jgentsch New Issue
2012-10-23 06:25 jgentsch File Added: pgpool_V3_2_STABLE_shmem_segfault.txt
2012-10-23 06:28 jgentsch Note Added: 0000120
2012-10-23 06:54 jgentsch Note Added: 0000121
2012-10-26 19:09 t-ishii Assigned To => t-ishii
2012-10-26 19:09 t-ishii Status new => assigned
2012-10-27 06:25 t-ishii Note Added: 0000126
2012-11-03 00:56 jgentsch Note Added: 0000137
2012-11-09 09:28 t-ishii Note Added: 0000142
2012-11-09 09:29 t-ishii File Added: query_cache.patch
2012-11-13 22:08 a.mizurov Note Added: 0000144
2012-11-14 06:12 a.mizurov Note Added: 0000145
2012-11-14 07:40 t-ishii Note Added: 0000146
2012-11-15 00:29 a.mizurov File Added: error_pgpool.txt
2012-11-15 00:30 a.mizurov Note Added: 0000150
2012-11-15 00:34 a.mizurov Note Edited: 0000150
2012-11-19 16:56 t-ishii Note Added: 0000157
2012-11-26 16:05 t-ishii Status assigned => feedback
2013-02-09 02:49 jgentsch Note Added: 0000224
2013-02-09 02:49 jgentsch Status feedback => assigned
2013-02-09 11:15 t-ishii Note Added: 0000225
2013-02-09 11:15 t-ishii Status assigned => resolved