View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000525 | Pgpool-II | Bug | public | 2019-06-25 14:28 | 2019-08-15 19:15 |
| Reporter | guobo507 | Assigned To | t-ishii | ||
| Priority | normal | Severity | major | Reproducibility | sometimes |
| Status | resolved | Resolution | open | ||
| Platform | x86_64 | OS | CentOS | OS Version | 7.6.1810 (Core) |
| Product Version | 4.0.3 | ||||
| Target Version | 4.0.6 | Fixed in Version | 4.0.6 | ||
| Summary | 0000525: Segmentation fault occurs when memory_cache_enable is set to on | ||||
| Description | Hi there, When memory_cache_enable = on , whether using shm or memcacahed，my java project always has the following error: org.hibernate.exception.JDBCConnectionException: could not execute query at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:115) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:109) at org.hibernate.loader.Loader.doList(Loader.java:2614) at org.hibernate.loader.Loader.doList(Loader.java:2594) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2423) at org.hibernate.loader.Loader.list(Loader.java:2418) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:501) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:371) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:216) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1326) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:87) at com.yinhai.core.service.ta3.domain.dao.HibernateDAO.selectFildsList(HibernateDAO.java:439) at com.yinhai.modules.codetable.domain.dao.impl.Aa10a1DaoImpl.selectCodeList(Aa10a1DaoImpl.java:60) at com.yinhai.modules.codetable.domain.dao.impl.Aa10a1DaoImpl$$FastClassBySpringCGLIB$$6fcdd538.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.yinhai.modules.codetable.domain.dao.impl.Aa10a1DaoImpl$$EnhancerBySpringCGLIB$$a9bb5f6d.selectCodeList(<generated>) at com.yinhai.modules.codetable.domain.bpo.AppCodeBpoImpl.queryAppCode(AppCodeBpoImpl.java:67) at com.yinhai.modules.codetable.domain.bpo.AppCodeBpoImpl$$FastClassBySpringCGLIB$$e718e8c7.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.yinhai.modules.codetable.domain.bpo.AppCodeBpoImpl$$EnhancerBySpringCGLIB$$f581cb7a.queryAppCode(<generated>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy77.queryAppCode(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy77.queryAppCode(Unknown Source) at com.yinhai.modules.codetable.domain.bpo.AppCodeCacheBpoImpl.refreshAll(AppCodeCacheBpoImpl.java:141) at com.yinhai.modules.codetable.domain.bpo.AppCodeCacheBpoImpl$$FastClassBySpringCGLIB$$1b9a2859.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:721) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:656) at com.yinhai.modules.codetable.domain.bpo.AppCodeCacheBpoImpl$$EnhancerBySpringCGLIB$$2bf86ec2.refreshAll(<generated>) at com.yinhai.modules.codetable.domain.bpo.CodeTableInitService.initAppCode(CodeTableInitService.java:38) at com.yinhai.modules.codetable.domain.bpo.CodeTableInitService.init(CodeTableInitService.java:27) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1758) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1695) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542) at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443) at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325) at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107) at com.yinhai.project.listener.StartupListener.contextInitialized(StartupListener.java:18) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4861) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5322) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:974) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1850) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend. at org.postgresql.core.v3.QueryExecutorImpl.fetch(QueryExecutorImpl.java:2377) at org.postgresql.jdbc.PgResultSet.next(PgResultSet.java:1856) at com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:654) at com.alibaba.druid.filter.FilterAdapter.resultSet_next(FilterAdapter.java:1885) at com.alibaba.druid.filter.FilterChainImpl.resultSet_next(FilterChainImpl.java:651) at com.alibaba.druid.proxy.jdbc.ResultSetProxyImpl.next(ResultSetProxyImpl.java:882) at com.alibaba.druid.pool.DruidPooledResultSet.next(DruidPooledResultSet.java:69) at org.hibernate.loader.Loader.processResultSet(Loader.java:968) at org.hibernate.loader.Loader.doQuery(Loader.java:930) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:336) at org.hibernate.loader.Loader.doList(Loader.java:2611) ... 101 common frames omitted Caused by: java.io.EOFException: null at org.postgresql.core.PGStream.receiveChar(PGStream.java:308) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1952) at org.postgresql.core.v3.QueryExecutorImpl.fetch(QueryExecutorImpl.java:2372) ... 111 common frames omitted ...... The system dmesg message shows the following information when error occur: Jun 20 06:25:24 localhost kernel: pgpool[8971]: segfault at ffffffff00000272 ip 00007f967eae06e7 sp 00007ffce13c4c58 error 6 in libc-2.12.so[7f967ea57000+18a000] Jun 20 06:25:27 localhost abrt[9163]: Saved core dump of pid 8971 (/usr/pgpool-11/bin/pgpool) to /var/spool/abrt/ccpp-2019-06-20-06:25:24-8971 (1048576000 bytes) Jun 20 06:25:27 localhost abrtd: Directory 'ccpp-2019-06-20-06:25:24-8971' creation detected Jun 20 06:25:27 localhost pgpool[8958]: [7-1] 2019-06-20 14:25:27: pid 8958: WARNING: child process with pid: 8971 was terminated by segmentation fault Jun 20 06:25:27 localhost pgpool[8958]: [8-1] 2019-06-20 14:25:27: pid 8958: LOG: fork a new child process with pid: 9165 Jun 20 06:25:27 localhost pgpool[8974]: [6-1] 2019-06-20 14:25:27: pid 8974: LOG: new connection received Jun 20 06:25:27 localhost pgpool[8974]: [6-2] 2019-06-20 14:25:27: pid 8974: DETAIL: connecting host=localhost port=59906 Jun 20 06:25:27 localhost pgpool[8974]: [7-1] 2019-06-20 14:25:27: pid 8974: LOG: Parse message from frontend. Jun 20 06:25:27 localhost pgpool[8974]: [7-2] 2019-06-20 14:25:27: pid 8974: DETAIL: statement: "", query: "SET extra_float_digits = 3" Jun 20 06:25:27 localhost pgpool[8974]: [8-1] 2019-06-20 14:25:27: pid 8974: LOG: DB node id: 0 backend pid: 9167 statement: Parse: SET extra_float_digits = 3 Jun 20 06:25:27 localhost pgpool[8974]: [9-1] 2019-06-20 14:25:27: pid 8974: LOG: Bind message from frontend. ...... My web page prompt error 500 message. My system logs (messages) is: messages.2_with-memory_cache_enabled-on.txt, which also contains the pgpool-II log. By the way, when memory_cache_enable = off, everything works fine. The rpm packages we use are all from the pgdg yum repository. can somebody help？ | ||||
| Tags | No tags attached. | ||||
|
|
|
|
|
Can you share Pgpool-II log with log_client_messages = on? Also stack trace of Pgpool-II would be valuable info. |
|
|
In my pgpool.conf, log_client_messages is already enabled. When error occur, the root user received an email message and I have uploaded the file（mail-info.txt）. |
|
|
Thanks for the logs. I will look into them. |
|
|
Still struggling to find the cause of the segfault. It seems your stack trace's pid (12516) is different from the ones (8971, 8974) and appeared in the pgpool log (messages.2_with-memory_cache_enabled-on). Do you have minimum test data to reproduce the problem? I was not able to reproduce the segfault. |
|
|
TKS for the reply. i will reproduce the segfault and collect the information you need later... thanks again! |
|
|
Hi, it may be difficult for you to reproduce the segfaul, because at the moment I noticed that the error only appeared on one of our java projects. And, only appeared when we set memory_cache_enable = on. I have reproduced the segfault, the stack trace information is in the trace.txt file. The pgpool-II log is in mail.txt, which is the mail received by the root user when error occur. Apache server log is in java-log.txt file, which only include the error info. TKS a lot! |
|
|
Thank for the info. It seems pgpool segfaults at the same place as before. Can you please try out attached patch? It seems Pgpool-II may access wrong pointer. |
|
|
|
|
|
Thanks very much, but can you tell me how to use it? I have never patched pgpool-II... |
|
|
You need to install Pgpool-II from the source code. The instruction how to do it can be found in the manual: http://www.pgpool.net/docs/latest/en/html/installation.html Before going further, you can apply the patch like this: tar xf pgpool-II-4.0.5.tar.gz cd pgppool-II--4.0.5 patch -b -p1 < /path/to/bug525.diff ./configure .... |
|
|
tks, i will try it later... |
|
|
TKS for the patch, I have tested it on pgpool-II 4.0.3 and 4.0.5, both works fine. |
|
|
Glad to hear that. I am going to commit the patch. Thanks! |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2019-06-25 14:28 | guobo507 | New Issue | |
| 2019-06-25 14:28 | guobo507 | Issue generated from: 0000523 | |
| 2019-06-25 14:38 | t-ishii | Note Added: 0002674 | |
| 2019-06-25 18:13 | guobo507 | File Added: mail-info.txt | |
| 2019-06-25 18:13 | guobo507 | File Added: stack-trace-pgpool.txt | |
| 2019-06-25 18:13 | guobo507 | Note Added: 0002676 | |
| 2019-06-26 10:58 | t-ishii | Assigned To | => t-ishii |
| 2019-06-26 10:58 | t-ishii | Status | new => assigned |
| 2019-06-26 10:58 | t-ishii | Note Added: 0002681 | |
| 2019-07-02 14:56 | t-ishii | Note Added: 0002686 | |
| 2019-07-02 14:56 | t-ishii | Status | assigned => feedback |
| 2019-07-02 16:10 | guobo507 | Note Added: 0002687 | |
| 2019-07-02 16:10 | guobo507 | Status | feedback => assigned |
| 2019-07-02 18:25 | guobo507 | File Added: mail.txt | |
| 2019-07-02 18:25 | guobo507 | File Added: trace.txt | |
| 2019-07-02 18:25 | guobo507 | File Added: java-log.txt | |
| 2019-07-02 18:25 | guobo507 | Note Added: 0002688 | |
| 2019-07-02 18:50 | t-ishii | Note Added: 0002689 | |
| 2019-07-02 18:51 | t-ishii | File Added: bug525.diff | |
| 2019-07-02 18:52 | t-ishii | Status | assigned => feedback |
| 2019-07-02 23:01 | guobo507 | Note Added: 0002690 | |
| 2019-07-02 23:01 | guobo507 | Status | feedback => assigned |
| 2019-07-03 08:49 | t-ishii | Note Added: 0002691 | |
| 2019-07-03 08:50 | t-ishii | Note Edited: 0002691 | |
| 2019-07-03 08:51 | t-ishii | Status | assigned => feedback |
| 2019-07-03 08:58 | guobo507 | Note Added: 0002692 | |
| 2019-07-03 08:58 | guobo507 | Status | feedback => assigned |
| 2019-07-03 12:31 | guobo507 | Note Added: 0002693 | |
| 2019-07-03 12:53 | t-ishii | Note Added: 0002694 | |
| 2019-07-03 12:54 | t-ishii | Status | assigned => resolved |
| 2019-07-03 12:54 | t-ishii | Target Version | => 4.0.6 |
| 2019-08-15 19:15 | administrator | Fixed in Version | => 4.0.6 |
