[Pgpool-general] FW: Child process was terminated by segmentation fault
Silvio Svečnjak
silvio.svecnjak at bull.hr
Tue Nov 22 10:38:06 UTC 2011
Thank you for your responses and sorry about the delay. Unfortunately, it is difficult for us to extract that part of the code from our java application as it's a complex transactional system with lots of dependencies. On the other hand we have checked and verified that this transaction is isolated in the code, i.e. this is the only transaction that is being committed to DB in this scope so that would mean that the code you were using to test is similar enough.
However, we replaced the usage of NOW with timestamp prepared in java code in the SQL statement which caused the crash and the problem doesn't seem to appear any more. Since this is not the only place in our code where we use NOW in statements, it is odd that the problem doesn't appear anywhere else, but we'll continue testing and maybe it reappears - then we could be sure that this is what's causing the problem.
Seeing your environment I can notice only that you were using newer version of postgreSQL (9.1.1 compared to 8.4.7 which we are using), so maybe that could also be the cause of the problem or the reason why you can't reproduce the error. In any case, I'll let you know if we experience any more problems during our tests.
Regards,
silvio
-----Original Message-----
From: Tatsuo Ishii [mailto:ishii at sraoss.co.jp]
Sent: 18. studeni 2011 5:57
To: Silvio Svečnjak
Cc: pgpool-general at pgfoundry.org
Subject: Re: [Pgpool-general] FW: Child process was terminated by segmentation fault
> Thank you for your effort. The problem occurs almost always, but not every time, when the following SQL query is issued:
>
> UPDATE ahs_transaction SET end_timestamp = NOW(), response_code = $1 WHERE id = $2
I did some tests with code something like below:
PreparedStatement pstmt = null;
for (int i = 0; i < N; i++) {
connection.setAutoCommit(false);
pstmt = connection.prepareStatement("UPDATE ahs_transaction SET end_timestamp = NOW(), response_code = ? WHERE id = ?");
pstmt.setInt(1, i);
pstmt.setInt(2, i);
pstmt.executeUpdate();
pstmt.close();
connection.commit();
But had not seen crash.
> Where ahs_transaction table is defined as follows:
>
> CREATE TABLE ahs_transaction
> (
> ID BIGSERIAL NOT NULL PRIMARY KEY,
> start_timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
> end_timestamp TIMESTAMP,
> request_ID INTEGER NOT NULL REFERENCES ahs_request(ID),
> response_code SMALLINT
> );
Because the table references another table, I removed the foreign key
when I did the test:
CREATE TABLE ahs_transaction
(
ID BIGSERIAL NOT NULL PRIMARY KEY,
start_timestamp TIMESTAMP NOT NULL DEFAULT NOW(),
end_timestamp TIMESTAMP,
request_ID INTEGER NOT NULL,
response_code SMALLINT
);
In summary, I could not reproduce your problem here. More details about my env:
OS: Linux kernel 2.6.35 x86_64
jdbc driver: postgresql-9.1-901.jdbc4.jar
PostgreSQL 9.1.1
java-1.6.0-openjdk-1.6.0.0-4.b22vl6.x86_64
It would be helpfull if you give me standalone sample Java program and
table (and data if neccessary) which can reproduce the proble.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp
> On this transaction our java application which is issuing the query receives an SQL exception (I/O error).
> I hope this helps. If I can help you with anything else, let me know.
> Regards,
>
> silvio
>
>
>
>
> -----Original Message-----
> From: Tatsuo Ishii [mailto:ishii at sraoss.co.jp]
> Sent: 17. studeni 2011 0:33
> To: Silvio Svečnjak
> Cc: pgpool-general at pgfoundry.org
> Subject: Re: [Pgpool-general] FW: Child process was terminated by segmentation fault
>
> Thanks. This backtrace is much better than mine (probably the best at
> this point). I'm going to debug using the info. In the mean time I
> might need additional info such as variable contents from the core. If
> it occurs, I will let you know. So please keep the core file
> executable file of pgpool and sources.
>
> Also if you know how to reproduce the problem, please let me know.
> With the info, I could reproduce the problem here and make it very
> easy(thus quickly) to find the cause of the bug. Because core is just
> the result of the bug, sometimes it is very hard to find the cause of
> the bug. Knowing how to reporduce the problem is the best way to find
> the cause.
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese: http://www.sraoss.co.jp
>
>> 2 files I sent you were the backtrace from the core dump file ($ gdb /usr/local/bin/pgpool core-file + (gdb)bt). I have repeated the error and took the backtrace again (file is enclosed), but the backtrace is the same. How to get additional info in the backtrace?
>> Regards,
>>
>> silvio
>>
>>
>>
>> -----Original Message-----
>> From: Tatsuo Ishii [mailto:ishii at sraoss.co.jp]
>> Sent: 16. studeni 2011 12:41
>> To: Silvio Svečnjak
>> Cc: pgpool-general at pgfoundry.org
>> Subject: Re: [Pgpool-general] FW: Child process was terminated by segmentation fault
>>
>> Can you show me backtrace?
>>
>> $ gdb pgpool core-file
>> (gdb)bt
>>
>> I expect to get more usefull info because the core was generated by
>> your pgpool binary.
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese: http://www.sraoss.co.jp
>>
>>> Hello!
>>>
>>> Could you please clarify what would be more useful info for further analysis? Thanks,
>>>
>>> silvio
>>>
>>>
>>> -----Original Message-----
>>> From: Tatsuo Ishii [mailto:ishii at sraoss.co.jp]
>>> Sent: 16. studeni 2011 2:03
>>> To: Silvio Svečnjak
>>> Cc: pgpool-general at pgfoundry.org
>>> Subject: Re: [Pgpool-general] FW: Child process was terminated by segmentation fault
>>>
>>> Silvio,
>>>
>>> Thank you for sending cores. I got backtrace this:
>>>
>>> gdb .libs/pgpool ~/core-pgpool-11-0-0-20473-1321347122
>>> GNU gdb (GDB) 7.2
>>> Copyright (C) 2010 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-vine-linux".
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/>...
>>> Reading symbols from /home/t-ishii/work/pgfoundry/pgpool-II-3.1/.libs/pgpool...done.
>>>
>>> warning: exec file is newer than core file.
>>> [New Thread 20473]
>>> Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
>>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>>>
>>> warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff50f70000
>>> Core was generated by `pgpool: vcms'.
>>> Program terminated with signal 11, Segmentation fault.
>>> #0 0x00000000004626ef in core_yylex_init (ptr_yy_globals=0x100) at scan.c:3444
>>> 3444 memset(*ptr_yy_globals,0x00,sizeof(struct yyguts_t));
>>> (gdb) bt
>>> #0 0x00000000004626ef in core_yylex_init (ptr_yy_globals=0x100) at scan.c:3444
>>> #1 0x00000000000000b2 in ?? ()
>>> #2 0x0000000000000050 in ?? ()
>>> #3 0x000000001c1978b0 in ?? ()
>>> #4 0x0000000000000000 in ?? ()
>>> (gdb)
>>>
>>> It seems the backtrace does not show too much usefull info. I think
>>> this is because the stack frame was trashed badly by a overrun of
>>> memory copy. I'm looking into the problem anyway but it would be great
>>> if you or someone provide me more usefull info...
>>> --
>>> Tatsuo Ishii
>>> SRA OSS, Inc. Japan
>>> English: http://www.sraoss.co.jp/index_en.php
>>> Japanese: http://www.sraoss.co.jp
>>>
>>>> Thanks for your quick response. Dump/trace files are enclosed - I hope they contain the information you need - if not please let me know what else could help you.
>>>> Regards,
>>>>
>>>> silvio
>>>>
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Tatsuo Ishii [mailto:ishii at sraoss.co.jp]
>>>> Sent: 14. studeni 2011 16:05
>>>> To: Silvio Svečnjak
>>>> Cc: pgpool-general at pgfoundry.org
>>>> Subject: Re: [Pgpool-general] FW: Child process was terminated by segmentation fault
>>>>
>>>>> We're using pgpool 3.1 with PostgreSQL 8.4.7 on RHEL 5.3 x64 where pgpool is configured for replication only on 2 backends. Pgpool is used by a real time java application which occasionally (without any visible pattern) catches an exception (org.postgresql.util.PSQLException: An I/O error occured while sending to the backend).
>>>>>
>>>>>
>>>>>
>>>>> We've determined that the exception is caused by pgpool as at the same time pgpool records the following error:
>>>>>
>>>>>
>>>>>
>>>>> 2011-11-14 14:14:45 ERROR: pid 10783: Child process 10812 was terminated by segmentation fault
>>>>>
>>>>> 2011-11-14 14:14:45 DEBUG: pid 10783: child 10812 exits with status 11 by signal 11
>>>>>
>>>>>
>>>>>
>>>>> Bigger chunk of log file is enclosed as well as the pgpool configuration file.
>>>>>
>>>>>
>>>>>
>>>>> After some research on the mailing lists, I found that some patches have been released for version 3.1 which may resolve the issue. So I downloaded and deployed the latest stable release pgpool2-V3_1_STABLE-8b47d55, but it didn't solve the problem.
>>>>>
>>>>>
>>>>>
>>>>> I would appreciate very much if someone could provide any help with this? Thank you in advance,
>>>>
>>>> Can you take a core and show us the backtrace? It will be very helpfull.
>>>> --
>>>> Tatsuo Ishii
>>>> SRA OSS, Inc. Japan
>>>> English: http://www.sraoss.co.jp/index_en.php
>>>> Japanese: http://www.sraoss.co.jp
More information about the Pgpool-general
mailing list