[Pgpool-general] pgpool crash in realloc on freebsd

[Jason Coene]

Hi All,

I'm attempting to use pgpool with a high-traffic website.  The majority of
our queries are already cached, but I'm trying to use pgpool to limit the
toll postgres' startup/teardown process takes on a per-connection basis.

I have pgpool functioning, but it's not reliable.  Some child processes are
crashing with signal 6:

pid 43893 (pgpool), uid 0: exited on signal 6 (core dumped)
pid 43848 (pgpool), uid 0: exited on signal 6 (core dumped)
pid 43886 (pgpool), uid 0: exited on signal 6 (core dumped)
pid 43987 (pgpool), uid 0: exited on signal 6 (core dumped)
pid 44008 (pgpool), uid 0: exited on signal 6 (core dumped)

I did a quick backtrace (pasted below).

Does anyone have an idea why realloc may fail?  Pgpool is using only a small
amount of memory, its being run without limits (ulimit unlimited) and the
machine has plenty of spare RAM.

It's not using enough memory to get into FreeBSD's default maximum
per-process (512MB).

Thanks,

Jason

--------------------

C01> ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) 524288
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 32768
pipe size          (512 bytes, -p) 1
stack size            (kbytes, -s) 65536
cpu time             (seconds, -t) unlimited
max user processes            (-u) 5547
virtual memory        (kbytes, -v) unlimited

C01> top
Mem: 235M Active, 611M Inact, 131M Wired, 71M Cache, 112M Buf, 1960M Free

C01> uname -a
FreeBSD c01.gotfrag.com 5.2.1-RELEASE FreeBSD 5.2.1-RELEASE #0: Mon Feb 23
20:45:55 GMT 2004     root at wv1u.btc.adaptec.com:/usr/obj/usr/src/sys/GENERIC
i386

C01> gdb /usr/local/bin/pgpool pgpool.core
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...
Core was generated by `pgpool'.
Program terminated with signal 6, Aborted.
Reading symbols from /lib/libm.so.2...done.
Loaded symbols for /lib/libm.so.2
Reading symbols from /lib/libc.so.5...done.
Loaded symbols for /lib/libc.so.5
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x280e6dcf in kill () from /lib/libc.so.5
(gdb) bt
#0  0x280e6dcf in kill () from /lib/libc.so.5
#1  0x280db878 in raise () from /lib/libc.so.5
#2  0x28153f82 in abort () from /lib/libc.so.5
#3  0x281526fe in tcflow () from /lib/libc.so.5
#4  0x28152f9b in tcflow () from /lib/libc.so.5
#5  0x2815319b in tcflow () from /lib/libc.so.5
#6  0x28153607 in realloc () from /lib/libc.so.5
#7  0x08052316 in pool_read2 (cp=0x8071800, len=1009742944) at
pool_stream.c:206
#8  0x08051910 in SimpleForwardToFrontend (kind=48 '0', frontend=0x8071000,
backend=0x80650c0)
    at pool_process_query.c:1936
#9  0x0804eb89 in pool_process_query (frontend=0x8071000, backend=0x80650c0,
connection_reuse=1)
    at pool_process_query.c:388
#10 0x0804ac42 in do_child (unix_fd=4, inet_fd=5) at child.c:307
#11 0x08049f2c in fork_a_child (unix_fd=4, inet_fd=5) at main.c:507
#12 0x0804991e in main (argc=1, argv=0xbfbfed20) at main.c:258
#13 0x080493f2 in _start ()
(gdb) q