[Pgpool-hackers] Resource Leak on PGPool-II stop or restart

[Harald Haspl]

Hi,

thanks for the hint!
I was using the init-script (/etc/init.d/pgpool-II-90) to start/stop pgpool.
We will change the script and monitor the behaviour during our next tests.

best regards,
Harald



----- Original Message -----
From: "Tatsuo Ishii" <ishii at sraoss.co.jp>
To: "harald haspl" <harald.haspl at apus.co.at>
Cc: pgpool-hackers at pgfoundry.org
Sent: Monday, July 4, 2011 8:46:40 AM
Subject: Re: [Pgpool-hackers] Resource Leak on PGPool-II stop or restart

> I have the following issue with PGPool-II:
> I tried 3.0.4 and 3.1.0-alpha2 with the same result. 
> 
> It sometimes happens, that after stopping PGPool, some SHM-segements don't get freed.
> Precisely I'm talking about 5 SHM-segments and 1 semaphore (as reported by ipcs).

I think what happend here was, the pgpool main parent process, who is
the owner of those resources, are still running or foced to be
terminated by signal 9 or some such(which is not recommended). If the
former, you can stop pgpool by using -m f or -m i option. If the
latter, you have to clean up the remaining resources by yourself using
ipcs.

In summary, you should shutdown pgpool in recommended way (i.e. use
pgpool -m f) if you want shutdown pgpool regardless clients are still
there.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> When I then restart pgpool-II, it fails to start with the log-output: "Address already in use".
> This is just the consecutive fault, but the most noticeable.
> 
> The real failure seems to happen on shutdown, and - as further investigations have shown - probably only if a client is still connected to pgpool.
> In that case the shm-resources are not released again.
> It is possible to trace this whith ipcs. The number of used shm-segments increase after starting pgpool, and normally revert to the old value after stopping it again. But if a client is still busy on this database, the number of used shms doesn't decrease after a shutdown, and the next restart of pgpool fails.
> 
> 
> I have attached this part of the logfile, because it would be too long to have it inline here.
> 
> But an indication might be this line:
> 2011-06-28 13:14:36 DEBUG: pid 13370: child receives smart shutdown request but it's not in idle state
> (for the rest please have a look in the attachment).
> 
> 
> 
> Well, the number of 6 leaks on a restart doesn't seem to be much, but our system shall run in a high-available environment and it might be necessary that pgpool needs to be restarted by the cluster software after any process fails.
> Also could it be necessary to restart pgpool because of a configuration-change of the backends.
> 
> And during tests we already came to the point, when after 100 restarts it was not possible to start it any more because no resources where available.
> 
> Example of startup-log after 100 restarts: "could not create semaphores".
> 
> 2011-06-28 15:02:43 DEBUG: pid 6000: num_backends: 2 total_weight: 1.000000
> 2011-06-28 15:02:43 DEBUG: pid 6000: backend 0 weight: 2147483647.000000
> 2011-06-28 15:02:43 DEBUG: pid 6000: backend 1 weight: 0.000000
> pid file found but it seems bogus. Trying to start pgpool anyway...
> 2011-06-28 15:02:43 ERROR: pid 6000: pool_init_pool_passwd: couldn't open /etc/pgpool-II-90/pool_passwd. reason: Permission denied
> 2011-06-28 15:02:43 ERROR: pid 6000: could not create 3 semaphores: No space left on device
> 2011-06-28 15:02:43 ERROR: pid 6000: Unable to create semaphores. Exiting...
> 2011-06-28 15:02:43 DEBUG: pid 6000: shmem_exit(1)
> 
> 
> 
> I have attached a logfile which contains a startup, a shutdown and the afterwards failing restart.
> Maybe anyone can identify the source of this problem.
> 
> 
> And BTW: I cannot file this as a bug (or browse the buglist) at http://pgfoundry.org/tracker/?group_id=1000055
> it says: Database Error: ERROR: could not open relation "artifact_message": No such file or directory
> 
> 
> best regards,
> Harald
> 
> 
> 
> _______________________________________________
> Pgpool-hackers mailing list
> Pgpool-hackers at pgfoundry.org
> http://pgfoundry.org/mailman/listinfo/pgpool-hackers

-- 
Harald Haspl

apus | Software GmbH
Bahnhofstrasse 1, A-8074 Graz-Raaba
T | +43 316 401629 0   F | +43 316 401629 9
http://www.apus.co.at  harald.haspl at apus.co.at