[pgpool-general: 3937] Re: accept() scalability issues

[Michael Renner]

> On 07 Aug 2015, at 02:10, Tatsuo Ishii <ishii at postgresql.org> wrote:
>> Do you have any plans to implement an event- or lock-based approach in pgpool?
> 
> Long time ago I played with accept+semaphores approach to see if
> there's any performance improvement. The result was not positive and I
> gave up the approach. Unfortunately nor test code or test data remain
> though.


I’d be surprised if you don’t see performance problems on todays hardware with medium load. Here’s numbers from our environment.

We’ve got three pgpool nodes, next to pgpool there’s also a mongodb config server running there, doing very little work.

Each server has two Intel Xeon E5-2430 v2, 6 cores @ 2.50GHz and 16GB of 1333MHz DDR3 RAM - fairly recent hardware and not the slowest to boot with.


Here’s a vmstat -w 1 output from an idle node: 

http://nopaste.narf.at/show/25530/ <http://nopaste.narf.at/show/25530/>

About 5000 context switches/sec, not nice but not bad either.


Here’s the same command from the node running the active pgpool instance:

http://nopaste.narf.at/show/25531/ <http://nopaste.narf.at/show/25531/>

About ~400.000 context switches/second - about 60% of _ALL_ available CPU time is spent in the kernel.


tcpdump shows a connection rate of about 350 connections/sec: http://nopaste.narf.at/show/25533/ <http://nopaste.narf.at/show/25533/>


stracing a single pgpool process shows the wakeups due to the unfenced select/accept loop: http://nopaste.narf.at/show/25527/ <http://nopaste.narf.at/show/25527/>


http://nopaste.narf.at/show/jPt7GaDbsRLCxZKhKu9y/ <http://nopaste.narf.at/show/jPt7GaDbsRLCxZKhKu9y/> is the configuration we run with, relevant settings:


connection_cache = on
num_init_children = 1500
max_pool = 1
child_life_time = 300
child_max_connections = 2000
connection_life_time = 0
client_idle_limit = 600


So if there’s nothing horribly wrong/out of the ordinary with our setup, I’d pretty much say that with a high number of children fencing around select/accept is dearly needed.


best,
Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20150807/0559a6f1/attachment.html>