<div dir="ltr"><div>You got it!</div><div><br></div><div>It was my fault, I missed this entry when comparing it with 3.7.3, sorry about that.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Oct 10, 2021 at 9:51 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">In your pgpool.conf, I noticed this:<br>
<br>
relcache_size = 16384<br>
<br>
Since the default is 256, your setting will consume 64 times more<br>
memory for *each* pgpol child process. Since num_init_children is 150,<br>
which is 4 times larger than the default, your memory consumption for<br>
the relcache is 64*4 = 256 times larger than the default. I guess this<br>
is at least one of the reason which causes the memory issue.<br>
<br>
If all 150 relcache entries are filled, each process will consume 48MB<br>
of memory, which is 48 * 150 /1024 = 7GB, in total.<br>
<br>
With shared relacache enabled, each pgpool child can share the<br>
existing relacache info. So you don't need to assign that 16384 to<br>
relcache_size. Can you try with the default value of relcache_size (256)?<br>
<br>
>> I'm running a 4.4.276 kernel.<br>
>> <br>
>> 1) When psql connected:<br>
>> $ ps uax | grep pgpool | grep idle<br>
>> root 17167 0.0 0.0 39044 2576 ? S 11:30 0:00 pgpool:<br>
>> XXXX XXXX 192.168.XXX.XXX(34812) idle<br>
>> <br>
>> 2) After issue "begin;"<br>
>> $ ps uax | grep pgpool | grep idle<br>
>> root 17167 0.0 0.2 72200 37044 ? S 11:30 0:00 pgpool:<br>
>> XXXX XXXX 192.168.XXX.XXX(34812) idle in transaction<br>
>><br>
>> Then a installed pgpool on another machine running 5.14.3:<br>
>> <br>
>> 1) When psql connected:<br>
>> $ ps aux | grep pgpool | grep idle<br>
>> root 13430 0.0 0.0 13548 3096 ? S 11:58 0:00 pgpool:<br>
>> XXXX XXXX ::1(56142) idle<br>
>> <br>
>> 2) After issue "begin;"<br>
>> $ ps aux | grep pgpool | grep idle<br>
>> root 13430 0.0 0.2 46704 36804 ? S 11:58 0:00 pgpool:<br>
>> XXXX XXXX ::1(56142) idle in transaction<br>
>> <br>
>> Attached is my full pgpool.conf, maybe I'm missing something obvious.<br>
> <br>
> Thanks. I will look into that.<br>
> <br>
>> On Fri, Oct 8, 2021 at 9:55 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
>> <br>
>>> Just in case, my Linux box runs kernel 5.11.0-37.<br>
>>><br>
>>> > No. I have run similar test myself and did not notice any significant<br>
>>> > memory consuming using master HEAD (which will be Pgpool-II 4.3).<br>
>>> ><br>
>>> > 1) After just psql connected.<br>
>>> > t-ishii$ ps aux|grep pgpool|grep idle<br>
>>> > t-ishii 432882 0.0 0.0 147192 3404 pts/1 S 09:10 0:00<br>
>>> pgpool: t-ishii test idle<br>
>>> ><br>
>>> > 2) Issue "begin"<br>
>>> > t-ishii 432882 0.0 0.0 147712 5948 pts/1 S 09:10 0:00<br>
>>> pgpool: t-ishii test idle in transaction<br>
>>> ><br>
>>> > 3) Issue "select * from t1"<br>
>>> > t-ishii 432882 0.0 0.0 147712 5948 pts/1 S 09:10 0:00<br>
>>> pgpool: t-ishii test idle in transaction<br>
>>> ><br>
>>> > As you can see RSS increased after "begin", but it was just from 3404 to<br>
>>> 5948.<br>
>>> > It was not like 2564 to 37040 in your test.<br>
>>> ><br>
>>> > Can you do the same test using psql?<br>
>>> ><br>
>>> >> Any new ideias?<br>
>>> >><br>
>>> >> Can we do anything else to track it down?<br>
>>> >><br>
>>> >><br>
>>> >><br>
>>> >> On Thu, Sep 30, 2021 at 9:04 AM Luiz Pasqual <<a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a>> wrote:<br>
>>> >><br>
>>> >>> It just shows:<br>
>>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [3-1] 2021-09-30 09:00:41: pid<br>
>>> 2537:<br>
>>> >>> LOG: Query message from frontend.<br>
>>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [3-2] 2021-09-30 09:00:41: pid<br>
>>> 2537:<br>
>>> >>> DETAIL: query: "BEGIN"<br>
>>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [4-1] 2021-09-30 09:00:41: pid<br>
>>> 2537:<br>
>>> >>> LOG: Query message from frontend.<br>
>>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [4-2] 2021-09-30 09:00:41: pid<br>
>>> 2537:<br>
>>> >>> DETAIL: query: "SELECT * from uf"<br>
>>> >>><br>
>>> >>><br>
>>> >>> On Wed, Sep 29, 2021 at 6:38 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> wrote:<br>
>>> >>><br>
>>> >>>> I am not familiar with python. It would be helpful if you can share<br>
>>> >>>> the pgpool log with log_client_messages enabled while running the<br>
>>> >>>> python session. This will show what messages the python client sends<br>
>>> >>>> to pgpool.<br>
>>> >>>><br>
>>> >>>> > There is nothing special, it's a python application using psycopg to<br>
>>> >>>> > connect.<br>
>>> >>>> ><br>
>>> >>>> > I've made a very simple test here, using:<br>
>>> >>>> > Python 2.7.17<br>
>>> >>>> > Psycopg 2.8.6<br>
>>> >>>> > We have another setup running with Python 3, it seems to have the<br>
>>> same<br>
>>> >>>> > result.<br>
>>> >>>> ><br>
>>> >>>> > $ bin/python<br>
>>> >>>> > Python 2.7.17 (default, Oct 20 2019, 14:16:51)<br>
>>> >>>> > [GCC 5.5.0] on linux2<br>
>>> >>>> > Type "help", "copyright", "credits" or "license" for more<br>
>>> information.<br>
>>> >>>> >>>> import psycopg2<br>
>>> >>>> >>>> conn = psycopg2.connect("dbname='XXXX' user='XXXX'<br>
>>> >>>> > host='192.168.XXX.XXX' password='XXXXXX' port='9999'")<br>
>>> >>>> >>>> cur = conn.cursor()<br>
>>> >>>> >>>> cur.execute("""SELECT * from XX""")<br>
>>> >>>> ><br>
>>> >>>> > This is a very simple select, it returns 27 results with 2 columns,<br>
>>> an<br>
>>> >>>> ID e<br>
>>> >>>> > a string (two characters in each row)<br>
>>> >>>> ><br>
>>> >>>> > pgpool 3.7.3:<br>
>>> >>>> > After "psycopg.connect":<br>
>>> >>>> > root 3230 0.0 0.0 36200 2544 ? S 10:16 0:00<br>
>>> pgpool:<br>
>>> >>>> > XXXX XXXX 192.168.XXX.XXX(59576) idle<br>
>>> >>>> ><br>
>>> >>>> > After "cur.execute":<br>
>>> >>>> > root 3230 0.0 0.0 41400 9272 ? S 10:16 0:00<br>
>>> pgpool:<br>
>>> >>>> > XXXX XXXX 192.168.XXX.XXX(59576) idle in transaction<br>
>>> >>>> ><br>
>>> >>>> > pgpool 4.1.8:<br>
>>> >>>> > After "psycopg.connect":<br>
>>> >>>> > root 5317 0.0 0.0 39044 2564 ? S 11:35 0:00<br>
>>> pgpool:<br>
>>> >>>> > zope nube 192.168.XXX.XXX(54784) idle<br>
>>> >>>> ><br>
>>> >>>> > After "cur.execute":<br>
>>> >>>> > root 5317 0.0 0.2 72200 37040 ? S 11:35 0:00<br>
>>> pgpool:<br>
>>> >>>> > zope nube 192.168.XXX.XXX(54784) idle in transaction<br>
>>> >>>> ><br>
>>> >>>> > Do you think this is enough to reproduce? Or do you need something<br>
>>> >>>> > different? Let me know, I'm here to help.<br>
>>> >>>> ><br>
>>> >>>> ><br>
>>> >>>> > On Wed, Sep 29, 2021 at 8:54 AM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> >>>> wrote:<br>
>>> >>>> ><br>
>>> >>>> >> So enable_shared_relcache is irrelevant.<br>
>>> >>>> >><br>
>>> >>>> >> Problem is, I cannot reproduce your problem here. It would be<br>
>>> greately<br>
>>> >>>> >> helpful if you can provide a self containing test case. If you<br>
>>> cannot,<br>
>>> >>>> >> I would like to find a pattern (query) which triggers the problem.<br>
>>> >>>> >><br>
>>> >>>> >> - Does the query use extended protocol?<br>
>>> >>>> >><br>
>>> >>>> >> - What kind of queries does the application use?<br>
>>> >>>> >> INSERT/UPDATE/SELECT/DELETE??<br>
>>> >>>> >><br>
>>> >>>> >> - IS there anything special with the application?<br>
>>> >>>> >><br>
>>> >>>> >> > I tested 4.1.0 now and it shows the same behaviour.<br>
>>> >>>> >> ><br>
>>> >>>> >> > Also, disabling enable_shared_relcache did not solve the problem.<br>
>>> >>>> >> ><br>
>>> >>>> >> > On Tue, Sep 28, 2021 at 9:10 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a><br>
>>> ><br>
>>> >>>> wrote:<br>
>>> >>>> >> ><br>
>>> >>>> >> >> Ok. Let's assume 4.1.0 introduced the problem for now (it's<br>
>>> still<br>
>>> >>>> >> >> possible that certain minor version of 4.1 introduced the<br>
>>> issue, but<br>
>>> >>>> >> >> for now I neglect the possibility).<br>
>>> >>>> >> >><br>
>>> >>>> >> >> 4.1.0 includes following major changes especially related to<br>
>>> memory.<br>
>>> >>>> >> >><br>
>>> >>>> >> >> (1) Shared relation cache allows to reuse relation cache among<br>
>>> >>>> >> >> sessions to reduce internal queries against PostgreSQL<br>
>>> system<br>
>>> >>>> >> >> catalogs.<br>
>>> >>>> >> >><br>
>>> >>>> >> >> (this feature uses shared memory even if<br>
>>> memory_cache_enabled =<br>
>>> >>>> >> off).<br>
>>> >>>> >> >><br>
>>> >>>> >> >> (2) Have separate SQL parser for DML statements to eliminate<br>
>>> >>>> >> >> unnecessary parsing effort.<br>
>>> >>>> >> >><br>
>>> >>>> >> >> (this allocates additional process memory for certain query)<br>
>>> >>>> >> >><br>
>>> >>>> >> >> For (1) we can disable the feature by:<br>
>>> >>>> >> >><br>
>>> >>>> >> >> enable_shared_relcache = off<br>
>>> >>>> >> >><br>
>>> >>>> >> >> Can you try it out?<br>
>>> >>>> >> >><br>
>>> >>>> >> >> > I did those tests:<br>
>>> >>>> >> >> > 4.0.15 is running fine, when I upgraded to 4.1.8 it started to<br>
>>> >>>> >> consume a<br>
>>> >>>> >> >> > lot of memory.<br>
>>> >>>> >> >> ><br>
>>> >>>> >> >> > On Tue, Sep 28, 2021 at 8:30 AM Tatsuo Ishii <<br>
>>> <a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> >>>> >> wrote:<br>
>>> >>>> >> >> ><br>
>>> >>>> >> >> >> Ok. 4.2.1 consumes 10 times of memory than 3.7.3 (maybe 5,<br>
>>> not 10<br>
>>> >>>> >> >> >> because in your test connections of 4.2.1 is twice of 3.7.3).<br>
>>> >>>> >> >> >><br>
>>> >>>> >> >> >> Next thing I need to know is, from which version of pgpool<br>
>>> >>>> started to<br>
>>> >>>> >> >> >> consume lot of memory. Is it possible to run the same test<br>
>>> >>>> against<br>
>>> >>>> >> >> >> 4.0 and 4.1? Ideally from which version, including minor<br>
>>> version<br>
>>> >>>> will<br>
>>> >>>> >> >> >> be a lot helpful. I wish I could the test myself but I don't<br>
>>> >>>> know to<br>
>>> >>>> >> >> >> do it on my environment.<br>
>>> >>>> >> >> >><br>
>>> >>>> >> >> >> > Sure.<br>
>>> >>>> >> >> >> ><br>
>>> >>>> >> >> >> > On 3.7.3:<br>
>>> >>>> >> >> >> > $ ps aux | grep pgpool | awk '{sum += $6}END{print sum}'<br>
>>> >>>> >> >> >> > 896752<br>
>>> >>>> >> >> >> ><br>
>>> >>>> >> >> >> > On 4.2.1:<br>
>>> >>>> >> >> >> > $ ps aux | grep pgpool | awk '{sum += $6}END{print sum}'<br>
>>> >>>> >> >> >> > 9969280<br>
>>> >>>> >> >> >> ><br>
>>> >>>> >> >> >> ><br>
>>> >>>> >> >> >> > On Mon, Sep 27, 2021 at 11:41 PM Tatsuo Ishii <<br>
>>> >>>> <a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> >>>> >> >> >> wrote:<br>
>>> >>>> >> >> >> ><br>
>>> >>>> >> >> >> >> Ok, let's see how much RSS pgpool consume in total. Can<br>
>>> you<br>
>>> >>>> share<br>
>>> >>>> >> the<br>
>>> >>>> >> >> >> >> result of the command for both pgpool 4.2 and 3.7?<br>
>>> >>>> >> >> >> >><br>
>>> >>>> >> >> >> >> ps aux | grep pgpool | awk '{sum += $6}END{print sum}'<br>
>>> >>>> >> >> >> >><br>
>>> >>>> >> >> >> >> > I see.<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > The problem is, the OOM happens when all RAM is already<br>
>>> >>>> >> >> compromised by<br>
>>> >>>> >> >> >> >> > pgpool (all 16Gb of RAM is dedicated to pgpool), so any<br>
>>> >>>> common<br>
>>> >>>> >> >> request<br>
>>> >>>> >> >> >> >> will<br>
>>> >>>> >> >> >> >> > die anyway.<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > It happens on versions: 4.2.5 and 4.2.1. Now, I'm<br>
>>> running<br>
>>> >>>> the<br>
>>> >>>> >> same<br>
>>> >>>> >> >> >> >> > application on 3.7.3 and it's running great.<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > I don't know if it helps, but just to demonstrate the<br>
>>> >>>> amount of<br>
>>> >>>> >> >> RAM is<br>
>>> >>>> >> >> >> >> > necessary to run a few connections I created another<br>
>>> VM, and<br>
>>> >>>> >> >> >> comparing:<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > 62 active connections running on 4.2.1 (same behaviour<br>
>>> on<br>
>>> >>>> >> 4.2.5):<br>
>>> >>>> >> >> >> >> > $ vmstat -a -SM<br>
>>> >>>> >> >> >> >> > procs -----------memory---------- ---swap-- -----io----<br>
>>> >>>> >> -system--<br>
>>> >>>> >> >> >> >> > ------cpu-----<br>
>>> >>>> >> >> >> >> > r b swpd free inact active si so bi<br>
>>> bo in<br>
>>> >>>> >> cs<br>
>>> >>>> >> >> us<br>
>>> >>>> >> >> >> sy<br>
>>> >>>> >> >> >> >> id<br>
>>> >>>> >> >> >> >> > wa st<br>
>>> >>>> >> >> >> >> > 0 0 0 4829 173 10772 0 0 2<br>
>>> 1 68<br>
>>> >>>> >> 56<br>
>>> >>>> >> >> 1<br>
>>> >>>> >> >> >> 1<br>
>>> >>>> >> >> >> >> 99<br>
>>> >>>> >> >> >> >> > 0 0<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > 31 active connections running on 3.7.3:<br>
>>> >>>> >> >> >> >> > $ vmstat -a -SM<br>
>>> >>>> >> >> >> >> > procs -----------memory---------- ---swap-- -----io----<br>
>>> >>>> >> -system--<br>
>>> >>>> >> >> >> >> > ------cpu-----<br>
>>> >>>> >> >> >> >> > r b swpd free inact active si so bi<br>
>>> bo in<br>
>>> >>>> >> cs<br>
>>> >>>> >> >> us<br>
>>> >>>> >> >> >> sy<br>
>>> >>>> >> >> >> >> id<br>
>>> >>>> >> >> >> >> > wa st<br>
>>> >>>> >> >> >> >> > 0 0 0 15174 73 635 0 0 1<br>
>>> 1 76<br>
>>> >>>> >> 3<br>
>>> >>>> >> >> 0<br>
>>> >>>> >> >> >> 0<br>
>>> >>>> >> >> >> >> 100<br>
>>> >>>> >> >> >> >> > 0 0<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > Both are running basically the same application.<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > Seeing the RAM consumption of 3.7.3, it does not seem<br>
>>> to be<br>
>>> >>>> >> >> receiving<br>
>>> >>>> >> >> >> >> such<br>
>>> >>>> >> >> >> >> > a big requests. Is there a way to track it down?<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > On Mon, Sep 27, 2021 at 9:07 AM Tatsuo Ishii <<br>
>>> >>>> >> <a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> >>>> >> >> >> wrote:<br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> At this point I cannot judge whether the problem is<br>
>>> caused<br>
>>> >>>> by a<br>
>>> >>>> >> >> >> pgpool<br>
>>> >>>> >> >> >> >> >> bug or client's resource request is too much.<br>
>>> >>>> >> >> >> >> >><br>
>>> >>>> >> >> >> >> >> Typical bug which requests too much memory allocation<br>
>>> is<br>
>>> >>>> not<br>
>>> >>>> >> >> >> something<br>
>>> >>>> >> >> >> >> >> like this because 33947648 = 32MB memory request<br>
>>> itself is<br>
>>> >>>> not<br>
>>> >>>> >> >> >> insane.<br>
>>> >>>> >> >> >> >> >><br>
>>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-2]<br>
>>> 2021-09-24<br>
>>> >>>> >> >> 09:14:10:<br>
>>> >>>> >> >> >> pid<br>
>>> >>>> >> >> >> >> >> > 12650: DETAIL: Failed on request of size 33947648.<br>
>>> >>>> >> >> >> >> >><br>
>>> >>>> >> >> >> >> >> (Please let us know what version of Pgpool-II you are<br>
>>> using<br>
>>> >>>> >> >> because<br>
>>> >>>> >> >> >> >> >> it's important information to identify any known bug).<br>
>>> >>>> >> >> >> >> >><br>
>>> >>>> >> >> >> >> >> In the mean time, however, I think 32MB memory request<br>
>>> is<br>
>>> >>>> not<br>
>>> >>>> >> very<br>
>>> >>>> >> >> >> >> >> common in pgpool. One thing I wonder is, whether your<br>
>>> >>>> >> application<br>
>>> >>>> >> >> >> >> >> issues SQL which requires large memory: e.g. very long<br>
>>> SQL<br>
>>> >>>> >> >> statement,<br>
>>> >>>> >> >> >> >> >> COPY for large data set. They will request large<br>
>>> read/write<br>
>>> >>>> >> >> buffer in<br>
>>> >>>> >> >> >> >> >> pgpool.<br>
>>> >>>> >> >> >> >> >><br>
>>> >>>> >> >> >> >> >> > We saw both, but pgpool aborting is way more common:<br>
>>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-1]<br>
>>> 2021-09-24<br>
>>> >>>> >> >> 09:14:10:<br>
>>> >>>> >> >> >> pid<br>
>>> >>>> >> >> >> >> >> > 12650: ERROR: out of memory<br>
>>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-2]<br>
>>> 2021-09-24<br>
>>> >>>> >> >> 09:14:10:<br>
>>> >>>> >> >> >> pid<br>
>>> >>>> >> >> >> >> >> > 12650: DETAIL: Failed on request of size 33947648.<br>
>>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-3]<br>
>>> 2021-09-24<br>
>>> >>>> >> >> 09:14:10:<br>
>>> >>>> >> >> >> pid<br>
>>> >>>> >> >> >> >> >> > 12650: LOCATION: mcxt.c:900<br>
>>> >>>> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> > Here, two other ways we saw at the logs, but those<br>
>>> just<br>
>>> >>>> >> occurred<br>
>>> >>>> >> >> >> once<br>
>>> >>>> >> >> >> >> >> each:<br>
>>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-1]<br>
>>> 2021-09-24<br>
>>> >>>> >> >> 07:33:14:<br>
>>> >>>> >> >> >> pid<br>
>>> >>>> >> >> >> >> >> 5874:<br>
>>> >>>> >> >> >> >> >> > FATAL: failed to fork a child<br>
>>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-2]<br>
>>> 2021-09-24<br>
>>> >>>> >> >> 07:33:14:<br>
>>> >>>> >> >> >> pid<br>
>>> >>>> >> >> >> >> >> 5874:<br>
>>> >>>> >> >> >> >> >> > DETAIL: system call fork() failed with reason:<br>
>>> Cannot<br>
>>> >>>> >> allocate<br>
>>> >>>> >> >> >> memory<br>
>>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-3]<br>
>>> 2021-09-24<br>
>>> >>>> >> >> 07:33:14:<br>
>>> >>>> >> >> >> pid<br>
>>> >>>> >> >> >> >> >> 5874:<br>
>>> >>>> >> >> >> >> >> > LOCATION: pgpool_main.c:681<br>
>>> >>>> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> > And:<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691518] pgpool<br>
>>> >>>> invoked<br>
>>> >>>> >> >> >> >> oom-killer:<br>
>>> >>>> >> >> >> >> >> > gfp_mask=0x24200ca, order=0, oom_score_adj=0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691525] CPU: 1<br>
>>> >>>> PID:<br>
>>> >>>> >> 1194<br>
>>> >>>> >> >> >> Comm:<br>
>>> >>>> >> >> >> >> >> > pgpool Not tainted 4.4.276 #1<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691527]<br>
>>> Hardware<br>
>>> >>>> name:<br>
>>> >>>> >> >> >> VMware,<br>
>>> >>>> >> >> >> >> >> Inc.<br>
>>> >>>> >> >> >> >> >> > VMware Virtual Platform/440BX Desktop Reference<br>
>>> Platform,<br>
>>> >>>> >> BIOS<br>
>>> >>>> >> >> 6.00<br>
>>> >>>> >> >> >> >> >> > 12/12/2018<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691528]<br>
>>> >>>> >> 0000000000000000<br>
>>> >>>> >> >> >> >> >> > ffff8803281cbae8 ffffffff81c930e7 ffff880420efe2c0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691530]<br>
>>> >>>> >> ffff880420efe2c0<br>
>>> >>>> >> >> >> >> >> > ffff8803281cbb50 ffffffff81c8d9da ffff8803281cbb08<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691532]<br>
>>> >>>> >> ffffffff81133f1a<br>
>>> >>>> >> >> >> >> >> > ffff8803281cbb80 ffffffff81182eb0 ffff8800bba393c0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691533] Call<br>
>>> >>>> Trace:<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691540]<br>
>>> >>>> >> >> >> [<ffffffff81c930e7>]<br>
>>> >>>> >> >> >> >> >> > dump_stack+0x57/0x6d<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691542]<br>
>>> >>>> >> >> >> [<ffffffff81c8d9da>]<br>
>>> >>>> >> >> >> >> >> > dump_header.isra.9+0x54/0x1ae<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691547]<br>
>>> >>>> >> >> >> [<ffffffff81133f1a>] ?<br>
>>> >>>> >> >> >> >> >> > __delayacct_freepages_end+0x2a/0x30<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691553]<br>
>>> >>>> >> >> >> [<ffffffff81182eb0>] ?<br>
>>> >>>> >> >> >> >> >> > do_try_to_free_pages+0x350/0x3d0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691556]<br>
>>> >>>> >> >> >> [<ffffffff811709f9>]<br>
>>> >>>> >> >> >> >> >> > oom_kill_process+0x209/0x3c0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691558]<br>
>>> >>>> >> >> >> [<ffffffff81170eeb>]<br>
>>> >>>> >> >> >> >> >> > out_of_memory+0x2db/0x2f0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691561]<br>
>>> >>>> >> >> >> [<ffffffff81176111>]<br>
>>> >>>> >> >> >> >> >> > __alloc_pages_nodemask+0xa81/0xae0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691565]<br>
>>> >>>> >> >> >> [<ffffffff811ad2cd>]<br>
>>> >>>> >> >> >> >> >> > __read_swap_cache_async+0xdd/0x130<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691567]<br>
>>> >>>> >> >> >> [<ffffffff811ad337>]<br>
>>> >>>> >> >> >> >> >> > read_swap_cache_async+0x17/0x40<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691569]<br>
>>> >>>> >> >> >> [<ffffffff811ad455>]<br>
>>> >>>> >> >> >> >> >> > swapin_readahead+0xf5/0x190<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691571]<br>
>>> >>>> >> >> >> [<ffffffff8119ce3f>]<br>
>>> >>>> >> >> >> >> >> > handle_mm_fault+0xf3f/0x15e0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691574]<br>
>>> >>>> >> >> >> [<ffffffff81c9d4e2>] ?<br>
>>> >>>> >> >> >> >> >> > __schedule+0x272/0x770<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691576]<br>
>>> >>>> >> >> >> [<ffffffff8104e241>]<br>
>>> >>>> >> >> >> >> >> > __do_page_fault+0x161/0x370<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691577]<br>
>>> >>>> >> >> >> [<ffffffff8104e49c>]<br>
>>> >>>> >> >> >> >> >> > do_page_fault+0xc/0x10<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691579]<br>
>>> >>>> >> >> >> [<ffffffff81ca3782>]<br>
>>> >>>> >> >> >> >> >> > page_fault+0x22/0x30<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691581]<br>
>>> Mem-Info:<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]<br>
>>> >>>> >> >> active_anon:3564689<br>
>>> >>>> >> >> >> >> >> > inactive_anon:445592 isolated_anon:0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]<br>
>>> >>>> >> active_file:462<br>
>>> >>>> >> >> >> >> >> > inactive_file:44 isolated_file:0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]<br>
>>> >>>> unevictable:0<br>
>>> >>>> >> >> >> dirty:2<br>
>>> >>>> >> >> >> >> >> > writeback:2212 unstable:0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]<br>
>>> >>>> >> >> >> slab_reclaimable:3433<br>
>>> >>>> >> >> >> >> >> > slab_unreclaimable:5859<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]<br>
>>> >>>> mapped:989<br>
>>> >>>> >> >> >> shmem:2607<br>
>>> >>>> >> >> >> >> >> > pagetables:16367 bounce:0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]<br>
>>> >>>> free:51773<br>
>>> >>>> >> >> >> >> free_pcp:189<br>
>>> >>>> >> >> >> >> >> > free_cma:0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691589] DMA<br>
>>> >>>> >> free:15904kB<br>
>>> >>>> >> >> >> >> min:128kB<br>
>>> >>>> >> >> >> >> >> > low:160kB high:192kB active_anon:0kB<br>
>>> inactive_anon:0kB<br>
>>> >>>> >> >> >> active_file:0kB<br>
>>> >>>> >> >> >> >> >> > inactive_file:0kB unevictable:0kB isolated(anon):0kB<br>
>>> >>>> >> >> >> >> isolated(file):0kB<br>
>>> >>>> >> >> >> >> >> > present:15992kB managed:15904kB mlocked:0kB dirty:0kB<br>
>>> >>>> >> >> writeback:0kB<br>
>>> >>>> >> >> >> >> >> > mapped:0kB shmem:0kB slab_reclaimable:0kB<br>
>>> >>>> >> slab_unreclaimable:0kB<br>
>>> >>>> >> >> >> >> >> > kernel_stack:0kB pagetables:0kB unstable:0kB<br>
>>> bounce:0kB<br>
>>> >>>> >> >> >> free_pcp:0kB<br>
>>> >>>> >> >> >> >> >> > local_pcp:0kB free_cma:0kB writeback_tmp:0kB<br>
>>> >>>> pages_scanned:0<br>
>>> >>>> >> >> >> >> >> > all_unreclaimable? yes<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691590]<br>
>>> >>>> >> >> lowmem_reserve[]: 0<br>
>>> >>>> >> >> >> >> 2960<br>
>>> >>>> >> >> >> >> >> > 15991 15991<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691596] DMA32<br>
>>> >>>> >> >> free:77080kB<br>
>>> >>>> >> >> >> >> >> > min:25000kB low:31248kB high:37500kB<br>
>>> >>>> active_anon:2352676kB<br>
>>> >>>> >> >> >> >> >> > inactive_anon:590536kB active_file:472kB<br>
>>> >>>> inactive_file:140kB<br>
>>> >>>> >> >> >> >> >> > unevictable:0kB isolated(anon):0kB isolated(file):0kB<br>
>>> >>>> >> >> >> >> present:3129216kB<br>
>>> >>>> >> >> >> >> >> > managed:3043556kB mlocked:0kB dirty:0kB<br>
>>> writeback:4144kB<br>
>>> >>>> >> >> >> mapped:1140kB<br>
>>> >>>> >> >> >> >> >> > shmem:3568kB slab_reclaimable:1028kB<br>
>>> >>>> >> slab_unreclaimable:3004kB<br>
>>> >>>> >> >> >> >> >> > kernel_stack:816kB pagetables:10988kB unstable:0kB<br>
>>> >>>> bounce:0kB<br>
>>> >>>> >> >> >> >> >> > free_pcp:312kB local_pcp:196kB free_cma:0kB<br>
>>> >>>> writeback_tmp:0kB<br>
>>> >>>> >> >> >> >> >> > pages_scanned:4292 all_unreclaimable? yes<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691597]<br>
>>> >>>> >> >> lowmem_reserve[]: 0<br>
>>> >>>> >> >> >> 0<br>
>>> >>>> >> >> >> >> >> 13031<br>
>>> >>>> >> >> >> >> >> > 13031<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691601] Normal<br>
>>> >>>> >> >> free:114108kB<br>
>>> >>>> >> >> >> >> >> > min:110032kB low:137540kB high:165048kB<br>
>>> >>>> >> active_anon:11906080kB<br>
>>> >>>> >> >> >> >> >> > inactive_anon:1191832kB active_file:1376kB<br>
>>> >>>> inactive_file:36kB<br>
>>> >>>> >> >> >> >> >> > unevictable:0kB isolated(anon):0kB isolated(file):0kB<br>
>>> >>>> >> >> >> >> present:13631488kB<br>
>>> >>>> >> >> >> >> >> > managed:13343784kB mlocked:0kB dirty:8kB<br>
>>> writeback:4704kB<br>
>>> >>>> >> >> >> >> mapped:2816kB<br>
>>> >>>> >> >> >> >> >> > shmem:6860kB slab_reclaimable:12704kB<br>
>>> >>>> >> slab_unreclaimable:20432kB<br>
>>> >>>> >> >> >> >> >> > kernel_stack:4848kB pagetables:54480kB unstable:0kB<br>
>>> >>>> >> bounce:0kB<br>
>>> >>>> >> >> >> >> >> > free_pcp:444kB local_pcp:196kB free_cma:0kB<br>
>>> >>>> writeback_tmp:0kB<br>
>>> >>>> >> >> >> >> >> > pages_scanned:105664 all_unreclaimable? yes<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691602]<br>
>>> >>>> >> >> lowmem_reserve[]: 0<br>
>>> >>>> >> >> >> 0<br>
>>> >>>> >> >> >> >> 0 0<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691603] DMA:<br>
>>> 0*4kB<br>
>>> >>>> >> 0*8kB<br>
>>> >>>> >> >> >> 0*16kB<br>
>>> >>>> >> >> >> >> >> > 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB<br>
>>> >>>> >> 1*1024kB<br>
>>> >>>> >> >> (U)<br>
>>> >>>> >> >> >> >> >> 1*2048kB<br>
>>> >>>> >> >> >> >> >> > (U) 3*4096kB (M) = 15904kB<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691610] DMA32:<br>
>>> >>>> 48*4kB<br>
>>> >>>> >> >> (ME)<br>
>>> >>>> >> >> >> >> 63*8kB<br>
>>> >>>> >> >> >> >> >> > (ME) 62*16kB (E) 46*32kB (UME) 35*64kB (UME) 26*128kB<br>
>>> >>>> (UME)<br>
>>> >>>> >> >> >> 25*256kB<br>
>>> >>>> >> >> >> >> >> (UME)<br>
>>> >>>> >> >> >> >> >> > 61*512kB (UME) 28*1024kB (UME) 1*2048kB (M) 0*4096kB<br>
>>> =<br>
>>> >>>> >> 77080kB<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691616]<br>
>>> Normal:<br>
>>> >>>> >> 165*4kB<br>
>>> >>>> >> >> >> (MEH)<br>
>>> >>>> >> >> >> >> >> > 317*8kB (UMEH) 442*16kB (UMEH) 289*32kB (UMEH)<br>
>>> 162*64kB<br>
>>> >>>> >> (UMEH)<br>
>>> >>>> >> >> >> >> 121*128kB<br>
>>> >>>> >> >> >> >> >> > (UMEH) 74*256kB (UMEH) 33*512kB (UMEH) 24*1024kB (ME)<br>
>>> >>>> >> 0*2048kB<br>
>>> >>>> >> >> >> >> 2*4096kB<br>
>>> >>>> >> >> >> >> >> (M)<br>
>>> >>>> >> >> >> >> >> > = 113980kB<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691623] 5552<br>
>>> total<br>
>>> >>>> >> >> pagecache<br>
>>> >>>> >> >> >> >> pages<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691624] 2355<br>
>>> >>>> pages in<br>
>>> >>>> >> >> swap<br>
>>> >>>> >> >> >> >> cache<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691625] Swap<br>
>>> cache<br>
>>> >>>> >> stats:<br>
>>> >>>> >> >> >> add<br>
>>> >>>> >> >> >> >> >> > 5385308, delete 5382953, find 1159094/1325033<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691626] Free<br>
>>> >>>> swap =<br>
>>> >>>> >> 0kB<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691626] Total<br>
>>> >>>> swap =<br>
>>> >>>> >> >> >> 4194300kB<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691627]<br>
>>> 4194174<br>
>>> >>>> pages<br>
>>> >>>> >> RAM<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691628] 0<br>
>>> pages<br>
>>> >>>> >> >> >> >> >> HighMem/MovableOnly<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691628] 93363<br>
>>> >>>> pages<br>
>>> >>>> >> >> reserved<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691989] Out of<br>
>>> >>>> memory:<br>
>>> >>>> >> >> Kill<br>
>>> >>>> >> >> >> >> >> process<br>
>>> >>>> >> >> >> >> >> > 8975 (pgpool) score 7 or sacrifice child<br>
>>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691995] Killed<br>
>>> >>>> process<br>
>>> >>>> >> >> 8975<br>
>>> >>>> >> >> >> >> >> (pgpool)<br>
>>> >>>> >> >> >> >> >> > total-vm:337504kB, anon-rss:166824kB, file-rss:1920kB<br>
>>> >>>> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> > On Mon, Sep 27, 2021 at 1:22 AM Tatsuo Ishii <<br>
>>> >>>> >> >> <a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> >>>> >> >> >> >> wrote:<br>
>>> >>>> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> >> Hi,<br>
>>> >>>> >> >> >> >> >> >><br>
>>> >>>> >> >> >> >> >> >> > Hello,<br>
>>> >>>> >> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> >> > Our pgpool is consuming A LOT of memory and<br>
>>> frequently<br>
>>> >>>> >> dies<br>
>>> >>>> >> >> >> with:<br>
>>> >>>> >> >> >> >> Out<br>
>>> >>>> >> >> >> >> >> of<br>
>>> >>>> >> >> >> >> >> >> > memory error.<br>
>>> >>>> >> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> >> > We have 2 backends, 1 master and 1 slave. Here<br>
>>> some<br>
>>> >>>> >> config:<br>
>>> >>>> >> >> >> >> >> >> > num_init_children = 150<br>
>>> >>>> >> >> >> >> >> >> > max_pool = 1<br>
>>> >>>> >> >> >> >> >> >> > child_life_time = 300<br>
>>> >>>> >> >> >> >> >> >> > child_max_connections = 1<br>
>>> >>>> >> >> >> >> >> >> > connection_life_time = 0<br>
>>> >>>> >> >> >> >> >> >> > client_idle_limit = 0<br>
>>> >>>> >> >> >> >> >> >> > connection_cache = on<br>
>>> >>>> >> >> >> >> >> >> > load_balance_mode = on<br>
>>> >>>> >> >> >> >> >> >> > memory_cache_enabled = off<br>
>>> >>>> >> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> >> > RAM: 16Gb<br>
>>> >>>> >> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> >> > Does anyone have a clue what's going on?<br>
>>> >>>> >> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> >> > Thank you.<br>
>>> >>>> >> >> >> >> >> >><br>
>>> >>>> >> >> >> >> >> >> Is that OOM killer or pgpool itself aborted with an<br>
>>> out<br>
>>> >>>> of<br>
>>> >>>> >> >> memory<br>
>>> >>>> >> >> >> >> >> >> error? If latter, can you share the pgpool log?<br>
>>> >>>> >> >> >> >> >> >> --<br>
>>> >>>> >> >> >> >> >> >> Tatsuo Ishii<br>
>>> >>>> >> >> >> >> >> >> SRA OSS, Inc. Japan<br>
>>> >>>> >> >> >> >> >> >> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>>> >>>> >> >> >> >> >> >> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>>> >>>> >> >> >> >> >> >><br>
>>> >>>> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> ><br>
>>> >>>> >> >> >> >> >> > --<br>
>>> >>>> >> >> >> >> >> > Luiz Fernando Pasqual S. Souza<br>
>>> >>>> >> >> >> >> >> > mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
>>> >>>> >> >> >> >> >><br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> ><br>
>>> >>>> >> >> >> >> > --<br>
>>> >>>> >> >> >> >> > Luiz Fernando Pasqual S. Souza<br>
>>> >>>> >> >> >> >> > mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
>>> >>>> >> >> >> >><br>
>>> >>>> >> >> >> ><br>
>>> >>>> >> >> >> ><br>
>>> >>>> >> >> >> > --<br>
>>> >>>> >> >> >> > Luiz Fernando Pasqual S. Souza<br>
>>> >>>> >> >> >> > mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
>>> >>>> >> >> >><br>
>>> >>>> >> >> ><br>
>>> >>>> >> >> ><br>
>>> >>>> >> >> > --<br>
>>> >>>> >> >> > Luiz Fernando Pasqual S. Souza<br>
>>> >>>> >> >> > mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
>>> >>>> >> >> _______________________________________________<br>
>>> >>>> >> >> pgpool-general mailing list<br>
>>> >>>> >> >> <a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
>>> >>>> >> >> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
>>> >>>> >> >><br>
>>> >>>> >> ><br>
>>> >>>> >> ><br>
>>> >>>> >> > --<br>
>>> >>>> >> > Luiz Fernando Pasqual S. Souza<br>
>>> >>>> >> > mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
>>> >>>> >><br>
>>> >>>> ><br>
>>> >>>> ><br>
>>> >>>> > --<br>
>>> >>>> > Luiz Fernando Pasqual S. Souza<br>
>>> >>>> > mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
>>> >>>><br>
>>> >>><br>
>>> >>><br>
>>> >>> --<br>
>>> >>> Luiz Fernando Pasqual S. Souza<br>
>>> >>> mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
>>> >>><br>
>>> >><br>
>>> >><br>
>>> >> --<br>
>>> >> Luiz Fernando Pasqual S. Souza<br>
>>> >> mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
>>> > _______________________________________________<br>
>>> > pgpool-general mailing list<br>
>>> > <a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
>>> > <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
>>><br>
>> <br>
>> <br>
>> -- <br>
>> Luiz Fernando Pasqual S. Souza<br>
>> mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a><br>
> _______________________________________________<br>
> pgpool-general mailing list<br>
> <a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
_______________________________________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Luiz Fernando Pasqual S. Souza<br>mail: <a href="mailto:luiz@pasquall.com" target="_blank">luiz@pasquall.com</a></div>