[pgpool-general: 7748] Re: Out of memory

Tatsuo Ishii ishii at sraoss.co.jp
Sat Oct 9 09:18:07 JST 2021


No. I have run similar test myself and did not notice any significant
memory consuming using master HEAD (which will be Pgpool-II 4.3).

1) After just psql connected.
t-ishii$ ps aux|grep pgpool|grep idle
t-ishii   432882  0.0  0.0 147192  3404 pts/1    S    09:10   0:00 pgpool: t-ishii test  idle

2) Issue "begin"
t-ishii   432882  0.0  0.0 147712  5948 pts/1    S    09:10   0:00 pgpool: t-ishii test  idle in transaction

3) Issue "select * from t1"
t-ishii   432882  0.0  0.0 147712  5948 pts/1    S    09:10   0:00 pgpool: t-ishii test  idle in transaction

As you can see RSS increased after "begin", but it was just from 3404 to 5948.
It was not like 2564 to 37040 in your test.

Can you do the same test using psql?

> Any new ideias?
> 
> Can we do anything else to track it down?
> 
> 
> 
> On Thu, Sep 30, 2021 at 9:04 AM Luiz Pasqual <luiz at pasquall.com> wrote:
> 
>> It just shows:
>> Sep 30 09:00:41 pgpool pgpool[2537]: [3-1] 2021-09-30 09:00:41: pid 2537:
>> LOG:  Query message from frontend.
>> Sep 30 09:00:41 pgpool pgpool[2537]: [3-2] 2021-09-30 09:00:41: pid 2537:
>> DETAIL:  query: "BEGIN"
>> Sep 30 09:00:41 pgpool pgpool[2537]: [4-1] 2021-09-30 09:00:41: pid 2537:
>> LOG:  Query message from frontend.
>> Sep 30 09:00:41 pgpool pgpool[2537]: [4-2] 2021-09-30 09:00:41: pid 2537:
>> DETAIL:  query: "SELECT * from uf"
>>
>>
>> On Wed, Sep 29, 2021 at 6:38 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>>
>>> I am not familiar with python. It would be helpful if you can share
>>> the pgpool log with log_client_messages enabled while running the
>>> python session. This will show what messages the python client sends
>>> to pgpool.
>>>
>>> > There is nothing special, it's a python application using psycopg to
>>> > connect.
>>> >
>>> > I've made a very simple test here, using:
>>> > Python 2.7.17
>>> > Psycopg 2.8.6
>>> > We have another setup running with Python 3, it seems to have the same
>>> > result.
>>> >
>>> > $ bin/python
>>> > Python 2.7.17 (default, Oct 20 2019, 14:16:51)
>>> > [GCC 5.5.0] on linux2
>>> > Type "help", "copyright", "credits" or "license" for more information.
>>> >>>> import psycopg2
>>> >>>> conn = psycopg2.connect("dbname='XXXX' user='XXXX'
>>> > host='192.168.XXX.XXX' password='XXXXXX' port='9999'")
>>> >>>> cur = conn.cursor()
>>> >>>> cur.execute("""SELECT * from XX""")
>>> >
>>> > This is a very simple select, it returns 27 results with 2 columns, an
>>> ID e
>>> > a string (two characters in each row)
>>> >
>>> > pgpool 3.7.3:
>>> > After "psycopg.connect":
>>> > root      3230  0.0  0.0  36200  2544 ?        S    10:16   0:00 pgpool:
>>> > XXXX XXXX 192.168.XXX.XXX(59576) idle
>>> >
>>> > After "cur.execute":
>>> > root      3230  0.0  0.0  41400  9272 ?        S    10:16   0:00 pgpool:
>>> > XXXX XXXX 192.168.XXX.XXX(59576) idle in transaction
>>> >
>>> > pgpool 4.1.8:
>>> > After "psycopg.connect":
>>> > root      5317  0.0  0.0  39044  2564 ?        S    11:35   0:00 pgpool:
>>> > zope nube 192.168.XXX.XXX(54784) idle
>>> >
>>> > After "cur.execute":
>>> > root      5317  0.0  0.2  72200 37040 ?        S    11:35   0:00 pgpool:
>>> > zope nube 192.168.XXX.XXX(54784) idle in transaction
>>> >
>>> > Do you think this is enough to reproduce? Or do you need something
>>> > different? Let me know, I'm here to help.
>>> >
>>> >
>>> > On Wed, Sep 29, 2021 at 8:54 AM Tatsuo Ishii <ishii at sraoss.co.jp>
>>> wrote:
>>> >
>>> >> So enable_shared_relcache is irrelevant.
>>> >>
>>> >> Problem is, I cannot reproduce your problem here. It would be greately
>>> >> helpful if you can provide a self containing test case. If you cannot,
>>> >> I would like to find a pattern (query) which triggers the problem.
>>> >>
>>> >> - Does the query use extended protocol?
>>> >>
>>> >> - What kind of queries does the application use?
>>> >>   INSERT/UPDATE/SELECT/DELETE??
>>> >>
>>> >> - IS there anything special with the application?
>>> >>
>>> >> > I tested 4.1.0 now and it shows the same behaviour.
>>> >> >
>>> >> > Also, disabling enable_shared_relcache did not solve the problem.
>>> >> >
>>> >> > On Tue, Sep 28, 2021 at 9:10 PM Tatsuo Ishii <ishii at sraoss.co.jp>
>>> wrote:
>>> >> >
>>> >> >> Ok. Let's assume 4.1.0 introduced the problem for now (it's still
>>> >> >> possible that certain minor version of 4.1 introduced the issue, but
>>> >> >> for now I neglect the possibility).
>>> >> >>
>>> >> >> 4.1.0 includes following major changes especially related to memory.
>>> >> >>
>>> >> >> (1) Shared relation cache allows to reuse relation cache among
>>> >> >>     sessions to reduce internal queries against PostgreSQL system
>>> >> >>     catalogs.
>>> >> >>
>>> >> >>     (this feature uses shared memory even if memory_cache_enabled =
>>> >> off).
>>> >> >>
>>> >> >> (2) Have separate SQL parser for DML statements to eliminate
>>> >> >>     unnecessary parsing effort.
>>> >> >>
>>> >> >>     (this allocates additional process memory for certain query)
>>> >> >>
>>> >> >> For (1) we can disable the feature by:
>>> >> >>
>>> >> >> enable_shared_relcache = off
>>> >> >>
>>> >> >> Can you try it out?
>>> >> >>
>>> >> >> > I did those tests:
>>> >> >> > 4.0.15 is running fine, when I upgraded to 4.1.8 it started to
>>> >> consume a
>>> >> >> > lot of memory.
>>> >> >> >
>>> >> >> > On Tue, Sep 28, 2021 at 8:30 AM Tatsuo Ishii <ishii at sraoss.co.jp>
>>> >> wrote:
>>> >> >> >
>>> >> >> >> Ok. 4.2.1 consumes 10 times of memory than 3.7.3 (maybe 5, not 10
>>> >> >> >> because in your test connections of 4.2.1 is twice of 3.7.3).
>>> >> >> >>
>>> >> >> >> Next thing I need to know is, from which version of pgpool
>>> started to
>>> >> >> >> consume lot of memory.  Is it possible to run the same test
>>> against
>>> >> >> >> 4.0 and 4.1? Ideally from which version, including minor version
>>> will
>>> >> >> >> be a lot helpful. I wish I could the test myself but I don't
>>> know to
>>> >> >> >> do it on my environment.
>>> >> >> >>
>>> >> >> >> > Sure.
>>> >> >> >> >
>>> >> >> >> > On 3.7.3:
>>> >> >> >> > $ ps aux | grep pgpool | awk '{sum += $6}END{print sum}'
>>> >> >> >> > 896752
>>> >> >> >> >
>>> >> >> >> > On 4.2.1:
>>> >> >> >> > $ ps aux | grep pgpool | awk '{sum += $6}END{print sum}'
>>> >> >> >> > 9969280
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > On Mon, Sep 27, 2021 at 11:41 PM Tatsuo Ishii <
>>> ishii at sraoss.co.jp>
>>> >> >> >> wrote:
>>> >> >> >> >
>>> >> >> >> >> Ok, let's see how much RSS pgpool consume in total. Can you
>>> share
>>> >> the
>>> >> >> >> >> result of the command for both pgpool 4.2 and 3.7?
>>> >> >> >> >>
>>> >> >> >> >> ps aux | grep pgpool | awk '{sum += $6}END{print sum}'
>>> >> >> >> >>
>>> >> >> >> >> > I see.
>>> >> >> >> >> >
>>> >> >> >> >> > The problem is, the OOM happens when all RAM is already
>>> >> >> compromised by
>>> >> >> >> >> > pgpool (all 16Gb of RAM is dedicated to pgpool), so any
>>> common
>>> >> >> request
>>> >> >> >> >> will
>>> >> >> >> >> > die anyway.
>>> >> >> >> >> >
>>> >> >> >> >> > It happens on versions: 4.2.5 and 4.2.1. Now, I'm running
>>> the
>>> >> same
>>> >> >> >> >> > application on 3.7.3 and it's running great.
>>> >> >> >> >> >
>>> >> >> >> >> > I don't know if it helps, but just to demonstrate the
>>> amount of
>>> >> >> RAM is
>>> >> >> >> >> > necessary to run a few connections I created another VM, and
>>> >> >> >> comparing:
>>> >> >> >> >> >
>>> >> >> >> >> > 62 active connections running on 4.2.1 (same behaviour on
>>> >> 4.2.5):
>>> >> >> >> >> > $ vmstat -a -SM
>>> >> >> >> >> > procs -----------memory---------- ---swap-- -----io----
>>> >> -system--
>>> >> >> >> >> > ------cpu-----
>>> >> >> >> >> >  r  b   swpd   free  inact active   si   so    bi    bo   in
>>> >>  cs
>>> >> >> us
>>> >> >> >> sy
>>> >> >> >> >> id
>>> >> >> >> >> > wa st
>>> >> >> >> >> >  0  0      0   4829    173  10772    0    0     2     1   68
>>> >>  56
>>> >> >> 1
>>> >> >> >> 1
>>> >> >> >> >> 99
>>> >> >> >> >> >  0  0
>>> >> >> >> >> >
>>> >> >> >> >> > 31 active connections running on 3.7.3:
>>> >> >> >> >> > $ vmstat -a -SM
>>> >> >> >> >> > procs -----------memory---------- ---swap-- -----io----
>>> >> -system--
>>> >> >> >> >> > ------cpu-----
>>> >> >> >> >> >  r  b   swpd   free  inact active   si   so    bi    bo   in
>>> >>  cs
>>> >> >> us
>>> >> >> >> sy
>>> >> >> >> >> id
>>> >> >> >> >> > wa st
>>> >> >> >> >> >  0  0      0  15174     73    635    0    0     1     1   76
>>> >> 3
>>> >> >> 0
>>> >> >> >> 0
>>> >> >> >> >> 100
>>> >> >> >> >> >  0  0
>>> >> >> >> >> >
>>> >> >> >> >> > Both are running basically the same application.
>>> >> >> >> >> >
>>> >> >> >> >> > Seeing the RAM consumption of 3.7.3, it does not seem to be
>>> >> >> receiving
>>> >> >> >> >> such
>>> >> >> >> >> > a big requests. Is there a way to track it down?
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > On Mon, Sep 27, 2021 at 9:07 AM Tatsuo Ishii <
>>> >> ishii at sraoss.co.jp>
>>> >> >> >> wrote:
>>> >> >> >> >> >
>>> >> >> >> >> >> At this point I cannot judge whether the problem is caused
>>> by a
>>> >> >> >> pgpool
>>> >> >> >> >> >> bug or client's resource request is too much.
>>> >> >> >> >> >>
>>> >> >> >> >> >> Typical bug which requests too much memory allocation is
>>> not
>>> >> >> >> something
>>> >> >> >> >> >> like this because 33947648 = 32MB memory request itself is
>>> not
>>> >> >> >> insane.
>>> >> >> >> >> >>
>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-2] 2021-09-24
>>> >> >> 09:14:10:
>>> >> >> >> pid
>>> >> >> >> >> >> > 12650: DETAIL:  Failed on request of size 33947648.
>>> >> >> >> >> >>
>>> >> >> >> >> >> (Please let us know what version of Pgpool-II you are using
>>> >> >> because
>>> >> >> >> >> >> it's important information to identify any known bug).
>>> >> >> >> >> >>
>>> >> >> >> >> >> In the mean time, however, I think 32MB memory request is
>>> not
>>> >> very
>>> >> >> >> >> >> common in pgpool. One thing I wonder is, whether your
>>> >> application
>>> >> >> >> >> >> issues SQL which requires large memory: e.g. very long SQL
>>> >> >> statement,
>>> >> >> >> >> >> COPY for large data set. They will request large read/write
>>> >> >> buffer in
>>> >> >> >> >> >> pgpool.
>>> >> >> >> >> >>
>>> >> >> >> >> >> > We saw both, but pgpool aborting is way more common:
>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-1] 2021-09-24
>>> >> >> 09:14:10:
>>> >> >> >> pid
>>> >> >> >> >> >> > 12650: ERROR:  out of memory
>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-2] 2021-09-24
>>> >> >> 09:14:10:
>>> >> >> >> pid
>>> >> >> >> >> >> > 12650: DETAIL:  Failed on request of size 33947648.
>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-3] 2021-09-24
>>> >> >> 09:14:10:
>>> >> >> >> pid
>>> >> >> >> >> >> > 12650: LOCATION:  mcxt.c:900
>>> >> >> >> >> >> >
>>> >> >> >> >> >> > Here, two other ways we saw at the logs, but those just
>>> >> occurred
>>> >> >> >> once
>>> >> >> >> >> >> each:
>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-1] 2021-09-24
>>> >> >> 07:33:14:
>>> >> >> >> pid
>>> >> >> >> >> >> 5874:
>>> >> >> >> >> >> > FATAL:  failed to fork a child
>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-2] 2021-09-24
>>> >> >> 07:33:14:
>>> >> >> >> pid
>>> >> >> >> >> >> 5874:
>>> >> >> >> >> >> > DETAIL:  system call fork() failed with reason: Cannot
>>> >> allocate
>>> >> >> >> memory
>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-3] 2021-09-24
>>> >> >> 07:33:14:
>>> >> >> >> pid
>>> >> >> >> >> >> 5874:
>>> >> >> >> >> >> > LOCATION:  pgpool_main.c:681
>>> >> >> >> >> >> >
>>> >> >> >> >> >> > And:
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691518] pgpool
>>> invoked
>>> >> >> >> >> oom-killer:
>>> >> >> >> >> >> > gfp_mask=0x24200ca, order=0, oom_score_adj=0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691525] CPU: 1
>>> PID:
>>> >> 1194
>>> >> >> >> Comm:
>>> >> >> >> >> >> > pgpool Not tainted 4.4.276 #1
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691527] Hardware
>>> name:
>>> >> >> >> VMware,
>>> >> >> >> >> >> Inc.
>>> >> >> >> >> >> > VMware Virtual Platform/440BX Desktop Reference Platform,
>>> >> BIOS
>>> >> >> 6.00
>>> >> >> >> >> >> > 12/12/2018
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691528]
>>> >> 0000000000000000
>>> >> >> >> >> >> > ffff8803281cbae8 ffffffff81c930e7 ffff880420efe2c0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691530]
>>> >> ffff880420efe2c0
>>> >> >> >> >> >> > ffff8803281cbb50 ffffffff81c8d9da ffff8803281cbb08
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691532]
>>> >> ffffffff81133f1a
>>> >> >> >> >> >> > ffff8803281cbb80 ffffffff81182eb0 ffff8800bba393c0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691533] Call
>>> Trace:
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691540]
>>> >> >> >> [<ffffffff81c930e7>]
>>> >> >> >> >> >> > dump_stack+0x57/0x6d
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691542]
>>> >> >> >> [<ffffffff81c8d9da>]
>>> >> >> >> >> >> > dump_header.isra.9+0x54/0x1ae
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691547]
>>> >> >> >> [<ffffffff81133f1a>] ?
>>> >> >> >> >> >> > __delayacct_freepages_end+0x2a/0x30
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691553]
>>> >> >> >> [<ffffffff81182eb0>] ?
>>> >> >> >> >> >> > do_try_to_free_pages+0x350/0x3d0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691556]
>>> >> >> >> [<ffffffff811709f9>]
>>> >> >> >> >> >> > oom_kill_process+0x209/0x3c0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691558]
>>> >> >> >> [<ffffffff81170eeb>]
>>> >> >> >> >> >> > out_of_memory+0x2db/0x2f0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691561]
>>> >> >> >> [<ffffffff81176111>]
>>> >> >> >> >> >> > __alloc_pages_nodemask+0xa81/0xae0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691565]
>>> >> >> >> [<ffffffff811ad2cd>]
>>> >> >> >> >> >> > __read_swap_cache_async+0xdd/0x130
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691567]
>>> >> >> >> [<ffffffff811ad337>]
>>> >> >> >> >> >> > read_swap_cache_async+0x17/0x40
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691569]
>>> >> >> >> [<ffffffff811ad455>]
>>> >> >> >> >> >> > swapin_readahead+0xf5/0x190
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691571]
>>> >> >> >> [<ffffffff8119ce3f>]
>>> >> >> >> >> >> > handle_mm_fault+0xf3f/0x15e0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691574]
>>> >> >> >> [<ffffffff81c9d4e2>] ?
>>> >> >> >> >> >> > __schedule+0x272/0x770
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691576]
>>> >> >> >> [<ffffffff8104e241>]
>>> >> >> >> >> >> > __do_page_fault+0x161/0x370
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691577]
>>> >> >> >> [<ffffffff8104e49c>]
>>> >> >> >> >> >> > do_page_fault+0xc/0x10
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691579]
>>> >> >> >> [<ffffffff81ca3782>]
>>> >> >> >> >> >> > page_fault+0x22/0x30
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691581] Mem-Info:
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>>> >> >> active_anon:3564689
>>> >> >> >> >> >> > inactive_anon:445592 isolated_anon:0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>>> >> active_file:462
>>> >> >> >> >> >> > inactive_file:44 isolated_file:0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>>> unevictable:0
>>> >> >> >> dirty:2
>>> >> >> >> >> >> > writeback:2212 unstable:0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>>> >> >> >> slab_reclaimable:3433
>>> >> >> >> >> >> > slab_unreclaimable:5859
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>>> mapped:989
>>> >> >> >> shmem:2607
>>> >> >> >> >> >> > pagetables:16367 bounce:0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>>> free:51773
>>> >> >> >> >> free_pcp:189
>>> >> >> >> >> >> > free_cma:0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691589] DMA
>>> >> free:15904kB
>>> >> >> >> >> min:128kB
>>> >> >> >> >> >> > low:160kB high:192kB active_anon:0kB inactive_anon:0kB
>>> >> >> >> active_file:0kB
>>> >> >> >> >> >> > inactive_file:0kB unevictable:0kB isolated(anon):0kB
>>> >> >> >> >> isolated(file):0kB
>>> >> >> >> >> >> > present:15992kB managed:15904kB mlocked:0kB dirty:0kB
>>> >> >> writeback:0kB
>>> >> >> >> >> >> > mapped:0kB shmem:0kB slab_reclaimable:0kB
>>> >> slab_unreclaimable:0kB
>>> >> >> >> >> >> > kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB
>>> >> >> >> free_pcp:0kB
>>> >> >> >> >> >> > local_pcp:0kB free_cma:0kB writeback_tmp:0kB
>>> pages_scanned:0
>>> >> >> >> >> >> > all_unreclaimable? yes
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691590]
>>> >> >> lowmem_reserve[]: 0
>>> >> >> >> >> 2960
>>> >> >> >> >> >> > 15991 15991
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691596] DMA32
>>> >> >> free:77080kB
>>> >> >> >> >> >> > min:25000kB low:31248kB high:37500kB
>>> active_anon:2352676kB
>>> >> >> >> >> >> > inactive_anon:590536kB active_file:472kB
>>> inactive_file:140kB
>>> >> >> >> >> >> > unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> >> >> >> >> present:3129216kB
>>> >> >> >> >> >> > managed:3043556kB mlocked:0kB dirty:0kB writeback:4144kB
>>> >> >> >> mapped:1140kB
>>> >> >> >> >> >> > shmem:3568kB slab_reclaimable:1028kB
>>> >> slab_unreclaimable:3004kB
>>> >> >> >> >> >> > kernel_stack:816kB pagetables:10988kB unstable:0kB
>>> bounce:0kB
>>> >> >> >> >> >> > free_pcp:312kB local_pcp:196kB free_cma:0kB
>>> writeback_tmp:0kB
>>> >> >> >> >> >> > pages_scanned:4292 all_unreclaimable? yes
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691597]
>>> >> >> lowmem_reserve[]: 0
>>> >> >> >> 0
>>> >> >> >> >> >> 13031
>>> >> >> >> >> >> > 13031
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691601] Normal
>>> >> >> free:114108kB
>>> >> >> >> >> >> > min:110032kB low:137540kB high:165048kB
>>> >> active_anon:11906080kB
>>> >> >> >> >> >> > inactive_anon:1191832kB active_file:1376kB
>>> inactive_file:36kB
>>> >> >> >> >> >> > unevictable:0kB isolated(anon):0kB isolated(file):0kB
>>> >> >> >> >> present:13631488kB
>>> >> >> >> >> >> > managed:13343784kB mlocked:0kB dirty:8kB writeback:4704kB
>>> >> >> >> >> mapped:2816kB
>>> >> >> >> >> >> > shmem:6860kB slab_reclaimable:12704kB
>>> >> slab_unreclaimable:20432kB
>>> >> >> >> >> >> > kernel_stack:4848kB pagetables:54480kB unstable:0kB
>>> >> bounce:0kB
>>> >> >> >> >> >> > free_pcp:444kB local_pcp:196kB free_cma:0kB
>>> writeback_tmp:0kB
>>> >> >> >> >> >> > pages_scanned:105664 all_unreclaimable? yes
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691602]
>>> >> >> lowmem_reserve[]: 0
>>> >> >> >> 0
>>> >> >> >> >> 0 0
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691603] DMA: 0*4kB
>>> >> 0*8kB
>>> >> >> >> 0*16kB
>>> >> >> >> >> >> > 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB
>>> >> 1*1024kB
>>> >> >> (U)
>>> >> >> >> >> >> 1*2048kB
>>> >> >> >> >> >> > (U) 3*4096kB (M) = 15904kB
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691610] DMA32:
>>> 48*4kB
>>> >> >> (ME)
>>> >> >> >> >> 63*8kB
>>> >> >> >> >> >> > (ME) 62*16kB (E) 46*32kB (UME) 35*64kB (UME) 26*128kB
>>> (UME)
>>> >> >> >> 25*256kB
>>> >> >> >> >> >> (UME)
>>> >> >> >> >> >> > 61*512kB (UME) 28*1024kB (UME) 1*2048kB (M) 0*4096kB =
>>> >> 77080kB
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691616] Normal:
>>> >> 165*4kB
>>> >> >> >> (MEH)
>>> >> >> >> >> >> > 317*8kB (UMEH) 442*16kB (UMEH) 289*32kB (UMEH) 162*64kB
>>> >> (UMEH)
>>> >> >> >> >> 121*128kB
>>> >> >> >> >> >> > (UMEH) 74*256kB (UMEH) 33*512kB (UMEH) 24*1024kB (ME)
>>> >> 0*2048kB
>>> >> >> >> >> 2*4096kB
>>> >> >> >> >> >> (M)
>>> >> >> >> >> >> > = 113980kB
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691623] 5552 total
>>> >> >> pagecache
>>> >> >> >> >> pages
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691624] 2355
>>> pages in
>>> >> >> swap
>>> >> >> >> >> cache
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691625] Swap cache
>>> >> stats:
>>> >> >> >> add
>>> >> >> >> >> >> > 5385308, delete 5382953, find 1159094/1325033
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691626] Free
>>> swap  =
>>> >> 0kB
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691626] Total
>>> swap =
>>> >> >> >> 4194300kB
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691627] 4194174
>>> pages
>>> >> RAM
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691628] 0 pages
>>> >> >> >> >> >> HighMem/MovableOnly
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691628] 93363
>>> pages
>>> >> >> reserved
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691989] Out of
>>> memory:
>>> >> >> Kill
>>> >> >> >> >> >> process
>>> >> >> >> >> >> > 8975 (pgpool) score 7 or sacrifice child
>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691995] Killed
>>> process
>>> >> >> 8975
>>> >> >> >> >> >> (pgpool)
>>> >> >> >> >> >> > total-vm:337504kB, anon-rss:166824kB, file-rss:1920kB
>>> >> >> >> >> >> >
>>> >> >> >> >> >> >
>>> >> >> >> >> >> >
>>> >> >> >> >> >> > On Mon, Sep 27, 2021 at 1:22 AM Tatsuo Ishii <
>>> >> >> ishii at sraoss.co.jp>
>>> >> >> >> >> wrote:
>>> >> >> >> >> >> >
>>> >> >> >> >> >> >> Hi,
>>> >> >> >> >> >> >>
>>> >> >> >> >> >> >> > Hello,
>>> >> >> >> >> >> >> >
>>> >> >> >> >> >> >> > Our pgpool is consuming A LOT of memory and frequently
>>> >> dies
>>> >> >> >> with:
>>> >> >> >> >> Out
>>> >> >> >> >> >> of
>>> >> >> >> >> >> >> > memory error.
>>> >> >> >> >> >> >> >
>>> >> >> >> >> >> >> > We have 2 backends, 1 master and 1 slave. Here some
>>> >> config:
>>> >> >> >> >> >> >> > num_init_children = 150
>>> >> >> >> >> >> >> > max_pool = 1
>>> >> >> >> >> >> >> > child_life_time = 300
>>> >> >> >> >> >> >> > child_max_connections = 1
>>> >> >> >> >> >> >> > connection_life_time = 0
>>> >> >> >> >> >> >> > client_idle_limit = 0
>>> >> >> >> >> >> >> > connection_cache = on
>>> >> >> >> >> >> >> > load_balance_mode = on
>>> >> >> >> >> >> >> > memory_cache_enabled = off
>>> >> >> >> >> >> >> >
>>> >> >> >> >> >> >> > RAM: 16Gb
>>> >> >> >> >> >> >> >
>>> >> >> >> >> >> >> > Does anyone have a clue what's going on?
>>> >> >> >> >> >> >> >
>>> >> >> >> >> >> >> > Thank you.
>>> >> >> >> >> >> >>
>>> >> >> >> >> >> >> Is that OOM killer or pgpool itself aborted with an out
>>> of
>>> >> >> memory
>>> >> >> >> >> >> >> error?  If latter, can you share the pgpool log?
>>> >> >> >> >> >> >> --
>>> >> >> >> >> >> >> Tatsuo Ishii
>>> >> >> >> >> >> >> SRA OSS, Inc. Japan
>>> >> >> >> >> >> >> English: http://www.sraoss.co.jp/index_en.php
>>> >> >> >> >> >> >> Japanese:http://www.sraoss.co.jp
>>> >> >> >> >> >> >>
>>> >> >> >> >> >> >
>>> >> >> >> >> >> >
>>> >> >> >> >> >> > --
>>> >> >> >> >> >> > Luiz Fernando Pasqual S. Souza
>>> >> >> >> >> >> > mail: luiz at pasquall.com
>>> >> >> >> >> >>
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > --
>>> >> >> >> >> > Luiz Fernando Pasqual S. Souza
>>> >> >> >> >> > mail: luiz at pasquall.com
>>> >> >> >> >>
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> > --
>>> >> >> >> > Luiz Fernando Pasqual S. Souza
>>> >> >> >> > mail: luiz at pasquall.com
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Luiz Fernando Pasqual S. Souza
>>> >> >> > mail: luiz at pasquall.com
>>> >> >> _______________________________________________
>>> >> >> pgpool-general mailing list
>>> >> >> pgpool-general at pgpool.net
>>> >> >> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>> >> >>
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Luiz Fernando Pasqual S. Souza
>>> >> > mail: luiz at pasquall.com
>>> >>
>>> >
>>> >
>>> > --
>>> > Luiz Fernando Pasqual S. Souza
>>> > mail: luiz at pasquall.com
>>>
>>
>>
>> --
>> Luiz Fernando Pasqual S. Souza
>> mail: luiz at pasquall.com
>>
> 
> 
> -- 
> Luiz Fernando Pasqual S. Souza
> mail: luiz at pasquall.com


More information about the pgpool-general mailing list