[pgpool-general: 7753] Re: Out of memory

Tatsuo Ishii ishii at sraoss.co.jp
Mon Oct 11 08:18:05 JST 2021


> I'm running a 4.4.276 kernel.
> 
> 1) When psql connected:
> $ ps uax | grep pgpool | grep idle
> root     17167  0.0  0.0  39044  2576 ?        S    11:30   0:00 pgpool:
> XXXX XXXX 192.168.XXX.XXX(34812) idle
> 
> 2) After issue "begin;"
> $ ps uax | grep pgpool | grep idle
> root     17167  0.0  0.2  72200 37044 ?        S    11:30   0:00 pgpool:
> XXXX XXXX 192.168.XXX.XXX(34812) idle in transaction
>
> Then a installed pgpool on another machine running 5.14.3:
> 
> 1) When psql connected:
> $ ps aux | grep pgpool | grep idle
> root     13430  0.0  0.0  13548  3096 ?        S    11:58   0:00 pgpool:
> XXXX XXXX ::1(56142) idle
> 
> 2) After issue "begin;"
> $ ps aux | grep pgpool | grep idle
> root     13430  0.0  0.2  46704 36804 ?        S    11:58   0:00 pgpool:
> XXXX XXXX ::1(56142) idle in transaction
> 
> Attached is my full pgpool.conf, maybe I'm missing something obvious.

Thanks.  I will look into that.

> On Fri, Oct 8, 2021 at 9:55 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
> 
>> Just in case, my Linux box runs kernel 5.11.0-37.
>>
>> > No. I have run similar test myself and did not notice any significant
>> > memory consuming using master HEAD (which will be Pgpool-II 4.3).
>> >
>> > 1) After just psql connected.
>> > t-ishii$ ps aux|grep pgpool|grep idle
>> > t-ishii   432882  0.0  0.0 147192  3404 pts/1    S    09:10   0:00
>> pgpool: t-ishii test  idle
>> >
>> > 2) Issue "begin"
>> > t-ishii   432882  0.0  0.0 147712  5948 pts/1    S    09:10   0:00
>> pgpool: t-ishii test  idle in transaction
>> >
>> > 3) Issue "select * from t1"
>> > t-ishii   432882  0.0  0.0 147712  5948 pts/1    S    09:10   0:00
>> pgpool: t-ishii test  idle in transaction
>> >
>> > As you can see RSS increased after "begin", but it was just from 3404 to
>> 5948.
>> > It was not like 2564 to 37040 in your test.
>> >
>> > Can you do the same test using psql?
>> >
>> >> Any new ideias?
>> >>
>> >> Can we do anything else to track it down?
>> >>
>> >>
>> >>
>> >> On Thu, Sep 30, 2021 at 9:04 AM Luiz Pasqual <luiz at pasquall.com> wrote:
>> >>
>> >>> It just shows:
>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [3-1] 2021-09-30 09:00:41: pid
>> 2537:
>> >>> LOG:  Query message from frontend.
>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [3-2] 2021-09-30 09:00:41: pid
>> 2537:
>> >>> DETAIL:  query: "BEGIN"
>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [4-1] 2021-09-30 09:00:41: pid
>> 2537:
>> >>> LOG:  Query message from frontend.
>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [4-2] 2021-09-30 09:00:41: pid
>> 2537:
>> >>> DETAIL:  query: "SELECT * from uf"
>> >>>
>> >>>
>> >>> On Wed, Sep 29, 2021 at 6:38 PM Tatsuo Ishii <ishii at sraoss.co.jp>
>> wrote:
>> >>>
>> >>>> I am not familiar with python. It would be helpful if you can share
>> >>>> the pgpool log with log_client_messages enabled while running the
>> >>>> python session. This will show what messages the python client sends
>> >>>> to pgpool.
>> >>>>
>> >>>> > There is nothing special, it's a python application using psycopg to
>> >>>> > connect.
>> >>>> >
>> >>>> > I've made a very simple test here, using:
>> >>>> > Python 2.7.17
>> >>>> > Psycopg 2.8.6
>> >>>> > We have another setup running with Python 3, it seems to have the
>> same
>> >>>> > result.
>> >>>> >
>> >>>> > $ bin/python
>> >>>> > Python 2.7.17 (default, Oct 20 2019, 14:16:51)
>> >>>> > [GCC 5.5.0] on linux2
>> >>>> > Type "help", "copyright", "credits" or "license" for more
>> information.
>> >>>> >>>> import psycopg2
>> >>>> >>>> conn = psycopg2.connect("dbname='XXXX' user='XXXX'
>> >>>> > host='192.168.XXX.XXX' password='XXXXXX' port='9999'")
>> >>>> >>>> cur = conn.cursor()
>> >>>> >>>> cur.execute("""SELECT * from XX""")
>> >>>> >
>> >>>> > This is a very simple select, it returns 27 results with 2 columns,
>> an
>> >>>> ID e
>> >>>> > a string (two characters in each row)
>> >>>> >
>> >>>> > pgpool 3.7.3:
>> >>>> > After "psycopg.connect":
>> >>>> > root      3230  0.0  0.0  36200  2544 ?        S    10:16   0:00
>> pgpool:
>> >>>> > XXXX XXXX 192.168.XXX.XXX(59576) idle
>> >>>> >
>> >>>> > After "cur.execute":
>> >>>> > root      3230  0.0  0.0  41400  9272 ?        S    10:16   0:00
>> pgpool:
>> >>>> > XXXX XXXX 192.168.XXX.XXX(59576) idle in transaction
>> >>>> >
>> >>>> > pgpool 4.1.8:
>> >>>> > After "psycopg.connect":
>> >>>> > root      5317  0.0  0.0  39044  2564 ?        S    11:35   0:00
>> pgpool:
>> >>>> > zope nube 192.168.XXX.XXX(54784) idle
>> >>>> >
>> >>>> > After "cur.execute":
>> >>>> > root      5317  0.0  0.2  72200 37040 ?        S    11:35   0:00
>> pgpool:
>> >>>> > zope nube 192.168.XXX.XXX(54784) idle in transaction
>> >>>> >
>> >>>> > Do you think this is enough to reproduce? Or do you need something
>> >>>> > different? Let me know, I'm here to help.
>> >>>> >
>> >>>> >
>> >>>> > On Wed, Sep 29, 2021 at 8:54 AM Tatsuo Ishii <ishii at sraoss.co.jp>
>> >>>> wrote:
>> >>>> >
>> >>>> >> So enable_shared_relcache is irrelevant.
>> >>>> >>
>> >>>> >> Problem is, I cannot reproduce your problem here. It would be
>> greately
>> >>>> >> helpful if you can provide a self containing test case. If you
>> cannot,
>> >>>> >> I would like to find a pattern (query) which triggers the problem.
>> >>>> >>
>> >>>> >> - Does the query use extended protocol?
>> >>>> >>
>> >>>> >> - What kind of queries does the application use?
>> >>>> >>   INSERT/UPDATE/SELECT/DELETE??
>> >>>> >>
>> >>>> >> - IS there anything special with the application?
>> >>>> >>
>> >>>> >> > I tested 4.1.0 now and it shows the same behaviour.
>> >>>> >> >
>> >>>> >> > Also, disabling enable_shared_relcache did not solve the problem.
>> >>>> >> >
>> >>>> >> > On Tue, Sep 28, 2021 at 9:10 PM Tatsuo Ishii <ishii at sraoss.co.jp
>> >
>> >>>> wrote:
>> >>>> >> >
>> >>>> >> >> Ok. Let's assume 4.1.0 introduced the problem for now (it's
>> still
>> >>>> >> >> possible that certain minor version of 4.1 introduced the
>> issue, but
>> >>>> >> >> for now I neglect the possibility).
>> >>>> >> >>
>> >>>> >> >> 4.1.0 includes following major changes especially related to
>> memory.
>> >>>> >> >>
>> >>>> >> >> (1) Shared relation cache allows to reuse relation cache among
>> >>>> >> >>     sessions to reduce internal queries against PostgreSQL
>> system
>> >>>> >> >>     catalogs.
>> >>>> >> >>
>> >>>> >> >>     (this feature uses shared memory even if
>> memory_cache_enabled =
>> >>>> >> off).
>> >>>> >> >>
>> >>>> >> >> (2) Have separate SQL parser for DML statements to eliminate
>> >>>> >> >>     unnecessary parsing effort.
>> >>>> >> >>
>> >>>> >> >>     (this allocates additional process memory for certain query)
>> >>>> >> >>
>> >>>> >> >> For (1) we can disable the feature by:
>> >>>> >> >>
>> >>>> >> >> enable_shared_relcache = off
>> >>>> >> >>
>> >>>> >> >> Can you try it out?
>> >>>> >> >>
>> >>>> >> >> > I did those tests:
>> >>>> >> >> > 4.0.15 is running fine, when I upgraded to 4.1.8 it started to
>> >>>> >> consume a
>> >>>> >> >> > lot of memory.
>> >>>> >> >> >
>> >>>> >> >> > On Tue, Sep 28, 2021 at 8:30 AM Tatsuo Ishii <
>> ishii at sraoss.co.jp>
>> >>>> >> wrote:
>> >>>> >> >> >
>> >>>> >> >> >> Ok. 4.2.1 consumes 10 times of memory than 3.7.3 (maybe 5,
>> not 10
>> >>>> >> >> >> because in your test connections of 4.2.1 is twice of 3.7.3).
>> >>>> >> >> >>
>> >>>> >> >> >> Next thing I need to know is, from which version of pgpool
>> >>>> started to
>> >>>> >> >> >> consume lot of memory.  Is it possible to run the same test
>> >>>> against
>> >>>> >> >> >> 4.0 and 4.1? Ideally from which version, including minor
>> version
>> >>>> will
>> >>>> >> >> >> be a lot helpful. I wish I could the test myself but I don't
>> >>>> know to
>> >>>> >> >> >> do it on my environment.
>> >>>> >> >> >>
>> >>>> >> >> >> > Sure.
>> >>>> >> >> >> >
>> >>>> >> >> >> > On 3.7.3:
>> >>>> >> >> >> > $ ps aux | grep pgpool | awk '{sum += $6}END{print sum}'
>> >>>> >> >> >> > 896752
>> >>>> >> >> >> >
>> >>>> >> >> >> > On 4.2.1:
>> >>>> >> >> >> > $ ps aux | grep pgpool | awk '{sum += $6}END{print sum}'
>> >>>> >> >> >> > 9969280
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> > On Mon, Sep 27, 2021 at 11:41 PM Tatsuo Ishii <
>> >>>> ishii at sraoss.co.jp>
>> >>>> >> >> >> wrote:
>> >>>> >> >> >> >
>> >>>> >> >> >> >> Ok, let's see how much RSS pgpool consume in total. Can
>> you
>> >>>> share
>> >>>> >> the
>> >>>> >> >> >> >> result of the command for both pgpool 4.2 and 3.7?
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> ps aux | grep pgpool | awk '{sum += $6}END{print sum}'
>> >>>> >> >> >> >>
>> >>>> >> >> >> >> > I see.
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > The problem is, the OOM happens when all RAM is already
>> >>>> >> >> compromised by
>> >>>> >> >> >> >> > pgpool (all 16Gb of RAM is dedicated to pgpool), so any
>> >>>> common
>> >>>> >> >> request
>> >>>> >> >> >> >> will
>> >>>> >> >> >> >> > die anyway.
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > It happens on versions: 4.2.5 and 4.2.1. Now, I'm
>> running
>> >>>> the
>> >>>> >> same
>> >>>> >> >> >> >> > application on 3.7.3 and it's running great.
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > I don't know if it helps, but just to demonstrate the
>> >>>> amount of
>> >>>> >> >> RAM is
>> >>>> >> >> >> >> > necessary to run a few connections I created another
>> VM, and
>> >>>> >> >> >> comparing:
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > 62 active connections running on 4.2.1 (same behaviour
>> on
>> >>>> >> 4.2.5):
>> >>>> >> >> >> >> > $ vmstat -a -SM
>> >>>> >> >> >> >> > procs -----------memory---------- ---swap-- -----io----
>> >>>> >> -system--
>> >>>> >> >> >> >> > ------cpu-----
>> >>>> >> >> >> >> >  r  b   swpd   free  inact active   si   so    bi
>> bo   in
>> >>>> >>  cs
>> >>>> >> >> us
>> >>>> >> >> >> sy
>> >>>> >> >> >> >> id
>> >>>> >> >> >> >> > wa st
>> >>>> >> >> >> >> >  0  0      0   4829    173  10772    0    0     2
>>  1   68
>> >>>> >>  56
>> >>>> >> >> 1
>> >>>> >> >> >> 1
>> >>>> >> >> >> >> 99
>> >>>> >> >> >> >> >  0  0
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > 31 active connections running on 3.7.3:
>> >>>> >> >> >> >> > $ vmstat -a -SM
>> >>>> >> >> >> >> > procs -----------memory---------- ---swap-- -----io----
>> >>>> >> -system--
>> >>>> >> >> >> >> > ------cpu-----
>> >>>> >> >> >> >> >  r  b   swpd   free  inact active   si   so    bi
>> bo   in
>> >>>> >>  cs
>> >>>> >> >> us
>> >>>> >> >> >> sy
>> >>>> >> >> >> >> id
>> >>>> >> >> >> >> > wa st
>> >>>> >> >> >> >> >  0  0      0  15174     73    635    0    0     1
>>  1   76
>> >>>> >> 3
>> >>>> >> >> 0
>> >>>> >> >> >> 0
>> >>>> >> >> >> >> 100
>> >>>> >> >> >> >> >  0  0
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > Both are running basically the same application.
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > Seeing the RAM consumption of 3.7.3, it does not seem
>> to be
>> >>>> >> >> receiving
>> >>>> >> >> >> >> such
>> >>>> >> >> >> >> > a big requests. Is there a way to track it down?
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > On Mon, Sep 27, 2021 at 9:07 AM Tatsuo Ishii <
>> >>>> >> ishii at sraoss.co.jp>
>> >>>> >> >> >> wrote:
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> >> At this point I cannot judge whether the problem is
>> caused
>> >>>> by a
>> >>>> >> >> >> pgpool
>> >>>> >> >> >> >> >> bug or client's resource request is too much.
>> >>>> >> >> >> >> >>
>> >>>> >> >> >> >> >> Typical bug which requests too much memory allocation
>> is
>> >>>> not
>> >>>> >> >> >> something
>> >>>> >> >> >> >> >> like this because 33947648 = 32MB memory request
>> itself is
>> >>>> not
>> >>>> >> >> >> insane.
>> >>>> >> >> >> >> >>
>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-2]
>> 2021-09-24
>> >>>> >> >> 09:14:10:
>> >>>> >> >> >> pid
>> >>>> >> >> >> >> >> > 12650: DETAIL:  Failed on request of size 33947648.
>> >>>> >> >> >> >> >>
>> >>>> >> >> >> >> >> (Please let us know what version of Pgpool-II you are
>> using
>> >>>> >> >> because
>> >>>> >> >> >> >> >> it's important information to identify any known bug).
>> >>>> >> >> >> >> >>
>> >>>> >> >> >> >> >> In the mean time, however, I think 32MB memory request
>> is
>> >>>> not
>> >>>> >> very
>> >>>> >> >> >> >> >> common in pgpool. One thing I wonder is, whether your
>> >>>> >> application
>> >>>> >> >> >> >> >> issues SQL which requires large memory: e.g. very long
>> SQL
>> >>>> >> >> statement,
>> >>>> >> >> >> >> >> COPY for large data set. They will request large
>> read/write
>> >>>> >> >> buffer in
>> >>>> >> >> >> >> >> pgpool.
>> >>>> >> >> >> >> >>
>> >>>> >> >> >> >> >> > We saw both, but pgpool aborting is way more common:
>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-1]
>> 2021-09-24
>> >>>> >> >> 09:14:10:
>> >>>> >> >> >> pid
>> >>>> >> >> >> >> >> > 12650: ERROR:  out of memory
>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-2]
>> 2021-09-24
>> >>>> >> >> 09:14:10:
>> >>>> >> >> >> pid
>> >>>> >> >> >> >> >> > 12650: DETAIL:  Failed on request of size 33947648.
>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-3]
>> 2021-09-24
>> >>>> >> >> 09:14:10:
>> >>>> >> >> >> pid
>> >>>> >> >> >> >> >> > 12650: LOCATION:  mcxt.c:900
>> >>>> >> >> >> >> >> >
>> >>>> >> >> >> >> >> > Here, two other ways we saw at the logs, but those
>> just
>> >>>> >> occurred
>> >>>> >> >> >> once
>> >>>> >> >> >> >> >> each:
>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-1]
>> 2021-09-24
>> >>>> >> >> 07:33:14:
>> >>>> >> >> >> pid
>> >>>> >> >> >> >> >> 5874:
>> >>>> >> >> >> >> >> > FATAL:  failed to fork a child
>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-2]
>> 2021-09-24
>> >>>> >> >> 07:33:14:
>> >>>> >> >> >> pid
>> >>>> >> >> >> >> >> 5874:
>> >>>> >> >> >> >> >> > DETAIL:  system call fork() failed with reason:
>> Cannot
>> >>>> >> allocate
>> >>>> >> >> >> memory
>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-3]
>> 2021-09-24
>> >>>> >> >> 07:33:14:
>> >>>> >> >> >> pid
>> >>>> >> >> >> >> >> 5874:
>> >>>> >> >> >> >> >> > LOCATION:  pgpool_main.c:681
>> >>>> >> >> >> >> >> >
>> >>>> >> >> >> >> >> > And:
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691518] pgpool
>> >>>> invoked
>> >>>> >> >> >> >> oom-killer:
>> >>>> >> >> >> >> >> > gfp_mask=0x24200ca, order=0, oom_score_adj=0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691525] CPU: 1
>> >>>> PID:
>> >>>> >> 1194
>> >>>> >> >> >> Comm:
>> >>>> >> >> >> >> >> > pgpool Not tainted 4.4.276 #1
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691527]
>> Hardware
>> >>>> name:
>> >>>> >> >> >> VMware,
>> >>>> >> >> >> >> >> Inc.
>> >>>> >> >> >> >> >> > VMware Virtual Platform/440BX Desktop Reference
>> Platform,
>> >>>> >> BIOS
>> >>>> >> >> 6.00
>> >>>> >> >> >> >> >> > 12/12/2018
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691528]
>> >>>> >> 0000000000000000
>> >>>> >> >> >> >> >> > ffff8803281cbae8 ffffffff81c930e7 ffff880420efe2c0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691530]
>> >>>> >> ffff880420efe2c0
>> >>>> >> >> >> >> >> > ffff8803281cbb50 ffffffff81c8d9da ffff8803281cbb08
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691532]
>> >>>> >> ffffffff81133f1a
>> >>>> >> >> >> >> >> > ffff8803281cbb80 ffffffff81182eb0 ffff8800bba393c0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691533] Call
>> >>>> Trace:
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691540]
>> >>>> >> >> >> [<ffffffff81c930e7>]
>> >>>> >> >> >> >> >> > dump_stack+0x57/0x6d
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691542]
>> >>>> >> >> >> [<ffffffff81c8d9da>]
>> >>>> >> >> >> >> >> > dump_header.isra.9+0x54/0x1ae
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691547]
>> >>>> >> >> >> [<ffffffff81133f1a>] ?
>> >>>> >> >> >> >> >> > __delayacct_freepages_end+0x2a/0x30
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691553]
>> >>>> >> >> >> [<ffffffff81182eb0>] ?
>> >>>> >> >> >> >> >> > do_try_to_free_pages+0x350/0x3d0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691556]
>> >>>> >> >> >> [<ffffffff811709f9>]
>> >>>> >> >> >> >> >> > oom_kill_process+0x209/0x3c0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691558]
>> >>>> >> >> >> [<ffffffff81170eeb>]
>> >>>> >> >> >> >> >> > out_of_memory+0x2db/0x2f0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691561]
>> >>>> >> >> >> [<ffffffff81176111>]
>> >>>> >> >> >> >> >> > __alloc_pages_nodemask+0xa81/0xae0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691565]
>> >>>> >> >> >> [<ffffffff811ad2cd>]
>> >>>> >> >> >> >> >> > __read_swap_cache_async+0xdd/0x130
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691567]
>> >>>> >> >> >> [<ffffffff811ad337>]
>> >>>> >> >> >> >> >> > read_swap_cache_async+0x17/0x40
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691569]
>> >>>> >> >> >> [<ffffffff811ad455>]
>> >>>> >> >> >> >> >> > swapin_readahead+0xf5/0x190
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691571]
>> >>>> >> >> >> [<ffffffff8119ce3f>]
>> >>>> >> >> >> >> >> > handle_mm_fault+0xf3f/0x15e0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691574]
>> >>>> >> >> >> [<ffffffff81c9d4e2>] ?
>> >>>> >> >> >> >> >> > __schedule+0x272/0x770
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691576]
>> >>>> >> >> >> [<ffffffff8104e241>]
>> >>>> >> >> >> >> >> > __do_page_fault+0x161/0x370
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691577]
>> >>>> >> >> >> [<ffffffff8104e49c>]
>> >>>> >> >> >> >> >> > do_page_fault+0xc/0x10
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691579]
>> >>>> >> >> >> [<ffffffff81ca3782>]
>> >>>> >> >> >> >> >> > page_fault+0x22/0x30
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691581]
>> Mem-Info:
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>> >>>> >> >> active_anon:3564689
>> >>>> >> >> >> >> >> > inactive_anon:445592 isolated_anon:0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>> >>>> >> active_file:462
>> >>>> >> >> >> >> >> > inactive_file:44 isolated_file:0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>> >>>> unevictable:0
>> >>>> >> >> >> dirty:2
>> >>>> >> >> >> >> >> > writeback:2212 unstable:0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>> >>>> >> >> >> slab_reclaimable:3433
>> >>>> >> >> >> >> >> > slab_unreclaimable:5859
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>> >>>> mapped:989
>> >>>> >> >> >> shmem:2607
>> >>>> >> >> >> >> >> > pagetables:16367 bounce:0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
>> >>>> free:51773
>> >>>> >> >> >> >> free_pcp:189
>> >>>> >> >> >> >> >> > free_cma:0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691589] DMA
>> >>>> >> free:15904kB
>> >>>> >> >> >> >> min:128kB
>> >>>> >> >> >> >> >> > low:160kB high:192kB active_anon:0kB
>> inactive_anon:0kB
>> >>>> >> >> >> active_file:0kB
>> >>>> >> >> >> >> >> > inactive_file:0kB unevictable:0kB isolated(anon):0kB
>> >>>> >> >> >> >> isolated(file):0kB
>> >>>> >> >> >> >> >> > present:15992kB managed:15904kB mlocked:0kB dirty:0kB
>> >>>> >> >> writeback:0kB
>> >>>> >> >> >> >> >> > mapped:0kB shmem:0kB slab_reclaimable:0kB
>> >>>> >> slab_unreclaimable:0kB
>> >>>> >> >> >> >> >> > kernel_stack:0kB pagetables:0kB unstable:0kB
>> bounce:0kB
>> >>>> >> >> >> free_pcp:0kB
>> >>>> >> >> >> >> >> > local_pcp:0kB free_cma:0kB writeback_tmp:0kB
>> >>>> pages_scanned:0
>> >>>> >> >> >> >> >> > all_unreclaimable? yes
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691590]
>> >>>> >> >> lowmem_reserve[]: 0
>> >>>> >> >> >> >> 2960
>> >>>> >> >> >> >> >> > 15991 15991
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691596] DMA32
>> >>>> >> >> free:77080kB
>> >>>> >> >> >> >> >> > min:25000kB low:31248kB high:37500kB
>> >>>> active_anon:2352676kB
>> >>>> >> >> >> >> >> > inactive_anon:590536kB active_file:472kB
>> >>>> inactive_file:140kB
>> >>>> >> >> >> >> >> > unevictable:0kB isolated(anon):0kB isolated(file):0kB
>> >>>> >> >> >> >> present:3129216kB
>> >>>> >> >> >> >> >> > managed:3043556kB mlocked:0kB dirty:0kB
>> writeback:4144kB
>> >>>> >> >> >> mapped:1140kB
>> >>>> >> >> >> >> >> > shmem:3568kB slab_reclaimable:1028kB
>> >>>> >> slab_unreclaimable:3004kB
>> >>>> >> >> >> >> >> > kernel_stack:816kB pagetables:10988kB unstable:0kB
>> >>>> bounce:0kB
>> >>>> >> >> >> >> >> > free_pcp:312kB local_pcp:196kB free_cma:0kB
>> >>>> writeback_tmp:0kB
>> >>>> >> >> >> >> >> > pages_scanned:4292 all_unreclaimable? yes
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691597]
>> >>>> >> >> lowmem_reserve[]: 0
>> >>>> >> >> >> 0
>> >>>> >> >> >> >> >> 13031
>> >>>> >> >> >> >> >> > 13031
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691601] Normal
>> >>>> >> >> free:114108kB
>> >>>> >> >> >> >> >> > min:110032kB low:137540kB high:165048kB
>> >>>> >> active_anon:11906080kB
>> >>>> >> >> >> >> >> > inactive_anon:1191832kB active_file:1376kB
>> >>>> inactive_file:36kB
>> >>>> >> >> >> >> >> > unevictable:0kB isolated(anon):0kB isolated(file):0kB
>> >>>> >> >> >> >> present:13631488kB
>> >>>> >> >> >> >> >> > managed:13343784kB mlocked:0kB dirty:8kB
>> writeback:4704kB
>> >>>> >> >> >> >> mapped:2816kB
>> >>>> >> >> >> >> >> > shmem:6860kB slab_reclaimable:12704kB
>> >>>> >> slab_unreclaimable:20432kB
>> >>>> >> >> >> >> >> > kernel_stack:4848kB pagetables:54480kB unstable:0kB
>> >>>> >> bounce:0kB
>> >>>> >> >> >> >> >> > free_pcp:444kB local_pcp:196kB free_cma:0kB
>> >>>> writeback_tmp:0kB
>> >>>> >> >> >> >> >> > pages_scanned:105664 all_unreclaimable? yes
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691602]
>> >>>> >> >> lowmem_reserve[]: 0
>> >>>> >> >> >> 0
>> >>>> >> >> >> >> 0 0
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691603] DMA:
>> 0*4kB
>> >>>> >> 0*8kB
>> >>>> >> >> >> 0*16kB
>> >>>> >> >> >> >> >> > 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB
>> >>>> >> 1*1024kB
>> >>>> >> >> (U)
>> >>>> >> >> >> >> >> 1*2048kB
>> >>>> >> >> >> >> >> > (U) 3*4096kB (M) = 15904kB
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691610] DMA32:
>> >>>> 48*4kB
>> >>>> >> >> (ME)
>> >>>> >> >> >> >> 63*8kB
>> >>>> >> >> >> >> >> > (ME) 62*16kB (E) 46*32kB (UME) 35*64kB (UME) 26*128kB
>> >>>> (UME)
>> >>>> >> >> >> 25*256kB
>> >>>> >> >> >> >> >> (UME)
>> >>>> >> >> >> >> >> > 61*512kB (UME) 28*1024kB (UME) 1*2048kB (M) 0*4096kB
>> =
>> >>>> >> 77080kB
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691616]
>> Normal:
>> >>>> >> 165*4kB
>> >>>> >> >> >> (MEH)
>> >>>> >> >> >> >> >> > 317*8kB (UMEH) 442*16kB (UMEH) 289*32kB (UMEH)
>> 162*64kB
>> >>>> >> (UMEH)
>> >>>> >> >> >> >> 121*128kB
>> >>>> >> >> >> >> >> > (UMEH) 74*256kB (UMEH) 33*512kB (UMEH) 24*1024kB (ME)
>> >>>> >> 0*2048kB
>> >>>> >> >> >> >> 2*4096kB
>> >>>> >> >> >> >> >> (M)
>> >>>> >> >> >> >> >> > = 113980kB
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691623] 5552
>> total
>> >>>> >> >> pagecache
>> >>>> >> >> >> >> pages
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691624] 2355
>> >>>> pages in
>> >>>> >> >> swap
>> >>>> >> >> >> >> cache
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691625] Swap
>> cache
>> >>>> >> stats:
>> >>>> >> >> >> add
>> >>>> >> >> >> >> >> > 5385308, delete 5382953, find 1159094/1325033
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691626] Free
>> >>>> swap  =
>> >>>> >> 0kB
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691626] Total
>> >>>> swap =
>> >>>> >> >> >> 4194300kB
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691627]
>> 4194174
>> >>>> pages
>> >>>> >> RAM
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691628] 0
>> pages
>> >>>> >> >> >> >> >> HighMem/MovableOnly
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691628] 93363
>> >>>> pages
>> >>>> >> >> reserved
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691989] Out of
>> >>>> memory:
>> >>>> >> >> Kill
>> >>>> >> >> >> >> >> process
>> >>>> >> >> >> >> >> > 8975 (pgpool) score 7 or sacrifice child
>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691995] Killed
>> >>>> process
>> >>>> >> >> 8975
>> >>>> >> >> >> >> >> (pgpool)
>> >>>> >> >> >> >> >> > total-vm:337504kB, anon-rss:166824kB, file-rss:1920kB
>> >>>> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >
>> >>>> >> >> >> >> >> > On Mon, Sep 27, 2021 at 1:22 AM Tatsuo Ishii <
>> >>>> >> >> ishii at sraoss.co.jp>
>> >>>> >> >> >> >> wrote:
>> >>>> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >> Hi,
>> >>>> >> >> >> >> >> >>
>> >>>> >> >> >> >> >> >> > Hello,
>> >>>> >> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >> > Our pgpool is consuming A LOT of memory and
>> frequently
>> >>>> >> dies
>> >>>> >> >> >> with:
>> >>>> >> >> >> >> Out
>> >>>> >> >> >> >> >> of
>> >>>> >> >> >> >> >> >> > memory error.
>> >>>> >> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >> > We have 2 backends, 1 master and 1 slave. Here
>> some
>> >>>> >> config:
>> >>>> >> >> >> >> >> >> > num_init_children = 150
>> >>>> >> >> >> >> >> >> > max_pool = 1
>> >>>> >> >> >> >> >> >> > child_life_time = 300
>> >>>> >> >> >> >> >> >> > child_max_connections = 1
>> >>>> >> >> >> >> >> >> > connection_life_time = 0
>> >>>> >> >> >> >> >> >> > client_idle_limit = 0
>> >>>> >> >> >> >> >> >> > connection_cache = on
>> >>>> >> >> >> >> >> >> > load_balance_mode = on
>> >>>> >> >> >> >> >> >> > memory_cache_enabled = off
>> >>>> >> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >> > RAM: 16Gb
>> >>>> >> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >> > Does anyone have a clue what's going on?
>> >>>> >> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >> > Thank you.
>> >>>> >> >> >> >> >> >>
>> >>>> >> >> >> >> >> >> Is that OOM killer or pgpool itself aborted with an
>> out
>> >>>> of
>> >>>> >> >> memory
>> >>>> >> >> >> >> >> >> error?  If latter, can you share the pgpool log?
>> >>>> >> >> >> >> >> >> --
>> >>>> >> >> >> >> >> >> Tatsuo Ishii
>> >>>> >> >> >> >> >> >> SRA OSS, Inc. Japan
>> >>>> >> >> >> >> >> >> English: http://www.sraoss.co.jp/index_en.php
>> >>>> >> >> >> >> >> >> Japanese:http://www.sraoss.co.jp
>> >>>> >> >> >> >> >> >>
>> >>>> >> >> >> >> >> >
>> >>>> >> >> >> >> >> >
>> >>>> >> >> >> >> >> > --
>> >>>> >> >> >> >> >> > Luiz Fernando Pasqual S. Souza
>> >>>> >> >> >> >> >> > mail: luiz at pasquall.com
>> >>>> >> >> >> >> >>
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> >
>> >>>> >> >> >> >> > --
>> >>>> >> >> >> >> > Luiz Fernando Pasqual S. Souza
>> >>>> >> >> >> >> > mail: luiz at pasquall.com
>> >>>> >> >> >> >>
>> >>>> >> >> >> >
>> >>>> >> >> >> >
>> >>>> >> >> >> > --
>> >>>> >> >> >> > Luiz Fernando Pasqual S. Souza
>> >>>> >> >> >> > mail: luiz at pasquall.com
>> >>>> >> >> >>
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > --
>> >>>> >> >> > Luiz Fernando Pasqual S. Souza
>> >>>> >> >> > mail: luiz at pasquall.com
>> >>>> >> >> _______________________________________________
>> >>>> >> >> pgpool-general mailing list
>> >>>> >> >> pgpool-general at pgpool.net
>> >>>> >> >> http://www.pgpool.net/mailman/listinfo/pgpool-general
>> >>>> >> >>
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > --
>> >>>> >> > Luiz Fernando Pasqual S. Souza
>> >>>> >> > mail: luiz at pasquall.com
>> >>>> >>
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > Luiz Fernando Pasqual S. Souza
>> >>>> > mail: luiz at pasquall.com
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> Luiz Fernando Pasqual S. Souza
>> >>> mail: luiz at pasquall.com
>> >>>
>> >>
>> >>
>> >> --
>> >> Luiz Fernando Pasqual S. Souza
>> >> mail: luiz at pasquall.com
>> > _______________________________________________
>> > pgpool-general mailing list
>> > pgpool-general at pgpool.net
>> > http://www.pgpool.net/mailman/listinfo/pgpool-general
>>
> 
> 
> -- 
> Luiz Fernando Pasqual S. Souza
> mail: luiz at pasquall.com


More information about the pgpool-general mailing list