[pgpool-general: 7759] Re: Out of memory
Luiz Pasqual
luiz at pasquall.com
Mon Oct 11 21:35:44 JST 2021
You got it!
It was my fault, I missed this entry when comparing it with 3.7.3, sorry
about that.
On Sun, Oct 10, 2021 at 9:51 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
> In your pgpool.conf, I noticed this:
>
> relcache_size = 16384
>
> Since the default is 256, your setting will consume 64 times more
> memory for *each* pgpol child process. Since num_init_children is 150,
> which is 4 times larger than the default, your memory consumption for
> the relcache is 64*4 = 256 times larger than the default. I guess this
> is at least one of the reason which causes the memory issue.
>
> If all 150 relcache entries are filled, each process will consume 48MB
> of memory, which is 48 * 150 /1024 = 7GB, in total.
>
> With shared relacache enabled, each pgpool child can share the
> existing relacache info. So you don't need to assign that 16384 to
> relcache_size. Can you try with the default value of relcache_size (256)?
>
> >> I'm running a 4.4.276 kernel.
> >>
> >> 1) When psql connected:
> >> $ ps uax | grep pgpool | grep idle
> >> root 17167 0.0 0.0 39044 2576 ? S 11:30 0:00 pgpool:
> >> XXXX XXXX 192.168.XXX.XXX(34812) idle
> >>
> >> 2) After issue "begin;"
> >> $ ps uax | grep pgpool | grep idle
> >> root 17167 0.0 0.2 72200 37044 ? S 11:30 0:00 pgpool:
> >> XXXX XXXX 192.168.XXX.XXX(34812) idle in transaction
> >>
> >> Then a installed pgpool on another machine running 5.14.3:
> >>
> >> 1) When psql connected:
> >> $ ps aux | grep pgpool | grep idle
> >> root 13430 0.0 0.0 13548 3096 ? S 11:58 0:00 pgpool:
> >> XXXX XXXX ::1(56142) idle
> >>
> >> 2) After issue "begin;"
> >> $ ps aux | grep pgpool | grep idle
> >> root 13430 0.0 0.2 46704 36804 ? S 11:58 0:00 pgpool:
> >> XXXX XXXX ::1(56142) idle in transaction
> >>
> >> Attached is my full pgpool.conf, maybe I'm missing something obvious.
> >
> > Thanks. I will look into that.
> >
> >> On Fri, Oct 8, 2021 at 9:55 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
> >>
> >>> Just in case, my Linux box runs kernel 5.11.0-37.
> >>>
> >>> > No. I have run similar test myself and did not notice any significant
> >>> > memory consuming using master HEAD (which will be Pgpool-II 4.3).
> >>> >
> >>> > 1) After just psql connected.
> >>> > t-ishii$ ps aux|grep pgpool|grep idle
> >>> > t-ishii 432882 0.0 0.0 147192 3404 pts/1 S 09:10 0:00
> >>> pgpool: t-ishii test idle
> >>> >
> >>> > 2) Issue "begin"
> >>> > t-ishii 432882 0.0 0.0 147712 5948 pts/1 S 09:10 0:00
> >>> pgpool: t-ishii test idle in transaction
> >>> >
> >>> > 3) Issue "select * from t1"
> >>> > t-ishii 432882 0.0 0.0 147712 5948 pts/1 S 09:10 0:00
> >>> pgpool: t-ishii test idle in transaction
> >>> >
> >>> > As you can see RSS increased after "begin", but it was just from
> 3404 to
> >>> 5948.
> >>> > It was not like 2564 to 37040 in your test.
> >>> >
> >>> > Can you do the same test using psql?
> >>> >
> >>> >> Any new ideias?
> >>> >>
> >>> >> Can we do anything else to track it down?
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Thu, Sep 30, 2021 at 9:04 AM Luiz Pasqual <luiz at pasquall.com>
> wrote:
> >>> >>
> >>> >>> It just shows:
> >>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [3-1] 2021-09-30 09:00:41: pid
> >>> 2537:
> >>> >>> LOG: Query message from frontend.
> >>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [3-2] 2021-09-30 09:00:41: pid
> >>> 2537:
> >>> >>> DETAIL: query: "BEGIN"
> >>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [4-1] 2021-09-30 09:00:41: pid
> >>> 2537:
> >>> >>> LOG: Query message from frontend.
> >>> >>> Sep 30 09:00:41 pgpool pgpool[2537]: [4-2] 2021-09-30 09:00:41: pid
> >>> 2537:
> >>> >>> DETAIL: query: "SELECT * from uf"
> >>> >>>
> >>> >>>
> >>> >>> On Wed, Sep 29, 2021 at 6:38 PM Tatsuo Ishii <ishii at sraoss.co.jp>
> >>> wrote:
> >>> >>>
> >>> >>>> I am not familiar with python. It would be helpful if you can
> share
> >>> >>>> the pgpool log with log_client_messages enabled while running the
> >>> >>>> python session. This will show what messages the python client
> sends
> >>> >>>> to pgpool.
> >>> >>>>
> >>> >>>> > There is nothing special, it's a python application using
> psycopg to
> >>> >>>> > connect.
> >>> >>>> >
> >>> >>>> > I've made a very simple test here, using:
> >>> >>>> > Python 2.7.17
> >>> >>>> > Psycopg 2.8.6
> >>> >>>> > We have another setup running with Python 3, it seems to have
> the
> >>> same
> >>> >>>> > result.
> >>> >>>> >
> >>> >>>> > $ bin/python
> >>> >>>> > Python 2.7.17 (default, Oct 20 2019, 14:16:51)
> >>> >>>> > [GCC 5.5.0] on linux2
> >>> >>>> > Type "help", "copyright", "credits" or "license" for more
> >>> information.
> >>> >>>> >>>> import psycopg2
> >>> >>>> >>>> conn = psycopg2.connect("dbname='XXXX' user='XXXX'
> >>> >>>> > host='192.168.XXX.XXX' password='XXXXXX' port='9999'")
> >>> >>>> >>>> cur = conn.cursor()
> >>> >>>> >>>> cur.execute("""SELECT * from XX""")
> >>> >>>> >
> >>> >>>> > This is a very simple select, it returns 27 results with 2
> columns,
> >>> an
> >>> >>>> ID e
> >>> >>>> > a string (two characters in each row)
> >>> >>>> >
> >>> >>>> > pgpool 3.7.3:
> >>> >>>> > After "psycopg.connect":
> >>> >>>> > root 3230 0.0 0.0 36200 2544 ? S 10:16 0:00
> >>> pgpool:
> >>> >>>> > XXXX XXXX 192.168.XXX.XXX(59576) idle
> >>> >>>> >
> >>> >>>> > After "cur.execute":
> >>> >>>> > root 3230 0.0 0.0 41400 9272 ? S 10:16 0:00
> >>> pgpool:
> >>> >>>> > XXXX XXXX 192.168.XXX.XXX(59576) idle in transaction
> >>> >>>> >
> >>> >>>> > pgpool 4.1.8:
> >>> >>>> > After "psycopg.connect":
> >>> >>>> > root 5317 0.0 0.0 39044 2564 ? S 11:35 0:00
> >>> pgpool:
> >>> >>>> > zope nube 192.168.XXX.XXX(54784) idle
> >>> >>>> >
> >>> >>>> > After "cur.execute":
> >>> >>>> > root 5317 0.0 0.2 72200 37040 ? S 11:35 0:00
> >>> pgpool:
> >>> >>>> > zope nube 192.168.XXX.XXX(54784) idle in transaction
> >>> >>>> >
> >>> >>>> > Do you think this is enough to reproduce? Or do you need
> something
> >>> >>>> > different? Let me know, I'm here to help.
> >>> >>>> >
> >>> >>>> >
> >>> >>>> > On Wed, Sep 29, 2021 at 8:54 AM Tatsuo Ishii <
> ishii at sraoss.co.jp>
> >>> >>>> wrote:
> >>> >>>> >
> >>> >>>> >> So enable_shared_relcache is irrelevant.
> >>> >>>> >>
> >>> >>>> >> Problem is, I cannot reproduce your problem here. It would be
> >>> greately
> >>> >>>> >> helpful if you can provide a self containing test case. If you
> >>> cannot,
> >>> >>>> >> I would like to find a pattern (query) which triggers the
> problem.
> >>> >>>> >>
> >>> >>>> >> - Does the query use extended protocol?
> >>> >>>> >>
> >>> >>>> >> - What kind of queries does the application use?
> >>> >>>> >> INSERT/UPDATE/SELECT/DELETE??
> >>> >>>> >>
> >>> >>>> >> - IS there anything special with the application?
> >>> >>>> >>
> >>> >>>> >> > I tested 4.1.0 now and it shows the same behaviour.
> >>> >>>> >> >
> >>> >>>> >> > Also, disabling enable_shared_relcache did not solve the
> problem.
> >>> >>>> >> >
> >>> >>>> >> > On Tue, Sep 28, 2021 at 9:10 PM Tatsuo Ishii <
> ishii at sraoss.co.jp
> >>> >
> >>> >>>> wrote:
> >>> >>>> >> >
> >>> >>>> >> >> Ok. Let's assume 4.1.0 introduced the problem for now (it's
> >>> still
> >>> >>>> >> >> possible that certain minor version of 4.1 introduced the
> >>> issue, but
> >>> >>>> >> >> for now I neglect the possibility).
> >>> >>>> >> >>
> >>> >>>> >> >> 4.1.0 includes following major changes especially related to
> >>> memory.
> >>> >>>> >> >>
> >>> >>>> >> >> (1) Shared relation cache allows to reuse relation cache
> among
> >>> >>>> >> >> sessions to reduce internal queries against PostgreSQL
> >>> system
> >>> >>>> >> >> catalogs.
> >>> >>>> >> >>
> >>> >>>> >> >> (this feature uses shared memory even if
> >>> memory_cache_enabled =
> >>> >>>> >> off).
> >>> >>>> >> >>
> >>> >>>> >> >> (2) Have separate SQL parser for DML statements to eliminate
> >>> >>>> >> >> unnecessary parsing effort.
> >>> >>>> >> >>
> >>> >>>> >> >> (this allocates additional process memory for certain
> query)
> >>> >>>> >> >>
> >>> >>>> >> >> For (1) we can disable the feature by:
> >>> >>>> >> >>
> >>> >>>> >> >> enable_shared_relcache = off
> >>> >>>> >> >>
> >>> >>>> >> >> Can you try it out?
> >>> >>>> >> >>
> >>> >>>> >> >> > I did those tests:
> >>> >>>> >> >> > 4.0.15 is running fine, when I upgraded to 4.1.8 it
> started to
> >>> >>>> >> consume a
> >>> >>>> >> >> > lot of memory.
> >>> >>>> >> >> >
> >>> >>>> >> >> > On Tue, Sep 28, 2021 at 8:30 AM Tatsuo Ishii <
> >>> ishii at sraoss.co.jp>
> >>> >>>> >> wrote:
> >>> >>>> >> >> >
> >>> >>>> >> >> >> Ok. 4.2.1 consumes 10 times of memory than 3.7.3 (maybe
> 5,
> >>> not 10
> >>> >>>> >> >> >> because in your test connections of 4.2.1 is twice of
> 3.7.3).
> >>> >>>> >> >> >>
> >>> >>>> >> >> >> Next thing I need to know is, from which version of
> pgpool
> >>> >>>> started to
> >>> >>>> >> >> >> consume lot of memory. Is it possible to run the same
> test
> >>> >>>> against
> >>> >>>> >> >> >> 4.0 and 4.1? Ideally from which version, including minor
> >>> version
> >>> >>>> will
> >>> >>>> >> >> >> be a lot helpful. I wish I could the test myself but I
> don't
> >>> >>>> know to
> >>> >>>> >> >> >> do it on my environment.
> >>> >>>> >> >> >>
> >>> >>>> >> >> >> > Sure.
> >>> >>>> >> >> >> >
> >>> >>>> >> >> >> > On 3.7.3:
> >>> >>>> >> >> >> > $ ps aux | grep pgpool | awk '{sum += $6}END{print
> sum}'
> >>> >>>> >> >> >> > 896752
> >>> >>>> >> >> >> >
> >>> >>>> >> >> >> > On 4.2.1:
> >>> >>>> >> >> >> > $ ps aux | grep pgpool | awk '{sum += $6}END{print
> sum}'
> >>> >>>> >> >> >> > 9969280
> >>> >>>> >> >> >> >
> >>> >>>> >> >> >> >
> >>> >>>> >> >> >> > On Mon, Sep 27, 2021 at 11:41 PM Tatsuo Ishii <
> >>> >>>> ishii at sraoss.co.jp>
> >>> >>>> >> >> >> wrote:
> >>> >>>> >> >> >> >
> >>> >>>> >> >> >> >> Ok, let's see how much RSS pgpool consume in total.
> Can
> >>> you
> >>> >>>> share
> >>> >>>> >> the
> >>> >>>> >> >> >> >> result of the command for both pgpool 4.2 and 3.7?
> >>> >>>> >> >> >> >>
> >>> >>>> >> >> >> >> ps aux | grep pgpool | awk '{sum += $6}END{print sum}'
> >>> >>>> >> >> >> >>
> >>> >>>> >> >> >> >> > I see.
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > The problem is, the OOM happens when all RAM is
> already
> >>> >>>> >> >> compromised by
> >>> >>>> >> >> >> >> > pgpool (all 16Gb of RAM is dedicated to pgpool), so
> any
> >>> >>>> common
> >>> >>>> >> >> request
> >>> >>>> >> >> >> >> will
> >>> >>>> >> >> >> >> > die anyway.
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > It happens on versions: 4.2.5 and 4.2.1. Now, I'm
> >>> running
> >>> >>>> the
> >>> >>>> >> same
> >>> >>>> >> >> >> >> > application on 3.7.3 and it's running great.
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > I don't know if it helps, but just to demonstrate
> the
> >>> >>>> amount of
> >>> >>>> >> >> RAM is
> >>> >>>> >> >> >> >> > necessary to run a few connections I created another
> >>> VM, and
> >>> >>>> >> >> >> comparing:
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > 62 active connections running on 4.2.1 (same
> behaviour
> >>> on
> >>> >>>> >> 4.2.5):
> >>> >>>> >> >> >> >> > $ vmstat -a -SM
> >>> >>>> >> >> >> >> > procs -----------memory---------- ---swap--
> -----io----
> >>> >>>> >> -system--
> >>> >>>> >> >> >> >> > ------cpu-----
> >>> >>>> >> >> >> >> > r b swpd free inact active si so bi
> >>> bo in
> >>> >>>> >> cs
> >>> >>>> >> >> us
> >>> >>>> >> >> >> sy
> >>> >>>> >> >> >> >> id
> >>> >>>> >> >> >> >> > wa st
> >>> >>>> >> >> >> >> > 0 0 0 4829 173 10772 0 0 2
> >>> 1 68
> >>> >>>> >> 56
> >>> >>>> >> >> 1
> >>> >>>> >> >> >> 1
> >>> >>>> >> >> >> >> 99
> >>> >>>> >> >> >> >> > 0 0
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > 31 active connections running on 3.7.3:
> >>> >>>> >> >> >> >> > $ vmstat -a -SM
> >>> >>>> >> >> >> >> > procs -----------memory---------- ---swap--
> -----io----
> >>> >>>> >> -system--
> >>> >>>> >> >> >> >> > ------cpu-----
> >>> >>>> >> >> >> >> > r b swpd free inact active si so bi
> >>> bo in
> >>> >>>> >> cs
> >>> >>>> >> >> us
> >>> >>>> >> >> >> sy
> >>> >>>> >> >> >> >> id
> >>> >>>> >> >> >> >> > wa st
> >>> >>>> >> >> >> >> > 0 0 0 15174 73 635 0 0 1
> >>> 1 76
> >>> >>>> >> 3
> >>> >>>> >> >> 0
> >>> >>>> >> >> >> 0
> >>> >>>> >> >> >> >> 100
> >>> >>>> >> >> >> >> > 0 0
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > Both are running basically the same application.
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > Seeing the RAM consumption of 3.7.3, it does not
> seem
> >>> to be
> >>> >>>> >> >> receiving
> >>> >>>> >> >> >> >> such
> >>> >>>> >> >> >> >> > a big requests. Is there a way to track it down?
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > On Mon, Sep 27, 2021 at 9:07 AM Tatsuo Ishii <
> >>> >>>> >> ishii at sraoss.co.jp>
> >>> >>>> >> >> >> wrote:
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> >> At this point I cannot judge whether the problem is
> >>> caused
> >>> >>>> by a
> >>> >>>> >> >> >> pgpool
> >>> >>>> >> >> >> >> >> bug or client's resource request is too much.
> >>> >>>> >> >> >> >> >>
> >>> >>>> >> >> >> >> >> Typical bug which requests too much memory
> allocation
> >>> is
> >>> >>>> not
> >>> >>>> >> >> >> something
> >>> >>>> >> >> >> >> >> like this because 33947648 = 32MB memory request
> >>> itself is
> >>> >>>> not
> >>> >>>> >> >> >> insane.
> >>> >>>> >> >> >> >> >>
> >>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-2]
> >>> 2021-09-24
> >>> >>>> >> >> 09:14:10:
> >>> >>>> >> >> >> pid
> >>> >>>> >> >> >> >> >> > 12650: DETAIL: Failed on request of size
> 33947648.
> >>> >>>> >> >> >> >> >>
> >>> >>>> >> >> >> >> >> (Please let us know what version of Pgpool-II you
> are
> >>> using
> >>> >>>> >> >> because
> >>> >>>> >> >> >> >> >> it's important information to identify any known
> bug).
> >>> >>>> >> >> >> >> >>
> >>> >>>> >> >> >> >> >> In the mean time, however, I think 32MB memory
> request
> >>> is
> >>> >>>> not
> >>> >>>> >> very
> >>> >>>> >> >> >> >> >> common in pgpool. One thing I wonder is, whether
> your
> >>> >>>> >> application
> >>> >>>> >> >> >> >> >> issues SQL which requires large memory: e.g. very
> long
> >>> SQL
> >>> >>>> >> >> statement,
> >>> >>>> >> >> >> >> >> COPY for large data set. They will request large
> >>> read/write
> >>> >>>> >> >> buffer in
> >>> >>>> >> >> >> >> >> pgpool.
> >>> >>>> >> >> >> >> >>
> >>> >>>> >> >> >> >> >> > We saw both, but pgpool aborting is way more
> common:
> >>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-1]
> >>> 2021-09-24
> >>> >>>> >> >> 09:14:10:
> >>> >>>> >> >> >> pid
> >>> >>>> >> >> >> >> >> > 12650: ERROR: out of memory
> >>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-2]
> >>> 2021-09-24
> >>> >>>> >> >> 09:14:10:
> >>> >>>> >> >> >> pid
> >>> >>>> >> >> >> >> >> > 12650: DETAIL: Failed on request of size
> 33947648.
> >>> >>>> >> >> >> >> >> > Sep 24 09:14:10 pgpool pgpool[12650]: [426-3]
> >>> 2021-09-24
> >>> >>>> >> >> 09:14:10:
> >>> >>>> >> >> >> pid
> >>> >>>> >> >> >> >> >> > 12650: LOCATION: mcxt.c:900
> >>> >>>> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> > Here, two other ways we saw at the logs, but
> those
> >>> just
> >>> >>>> >> occurred
> >>> >>>> >> >> >> once
> >>> >>>> >> >> >> >> >> each:
> >>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-1]
> >>> 2021-09-24
> >>> >>>> >> >> 07:33:14:
> >>> >>>> >> >> >> pid
> >>> >>>> >> >> >> >> >> 5874:
> >>> >>>> >> >> >> >> >> > FATAL: failed to fork a child
> >>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-2]
> >>> 2021-09-24
> >>> >>>> >> >> 07:33:14:
> >>> >>>> >> >> >> pid
> >>> >>>> >> >> >> >> >> 5874:
> >>> >>>> >> >> >> >> >> > DETAIL: system call fork() failed with reason:
> >>> Cannot
> >>> >>>> >> allocate
> >>> >>>> >> >> >> memory
> >>> >>>> >> >> >> >> >> > Sep 24 07:33:14 pgpool pgpool[5874]: [434-3]
> >>> 2021-09-24
> >>> >>>> >> >> 07:33:14:
> >>> >>>> >> >> >> pid
> >>> >>>> >> >> >> >> >> 5874:
> >>> >>>> >> >> >> >> >> > LOCATION: pgpool_main.c:681
> >>> >>>> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> > And:
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691518]
> pgpool
> >>> >>>> invoked
> >>> >>>> >> >> >> >> oom-killer:
> >>> >>>> >> >> >> >> >> > gfp_mask=0x24200ca, order=0, oom_score_adj=0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691525]
> CPU: 1
> >>> >>>> PID:
> >>> >>>> >> 1194
> >>> >>>> >> >> >> Comm:
> >>> >>>> >> >> >> >> >> > pgpool Not tainted 4.4.276 #1
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691527]
> >>> Hardware
> >>> >>>> name:
> >>> >>>> >> >> >> VMware,
> >>> >>>> >> >> >> >> >> Inc.
> >>> >>>> >> >> >> >> >> > VMware Virtual Platform/440BX Desktop Reference
> >>> Platform,
> >>> >>>> >> BIOS
> >>> >>>> >> >> 6.00
> >>> >>>> >> >> >> >> >> > 12/12/2018
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691528]
> >>> >>>> >> 0000000000000000
> >>> >>>> >> >> >> >> >> > ffff8803281cbae8 ffffffff81c930e7
> ffff880420efe2c0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691530]
> >>> >>>> >> ffff880420efe2c0
> >>> >>>> >> >> >> >> >> > ffff8803281cbb50 ffffffff81c8d9da
> ffff8803281cbb08
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691532]
> >>> >>>> >> ffffffff81133f1a
> >>> >>>> >> >> >> >> >> > ffff8803281cbb80 ffffffff81182eb0
> ffff8800bba393c0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691533]
> Call
> >>> >>>> Trace:
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691540]
> >>> >>>> >> >> >> [<ffffffff81c930e7>]
> >>> >>>> >> >> >> >> >> > dump_stack+0x57/0x6d
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691542]
> >>> >>>> >> >> >> [<ffffffff81c8d9da>]
> >>> >>>> >> >> >> >> >> > dump_header.isra.9+0x54/0x1ae
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691547]
> >>> >>>> >> >> >> [<ffffffff81133f1a>] ?
> >>> >>>> >> >> >> >> >> > __delayacct_freepages_end+0x2a/0x30
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691553]
> >>> >>>> >> >> >> [<ffffffff81182eb0>] ?
> >>> >>>> >> >> >> >> >> > do_try_to_free_pages+0x350/0x3d0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691556]
> >>> >>>> >> >> >> [<ffffffff811709f9>]
> >>> >>>> >> >> >> >> >> > oom_kill_process+0x209/0x3c0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691558]
> >>> >>>> >> >> >> [<ffffffff81170eeb>]
> >>> >>>> >> >> >> >> >> > out_of_memory+0x2db/0x2f0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691561]
> >>> >>>> >> >> >> [<ffffffff81176111>]
> >>> >>>> >> >> >> >> >> > __alloc_pages_nodemask+0xa81/0xae0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691565]
> >>> >>>> >> >> >> [<ffffffff811ad2cd>]
> >>> >>>> >> >> >> >> >> > __read_swap_cache_async+0xdd/0x130
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691567]
> >>> >>>> >> >> >> [<ffffffff811ad337>]
> >>> >>>> >> >> >> >> >> > read_swap_cache_async+0x17/0x40
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691569]
> >>> >>>> >> >> >> [<ffffffff811ad455>]
> >>> >>>> >> >> >> >> >> > swapin_readahead+0xf5/0x190
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691571]
> >>> >>>> >> >> >> [<ffffffff8119ce3f>]
> >>> >>>> >> >> >> >> >> > handle_mm_fault+0xf3f/0x15e0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691574]
> >>> >>>> >> >> >> [<ffffffff81c9d4e2>] ?
> >>> >>>> >> >> >> >> >> > __schedule+0x272/0x770
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691576]
> >>> >>>> >> >> >> [<ffffffff8104e241>]
> >>> >>>> >> >> >> >> >> > __do_page_fault+0x161/0x370
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691577]
> >>> >>>> >> >> >> [<ffffffff8104e49c>]
> >>> >>>> >> >> >> >> >> > do_page_fault+0xc/0x10
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691579]
> >>> >>>> >> >> >> [<ffffffff81ca3782>]
> >>> >>>> >> >> >> >> >> > page_fault+0x22/0x30
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691581]
> >>> Mem-Info:
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
> >>> >>>> >> >> active_anon:3564689
> >>> >>>> >> >> >> >> >> > inactive_anon:445592 isolated_anon:0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
> >>> >>>> >> active_file:462
> >>> >>>> >> >> >> >> >> > inactive_file:44 isolated_file:0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
> >>> >>>> unevictable:0
> >>> >>>> >> >> >> dirty:2
> >>> >>>> >> >> >> >> >> > writeback:2212 unstable:0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
> >>> >>>> >> >> >> slab_reclaimable:3433
> >>> >>>> >> >> >> >> >> > slab_unreclaimable:5859
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
> >>> >>>> mapped:989
> >>> >>>> >> >> >> shmem:2607
> >>> >>>> >> >> >> >> >> > pagetables:16367 bounce:0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691584]
> >>> >>>> free:51773
> >>> >>>> >> >> >> >> free_pcp:189
> >>> >>>> >> >> >> >> >> > free_cma:0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691589]
> DMA
> >>> >>>> >> free:15904kB
> >>> >>>> >> >> >> >> min:128kB
> >>> >>>> >> >> >> >> >> > low:160kB high:192kB active_anon:0kB
> >>> inactive_anon:0kB
> >>> >>>> >> >> >> active_file:0kB
> >>> >>>> >> >> >> >> >> > inactive_file:0kB unevictable:0kB
> isolated(anon):0kB
> >>> >>>> >> >> >> >> isolated(file):0kB
> >>> >>>> >> >> >> >> >> > present:15992kB managed:15904kB mlocked:0kB
> dirty:0kB
> >>> >>>> >> >> writeback:0kB
> >>> >>>> >> >> >> >> >> > mapped:0kB shmem:0kB slab_reclaimable:0kB
> >>> >>>> >> slab_unreclaimable:0kB
> >>> >>>> >> >> >> >> >> > kernel_stack:0kB pagetables:0kB unstable:0kB
> >>> bounce:0kB
> >>> >>>> >> >> >> free_pcp:0kB
> >>> >>>> >> >> >> >> >> > local_pcp:0kB free_cma:0kB writeback_tmp:0kB
> >>> >>>> pages_scanned:0
> >>> >>>> >> >> >> >> >> > all_unreclaimable? yes
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691590]
> >>> >>>> >> >> lowmem_reserve[]: 0
> >>> >>>> >> >> >> >> 2960
> >>> >>>> >> >> >> >> >> > 15991 15991
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691596]
> DMA32
> >>> >>>> >> >> free:77080kB
> >>> >>>> >> >> >> >> >> > min:25000kB low:31248kB high:37500kB
> >>> >>>> active_anon:2352676kB
> >>> >>>> >> >> >> >> >> > inactive_anon:590536kB active_file:472kB
> >>> >>>> inactive_file:140kB
> >>> >>>> >> >> >> >> >> > unevictable:0kB isolated(anon):0kB
> isolated(file):0kB
> >>> >>>> >> >> >> >> present:3129216kB
> >>> >>>> >> >> >> >> >> > managed:3043556kB mlocked:0kB dirty:0kB
> >>> writeback:4144kB
> >>> >>>> >> >> >> mapped:1140kB
> >>> >>>> >> >> >> >> >> > shmem:3568kB slab_reclaimable:1028kB
> >>> >>>> >> slab_unreclaimable:3004kB
> >>> >>>> >> >> >> >> >> > kernel_stack:816kB pagetables:10988kB
> unstable:0kB
> >>> >>>> bounce:0kB
> >>> >>>> >> >> >> >> >> > free_pcp:312kB local_pcp:196kB free_cma:0kB
> >>> >>>> writeback_tmp:0kB
> >>> >>>> >> >> >> >> >> > pages_scanned:4292 all_unreclaimable? yes
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691597]
> >>> >>>> >> >> lowmem_reserve[]: 0
> >>> >>>> >> >> >> 0
> >>> >>>> >> >> >> >> >> 13031
> >>> >>>> >> >> >> >> >> > 13031
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691601]
> Normal
> >>> >>>> >> >> free:114108kB
> >>> >>>> >> >> >> >> >> > min:110032kB low:137540kB high:165048kB
> >>> >>>> >> active_anon:11906080kB
> >>> >>>> >> >> >> >> >> > inactive_anon:1191832kB active_file:1376kB
> >>> >>>> inactive_file:36kB
> >>> >>>> >> >> >> >> >> > unevictable:0kB isolated(anon):0kB
> isolated(file):0kB
> >>> >>>> >> >> >> >> present:13631488kB
> >>> >>>> >> >> >> >> >> > managed:13343784kB mlocked:0kB dirty:8kB
> >>> writeback:4704kB
> >>> >>>> >> >> >> >> mapped:2816kB
> >>> >>>> >> >> >> >> >> > shmem:6860kB slab_reclaimable:12704kB
> >>> >>>> >> slab_unreclaimable:20432kB
> >>> >>>> >> >> >> >> >> > kernel_stack:4848kB pagetables:54480kB
> unstable:0kB
> >>> >>>> >> bounce:0kB
> >>> >>>> >> >> >> >> >> > free_pcp:444kB local_pcp:196kB free_cma:0kB
> >>> >>>> writeback_tmp:0kB
> >>> >>>> >> >> >> >> >> > pages_scanned:105664 all_unreclaimable? yes
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691602]
> >>> >>>> >> >> lowmem_reserve[]: 0
> >>> >>>> >> >> >> 0
> >>> >>>> >> >> >> >> 0 0
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691603]
> DMA:
> >>> 0*4kB
> >>> >>>> >> 0*8kB
> >>> >>>> >> >> >> 0*16kB
> >>> >>>> >> >> >> >> >> > 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U)
> 0*512kB
> >>> >>>> >> 1*1024kB
> >>> >>>> >> >> (U)
> >>> >>>> >> >> >> >> >> 1*2048kB
> >>> >>>> >> >> >> >> >> > (U) 3*4096kB (M) = 15904kB
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691610]
> DMA32:
> >>> >>>> 48*4kB
> >>> >>>> >> >> (ME)
> >>> >>>> >> >> >> >> 63*8kB
> >>> >>>> >> >> >> >> >> > (ME) 62*16kB (E) 46*32kB (UME) 35*64kB (UME)
> 26*128kB
> >>> >>>> (UME)
> >>> >>>> >> >> >> 25*256kB
> >>> >>>> >> >> >> >> >> (UME)
> >>> >>>> >> >> >> >> >> > 61*512kB (UME) 28*1024kB (UME) 1*2048kB (M)
> 0*4096kB
> >>> =
> >>> >>>> >> 77080kB
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691616]
> >>> Normal:
> >>> >>>> >> 165*4kB
> >>> >>>> >> >> >> (MEH)
> >>> >>>> >> >> >> >> >> > 317*8kB (UMEH) 442*16kB (UMEH) 289*32kB (UMEH)
> >>> 162*64kB
> >>> >>>> >> (UMEH)
> >>> >>>> >> >> >> >> 121*128kB
> >>> >>>> >> >> >> >> >> > (UMEH) 74*256kB (UMEH) 33*512kB (UMEH) 24*1024kB
> (ME)
> >>> >>>> >> 0*2048kB
> >>> >>>> >> >> >> >> 2*4096kB
> >>> >>>> >> >> >> >> >> (M)
> >>> >>>> >> >> >> >> >> > = 113980kB
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691623]
> 5552
> >>> total
> >>> >>>> >> >> pagecache
> >>> >>>> >> >> >> >> pages
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691624]
> 2355
> >>> >>>> pages in
> >>> >>>> >> >> swap
> >>> >>>> >> >> >> >> cache
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691625]
> Swap
> >>> cache
> >>> >>>> >> stats:
> >>> >>>> >> >> >> add
> >>> >>>> >> >> >> >> >> > 5385308, delete 5382953, find 1159094/1325033
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691626]
> Free
> >>> >>>> swap =
> >>> >>>> >> 0kB
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691626]
> Total
> >>> >>>> swap =
> >>> >>>> >> >> >> 4194300kB
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691627]
> >>> 4194174
> >>> >>>> pages
> >>> >>>> >> RAM
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691628] 0
> >>> pages
> >>> >>>> >> >> >> >> >> HighMem/MovableOnly
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691628]
> 93363
> >>> >>>> pages
> >>> >>>> >> >> reserved
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691989]
> Out of
> >>> >>>> memory:
> >>> >>>> >> >> Kill
> >>> >>>> >> >> >> >> >> process
> >>> >>>> >> >> >> >> >> > 8975 (pgpool) score 7 or sacrifice child
> >>> >>>> >> >> >> >> >> > Sep 23 17:07:40 pgpool kernel: [157160.691995]
> Killed
> >>> >>>> process
> >>> >>>> >> >> 8975
> >>> >>>> >> >> >> >> >> (pgpool)
> >>> >>>> >> >> >> >> >> > total-vm:337504kB, anon-rss:166824kB,
> file-rss:1920kB
> >>> >>>> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> > On Mon, Sep 27, 2021 at 1:22 AM Tatsuo Ishii <
> >>> >>>> >> >> ishii at sraoss.co.jp>
> >>> >>>> >> >> >> >> wrote:
> >>> >>>> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >> Hi,
> >>> >>>> >> >> >> >> >> >>
> >>> >>>> >> >> >> >> >> >> > Hello,
> >>> >>>> >> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >> > Our pgpool is consuming A LOT of memory and
> >>> frequently
> >>> >>>> >> dies
> >>> >>>> >> >> >> with:
> >>> >>>> >> >> >> >> Out
> >>> >>>> >> >> >> >> >> of
> >>> >>>> >> >> >> >> >> >> > memory error.
> >>> >>>> >> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >> > We have 2 backends, 1 master and 1 slave. Here
> >>> some
> >>> >>>> >> config:
> >>> >>>> >> >> >> >> >> >> > num_init_children = 150
> >>> >>>> >> >> >> >> >> >> > max_pool = 1
> >>> >>>> >> >> >> >> >> >> > child_life_time = 300
> >>> >>>> >> >> >> >> >> >> > child_max_connections = 1
> >>> >>>> >> >> >> >> >> >> > connection_life_time = 0
> >>> >>>> >> >> >> >> >> >> > client_idle_limit = 0
> >>> >>>> >> >> >> >> >> >> > connection_cache = on
> >>> >>>> >> >> >> >> >> >> > load_balance_mode = on
> >>> >>>> >> >> >> >> >> >> > memory_cache_enabled = off
> >>> >>>> >> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >> > RAM: 16Gb
> >>> >>>> >> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >> > Does anyone have a clue what's going on?
> >>> >>>> >> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >> > Thank you.
> >>> >>>> >> >> >> >> >> >>
> >>> >>>> >> >> >> >> >> >> Is that OOM killer or pgpool itself aborted
> with an
> >>> out
> >>> >>>> of
> >>> >>>> >> >> memory
> >>> >>>> >> >> >> >> >> >> error? If latter, can you share the pgpool log?
> >>> >>>> >> >> >> >> >> >> --
> >>> >>>> >> >> >> >> >> >> Tatsuo Ishii
> >>> >>>> >> >> >> >> >> >> SRA OSS, Inc. Japan
> >>> >>>> >> >> >> >> >> >> English: http://www.sraoss.co.jp/index_en.php
> >>> >>>> >> >> >> >> >> >> Japanese:http://www.sraoss.co.jp
> >>> >>>> >> >> >> >> >> >>
> >>> >>>> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> >
> >>> >>>> >> >> >> >> >> > --
> >>> >>>> >> >> >> >> >> > Luiz Fernando Pasqual S. Souza
> >>> >>>> >> >> >> >> >> > mail: luiz at pasquall.com
> >>> >>>> >> >> >> >> >>
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> >
> >>> >>>> >> >> >> >> > --
> >>> >>>> >> >> >> >> > Luiz Fernando Pasqual S. Souza
> >>> >>>> >> >> >> >> > mail: luiz at pasquall.com
> >>> >>>> >> >> >> >>
> >>> >>>> >> >> >> >
> >>> >>>> >> >> >> >
> >>> >>>> >> >> >> > --
> >>> >>>> >> >> >> > Luiz Fernando Pasqual S. Souza
> >>> >>>> >> >> >> > mail: luiz at pasquall.com
> >>> >>>> >> >> >>
> >>> >>>> >> >> >
> >>> >>>> >> >> >
> >>> >>>> >> >> > --
> >>> >>>> >> >> > Luiz Fernando Pasqual S. Souza
> >>> >>>> >> >> > mail: luiz at pasquall.com
> >>> >>>> >> >> _______________________________________________
> >>> >>>> >> >> pgpool-general mailing list
> >>> >>>> >> >> pgpool-general at pgpool.net
> >>> >>>> >> >> http://www.pgpool.net/mailman/listinfo/pgpool-general
> >>> >>>> >> >>
> >>> >>>> >> >
> >>> >>>> >> >
> >>> >>>> >> > --
> >>> >>>> >> > Luiz Fernando Pasqual S. Souza
> >>> >>>> >> > mail: luiz at pasquall.com
> >>> >>>> >>
> >>> >>>> >
> >>> >>>> >
> >>> >>>> > --
> >>> >>>> > Luiz Fernando Pasqual S. Souza
> >>> >>>> > mail: luiz at pasquall.com
> >>> >>>>
> >>> >>>
> >>> >>>
> >>> >>> --
> >>> >>> Luiz Fernando Pasqual S. Souza
> >>> >>> mail: luiz at pasquall.com
> >>> >>>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Luiz Fernando Pasqual S. Souza
> >>> >> mail: luiz at pasquall.com
> >>> > _______________________________________________
> >>> > pgpool-general mailing list
> >>> > pgpool-general at pgpool.net
> >>> > http://www.pgpool.net/mailman/listinfo/pgpool-general
> >>>
> >>
> >>
> >> --
> >> Luiz Fernando Pasqual S. Souza
> >> mail: luiz at pasquall.com
> > _______________________________________________
> > pgpool-general mailing list
> > pgpool-general at pgpool.net
> > http://www.pgpool.net/mailman/listinfo/pgpool-general
> _______________________________________________
> pgpool-general mailing list
> pgpool-general at pgpool.net
> http://www.pgpool.net/mailman/listinfo/pgpool-general
>
--
Luiz Fernando Pasqual S. Souza
mail: luiz at pasquall.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211011/aa34f1d5/attachment-0001.htm>
More information about the pgpool-general
mailing list