[pgpool-general: 7572] Re: Resource temporarily unavailable
Rozmus Andrzej
Andrzej.Rozmus at asseco.pl
Thu May 13 17:06:45 JST 2021
Hello Tatsuo,
Thank you for your reply. I checked number of file descriptors but it can't be a problem. Below is result:
pgs00000162:~ # cat /proc/sys/fs/file-nr
18080 0 13147086
As i mentioned in my last message i have changed "TaskMax" value to "infinity" and pgpool seems to be working fine. I started pgpool service two days ago and till now none failure appeared:
pgs00000162:~ # systemctl status pgpool-II-11.service
● pgpool-II-11.service - PGPool-II Middleware Between PostgreSQL Servers And PostgreSQL Database Clients
Loaded: loaded (/usr/lib/systemd/system/pgpool-II-11.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2021-05-11 10:28:36 CEST; 1 day 23h ago
Main PID: 8500 (pgpool)
Tasks: 1009
CGroup: /system.slice/pgpool-II-11.service
My linux is Suse Enterprise Server 12 SP3 (trust me, it is not my choice).
Best regards and once again thank you for help.
Andrzej Rozmus
Starszy Technolog
Departament Systemów Administracji Rządowej
Dział Ministerstwo Finansów
Zespół Projektowy AIS/AES
Asseco Poland S.A.
ul. Branickiego 13
02-972 Warszawa
tel. kom. +48 502 270 059
andrzej.rozmus at asseco.pl
-----Original Message-----
From: Tatsuo Ishii <ishii at sraoss.co.jp>
Sent: Thursday, May 13, 2021 8:19 AM
To: Rozmus Andrzej <Andrzej.Rozmus at asseco.pl>
Cc: pgpool-general at pgpool.net; Wierzbicki Sebastian <sebastian.wierzbicki at asseco.pl>; tomasz.pajda at mf.gov.pl
Subject: Re: [pgpool-general: 7563] Resource temporarily unavailable
> Hello Tatsuo,
> This is the result:
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size 8388608 unlimited bytes
> Max core file size 0 unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 514179 514179 processes
> Max open files 1024 4096 files
> Max locked memory 65536 65536 bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 514179 514179 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
>
> Pgpool service starts without any problems and creates about 1008 tasks - problem apprears after some time. Today it happened again:
>
> pgs00000161:~ # systemctl status pgpool-II-11.service ●
> pgpool-II-11.service - PGPool-II Middleware Between PostgreSQL Servers And PostgreSQL Database Clients
> Loaded: loaded (/usr/lib/systemd/system/pgpool-II-11.service; disabled; vendor preset: disabled)
> Active: active (running) since Mon 2021-05-10 16:15:04 CEST; 17h ago
> Process: 20750 ExecStop=/usr/pgpool-11/bin/pgpool -f
> /etc/pgpool-II-11/pgpool.conf -m fast stop (code=killed, signal=INT) Main PID: 21255 (pgpool)
> Tasks: 3 (limit: 1024)
> CGroup: /system.slice/pgpool-II-11.service
> ├─21255 /usr/pgpool-11/bin/pgpool -f /etc/pgpool-II-11/pgpool.conf -n -D
> ├─22359 pgpool: health check process(0)
> └─22360 pgpool: health check process(1)
>
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2906-1] 2021-05-10
> 23:36:38: pid 21255: LOG: fork a new child process with pid: 14222
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2907-1] 2021-05-10
> 23:36:38: pid 21255: LOG: child process with pid: 7431 exits wi...tus
> 256 May 10 23:36:38 pgs00000161 pgpool[21255]: [2908-1] 2021-05-10
> 23:36:38: pid 21255: LOG: fork a new child process with pid: 14223
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2909-1] 2021-05-10
> 23:36:38: pid 21255: LOG: child process with pid: 8142 exits wi...tus
> 256 May 10 23:36:38 pgs00000161 pgpool[21255]: [2910-1] 2021-05-10
> 23:36:38: pid 21255: LOG: fork a new child process with pid: 14224
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2911-1] 2021-05-10
> 23:36:38: pid 21255: LOG: child process with pid: 9498 exits wi...tus
> 256 May 10 23:36:38 pgs00000161 pgpool[21255]: [2912-1] 2021-05-10
> 23:36:38: pid 21255: LOG: fork a new child process with pid: 14225
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2913-1] 2021-05-10
> 23:36:38: pid 21255: LOG: child process with pid: 4782 exits wi...tus
> 256 May 10 23:36:38 pgs00000161 pgpool[21255]: [2914-1] 2021-05-10
> 23:36:38: pid 21255: FATAL: failed to fork a child May 10 23:36:38
> pgs00000161 pgpool[21255]: [2914-2] 2021-05-10 23:36:38: pid 21255:
> DETAIL: system call fork() failed with rea...ailable
> Hint: Some lines were ellipsized, use -l to show in full.
>
> It looks like processes responsible for client connections disappear suddenly and new instances cannot be created but restart of pgpool service helps and it works well again some time.
> I have recommendation to set TasksMax to "infinity" or value much higher than num_init_children*max_pool, but i will be able to verify it during service window in next six hours.
Another possibility is number of file descriptors. On my Lunux box after startinf 3-node Pgpool-II with 3-node PostgreSQL, I see:
$ cat /proc/sys/fs/file-nr
21700 0 1612809
So 21700 file descriptors out of 1612809 are used. Do you see number of the used fds are closing to the max file descriptor on your system?
Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en.php&data=04%7C01%7CAndrzej.Rozmus%40asseco.pl%7C0754b21105be4cfce5b708d915d6fe44%7C88152bdecfa34a5cb981a785c624bb42%7C0%7C0%7C637564835400381362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hgfINGc6Awc3WqDuH9JBi1mMYQfZTxQIb2y3lB7F7mo%3D&reserved=0
Japanese:https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sraoss.co.jp%2F&data=04%7C01%7CAndrzej.Rozmus%40asseco.pl%7C0754b21105be4cfce5b708d915d6fe44%7C88152bdecfa34a5cb981a785c624bb42%7C0%7C0%7C637564835400381362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GKoSEnL3H8%2BXlpeDteC352q2LVgc4dr6KGsAbgJbkuA%3D&reserved=0
More information about the pgpool-general
mailing list