[pgpool-general: 7573] Re: Resource temporarily unavailable
Tatsuo Ishii
ishii at sraoss.co.jp
Thu May 13 17:54:53 JST 2021
> Hello Tatsuo,
> Thank you for your reply. I checked number of file descriptors but it can't be a problem. Below is result:
>
> pgs00000162:~ # cat /proc/sys/fs/file-nr
> 18080 0 13147086
>
> As i mentioned in my last message i have changed "TaskMax" value to "infinity" and pgpool seems to be working fine. I started pgpool service two days ago and till now none failure appeared:
Oh, OK. I though your problem wasn't solved yet. Glad to hear that now
your problem has gone.
> pgs00000162:~ # systemctl status pgpool-II-11.service
> ● pgpool-II-11.service - PGPool-II Middleware Between PostgreSQL Servers And PostgreSQL Database Clients
> Loaded: loaded (/usr/lib/systemd/system/pgpool-II-11.service; disabled; vendor preset: disabled)
> Active: active (running) since Tue 2021-05-11 10:28:36 CEST; 1 day 23h ago
> Main PID: 8500 (pgpool)
> Tasks: 1009
> CGroup: /system.slice/pgpool-II-11.service
>
> My linux is Suse Enterprise Server 12 SP3 (trust me, it is not my choice).
> Best regards and once again thank you for help.
>
> Andrzej Rozmus
> Starszy Technolog
> Departament Systemów Administracji Rządowej
> Dział Ministerstwo Finansów
> Zespół Projektowy AIS/AES
> Asseco Poland S.A.
> ul. Branickiego 13
> 02-972 Warszawa
> tel. kom. +48 502 270 059
> andrzej.rozmus at asseco.pl
>
>
>
> -----Original Message-----
> From: Tatsuo Ishii <ishii at sraoss.co.jp>
> Sent: Thursday, May 13, 2021 8:19 AM
> To: Rozmus Andrzej <Andrzej.Rozmus at asseco.pl>
> Cc: pgpool-general at pgpool.net; Wierzbicki Sebastian <sebastian.wierzbicki at asseco.pl>; tomasz.pajda at mf.gov.pl
> Subject: Re: [pgpool-general: 7563] Resource temporarily unavailable
>
>> Hello Tatsuo,
>> This is the result:
>> Limit Soft Limit Hard Limit Units
>> Max cpu time unlimited unlimited seconds
>> Max file size unlimited unlimited bytes
>> Max data size unlimited unlimited bytes
>> Max stack size 8388608 unlimited bytes
>> Max core file size 0 unlimited bytes
>> Max resident set unlimited unlimited bytes
>> Max processes 514179 514179 processes
>> Max open files 1024 4096 files
>> Max locked memory 65536 65536 bytes
>> Max address space unlimited unlimited bytes
>> Max file locks unlimited unlimited locks
>> Max pending signals 514179 514179 signals
>> Max msgqueue size 819200 819200 bytes
>> Max nice priority 0 0
>> Max realtime priority 0 0
>> Max realtime timeout unlimited unlimited us
>>
>> Pgpool service starts without any problems and creates about 1008 tasks - problem apprears after some time. Today it happened again:
>>
>> pgs00000161:~ # systemctl status pgpool-II-11.service ●
>> pgpool-II-11.service - PGPool-II Middleware Between PostgreSQL Servers And PostgreSQL Database Clients
>> Loaded: loaded (/usr/lib/systemd/system/pgpool-II-11.service; disabled; vendor preset: disabled)
>> Active: active (running) since Mon 2021-05-10 16:15:04 CEST; 17h ago
>> Process: 20750 ExecStop=/usr/pgpool-11/bin/pgpool -f
>> /etc/pgpool-II-11/pgpool.conf -m fast stop (code=killed, signal=INT) Main PID: 21255 (pgpool)
>> Tasks: 3 (limit: 1024)
>> CGroup: /system.slice/pgpool-II-11.service
>> ├─21255 /usr/pgpool-11/bin/pgpool -f /etc/pgpool-II-11/pgpool.conf -n -D
>> ├─22359 pgpool: health check process(0)
>> └─22360 pgpool: health check process(1)
>>
>> May 10 23:36:38 pgs00000161 pgpool[21255]: [2906-1] 2021-05-10
>> 23:36:38: pid 21255: LOG: fork a new child process with pid: 14222
>> May 10 23:36:38 pgs00000161 pgpool[21255]: [2907-1] 2021-05-10
>> 23:36:38: pid 21255: LOG: child process with pid: 7431 exits wi...tus
>> 256 May 10 23:36:38 pgs00000161 pgpool[21255]: [2908-1] 2021-05-10
>> 23:36:38: pid 21255: LOG: fork a new child process with pid: 14223
>> May 10 23:36:38 pgs00000161 pgpool[21255]: [2909-1] 2021-05-10
>> 23:36:38: pid 21255: LOG: child process with pid: 8142 exits wi...tus
>> 256 May 10 23:36:38 pgs00000161 pgpool[21255]: [2910-1] 2021-05-10
>> 23:36:38: pid 21255: LOG: fork a new child process with pid: 14224
>> May 10 23:36:38 pgs00000161 pgpool[21255]: [2911-1] 2021-05-10
>> 23:36:38: pid 21255: LOG: child process with pid: 9498 exits wi...tus
>> 256 May 10 23:36:38 pgs00000161 pgpool[21255]: [2912-1] 2021-05-10
>> 23:36:38: pid 21255: LOG: fork a new child process with pid: 14225
>> May 10 23:36:38 pgs00000161 pgpool[21255]: [2913-1] 2021-05-10
>> 23:36:38: pid 21255: LOG: child process with pid: 4782 exits wi...tus
>> 256 May 10 23:36:38 pgs00000161 pgpool[21255]: [2914-1] 2021-05-10
>> 23:36:38: pid 21255: FATAL: failed to fork a child May 10 23:36:38
>> pgs00000161 pgpool[21255]: [2914-2] 2021-05-10 23:36:38: pid 21255:
>> DETAIL: system call fork() failed with rea...ailable
>> Hint: Some lines were ellipsized, use -l to show in full.
>>
>> It looks like processes responsible for client connections disappear suddenly and new instances cannot be created but restart of pgpool service helps and it works well again some time.
>> I have recommendation to set TasksMax to "infinity" or value much higher than num_init_children*max_pool, but i will be able to verify it during service window in next six hours.
>
> Another possibility is number of file descriptors. On my Lunux box after startinf 3-node Pgpool-II with 3-node PostgreSQL, I see:
>
> $ cat /proc/sys/fs/file-nr
> 21700 0 1612809
>
> So 21700 file descriptors out of 1612809 are used. Do you see number of the used fds are closing to the max file descriptor on your system?
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en.php&data=04%7C01%7CAndrzej.Rozmus%40asseco.pl%7C0754b21105be4cfce5b708d915d6fe44%7C88152bdecfa34a5cb981a785c624bb42%7C0%7C0%7C637564835400381362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hgfINGc6Awc3WqDuH9JBi1mMYQfZTxQIb2y3lB7F7mo%3D&reserved=0
> Japanese:https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sraoss.co.jp%2F&data=04%7C01%7CAndrzej.Rozmus%40asseco.pl%7C0754b21105be4cfce5b708d915d6fe44%7C88152bdecfa34a5cb981a785c624bb42%7C0%7C0%7C637564835400381362%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=GKoSEnL3H8%2BXlpeDteC352q2LVgc4dr6KGsAbgJbkuA%3D&reserved=0
More information about the pgpool-general
mailing list