View Issue Details

IDProjectCategoryView StatusLast Update
0000721Pgpool-IIBugpublic2022-06-28 11:23
Reportershutao917 Assigned Topengbo  
PriorityimmediateSeveritymajorReproducibilityalways
Status closedResolutionopen 
OSCentOSOS Version 7.6.1810 
Product Version4.2.2 
Summary0000721: cpu load high frequently
Descriptionwe recently installed 4.2.2 in production env with parameter like below,about ervery 5 minutes cpu load get very high。we also use 4.0.5 for about 1 years with the same parameter but had no promble.

num_init_children = 1800
max_pool = 1
# - Life time -
child_life_time = 300
child_max_connections = 0
connection_life_time = 0
client_idle_limit = 0
TagsNo tags attached.

Activities

shutao917

2021-07-15 10:52

reporter  

image.png (173,562 bytes)   
image.png (173,562 bytes)   
cpu load.jpg (299,569 bytes)

pengbo

2021-07-15 14:23

developer   ~0003898

Could you share pgpool.conf?

I want to find which Process is causing the high CPU usage.
if it is possible, could you run the following command to check the CPU usage of each pgpool process?
  
    timeout 1800 sh -c "while :; do echo '' >> res.txt; date >> res.txt; ps auxwww | grep -e %CPU -e pgpool | grep -v grep | head -n 20 >> res.txt;sleep 60; done"

If every 5 minutes CPU load gets very high, are you running particular queries every 5 minutes?
Could you share the scenarios or queries issued by client to reproduce this issue?
If you enable "log_per_node_statement" parameter, the client queries will be output to the log file.

shutao917

2021-07-15 16:49

reporter   ~0003899

from the pgpool log file,i find two things:
1.a java promgram connection to pgpool and exec some pretty simple query,but not use connection pool。change it direct to postgres,and the promble sees to be impoved(see 1.jpg)
2.another promgram connection to an not exist database(see 2.jpg),we had not find which promgram it is yet.i think stop that,cpu load promble would be resolve
Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348979-1] 2021-07-15 14:47:29: pid 22205: LOG: new connection received
Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348979-2] 2021-07-15 14:47:29: pid 22205: DETAIL: connecting host=xxxxxxxxx port=57168
Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348980-1] 2021-07-15 14:47:29: pid 22205: LOG: backend throws an error message
Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348980-2] 2021-07-15 14:47:29: pid 22205: DETAIL: database "pre_bizcenter_hmc" does not exist
Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348981-1] 2021-07-15 14:47:29: pid 22205: ERROR: unable to forward message to frontend
Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348981-2] 2021-07-15 14:47:29: pid 22205: DETAIL: FATAL error occured on backend



so,i doubt is the child_life_time=300 leader this promble
1.jpg (119,418 bytes)   
1.jpg (119,418 bytes)   
2.jpg (443,959 bytes)

pengbo

2021-07-21 12:56

developer   ~0003903

I am not sure if it is caused by Pgpool-II.

> 2.another promgram connection to an not exist database(see 2.jpg),we had not find which promgram it is yet.i think stop that,cpu load promble would be resolve
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348979-1] 2021-07-15 14:47:29: pid 22205: LOG: new connection received
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348979-2] 2021-07-15 14:47:29: pid 22205: DETAIL: connecting host=xxxxxxxxx port=57168
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348980-1] 2021-07-15 14:47:29: pid 22205: LOG: backend throws an error message
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348980-2] 2021-07-15 14:47:29: pid 22205: DETAIL: database "pre_bizcenter_hmc" does not exist
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348981-1] 2021-07-15 14:47:29: pid 22205: ERROR: unable to forward message to frontend
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348981-2] 2021-07-15 14:47:29: pid 22205: DETAIL: FATAL error occured on backend

Does the ERROR above occur frequently?

shutao917

2021-07-21 20:17

reporter   ~0003904

> 2.another promgram connection to an not exist database(see 2.jpg),we had not find which promgram it is yet.i think stop that,cpu load promble would be resolve
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348979-1] 2021-07-15 14:47:29: pid 22205: LOG: new connection received
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348979-2] 2021-07-15 14:47:29: pid 22205: DETAIL: connecting host=xxxxxxxxx port=57168
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348980-1] 2021-07-15 14:47:29: pid 22205: LOG: backend throws an error message
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348980-2] 2021-07-15 14:47:29: pid 22205: DETAIL: database "pre_bizcenter_hmc" does not exist
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348981-1] 2021-07-15 14:47:29: pid 22205: ERROR: unable to forward message to frontend
> Jul 15 14:47:29 pg62 pgpool-prod-biz[22205]: [3348981-2] 2021-07-15 14:47:29: pid 22205: DETAIL: FATAL error occured on backend

This is a flink job。when occur,up to 200/s

pengbo

2021-07-27 15:37

developer   ~0003905

Sorry. I am not sure if it's the cause of pgpool because I can't reproduce this issue.
I think if you fix the program which is executing queries to database "pre_bizcenter_hmc", this issue might be resolved.

pengbo

2022-06-28 11:23

developer   ~0004075

No response over two weeks.
Close issue.

Issue History

Date Modified Username Field Change
2021-07-15 10:52 shutao917 New Issue
2021-07-15 10:52 shutao917 File Added: image.png
2021-07-15 10:52 shutao917 File Added: cpu load.jpg
2021-07-15 12:15 pengbo Assigned To => pengbo
2021-07-15 12:15 pengbo Status new => assigned
2021-07-15 12:15 pengbo Description Updated
2021-07-15 14:23 pengbo Note Added: 0003898
2021-07-15 14:23 pengbo Status assigned => feedback
2021-07-15 16:49 shutao917 Note Added: 0003899
2021-07-15 16:49 shutao917 File Added: 1.jpg
2021-07-15 16:49 shutao917 File Added: 2.jpg
2021-07-15 16:49 shutao917 Status feedback => assigned
2021-07-21 12:56 pengbo Note Added: 0003903
2021-07-21 12:56 pengbo Status assigned => feedback
2021-07-21 20:17 shutao917 Note Added: 0003904
2021-07-21 20:17 shutao917 Status feedback => assigned
2021-07-27 15:37 pengbo Note Added: 0003905
2021-07-30 15:01 pengbo Status assigned => feedback
2022-06-28 11:23 pengbo Note Added: 0004075
2022-06-28 11:23 pengbo Status feedback => closed