View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000336 | Pgpool-II | Bug | public | 2017-08-29 03:20 | 2017-11-13 18:02 |
| Reporter | jeeads | Assigned To | t-ishii | ||
| Priority | high | Severity | crash | Reproducibility | sometimes |
| Status | closed | Resolution | open | ||
| Platform | Linux | OS | Red Hat Enterprise Linux Server | OS Version | release 7.3 (Mai |
| Product Version | 3.6.2 | ||||
| Summary | 0000336: During load the server in production would fail connectivity with a slave and itself, resulting in removing nodes | ||||
| Description | Would get errors like this there was only node1/2. Pgpool was set on both Node1/2. e12:21:22 2017-08-28 12:21:22: pid 35076:LOG: failed to connect to PostgreSQL server on "node1.prod.x.org:5601" with error "Cannot assign requested address" e12:21:51 2017-08-28 12:21:51: pid 44700:LOG: failed to connect to PostgreSQL server on "node2.prod.x.org:5601" with error "Cannot assign requested address" Resulting in all future connections being routed to a readonly node or being unable to handle any traffic. | ||||
| Steps To Reproduce | Seems to be related to load as we where running without hardly any issues in all the other environments outside of prod. We just migrated to this on Saturday (low usage), then Monday during peak is when we had the issues. | ||||
| Tags | No tags attached. | ||||
|
|
|
|
|
Was able to reproduce once in another environment. This is in dmesg [2712161.110044] pgpool[55679]: segfault at 10 ip 000000000042e34d sp 00007fffbaed29a0 error 4 in pgpool[400000+139000] [2712161.111048] pgpool[56272]: segfault at 10 ip 0000000000426912 sp 00007fffbaed2c80 error 4 in pgpool[400000+139000] [2712161.111188] pgpool[56256]: segfault at 10 ip 00000000004309db sp 00007fffbaed2d40 error 4 in pgpool[400000+139000] [2712161.921487] pgpool[55754]: segfault at 0 ip 00000000004252d6 sp 00007fffbaed2390 error 4 in pgpool[400000+139000] [2712163.689768] pgpool[55985]: segfault at 10 ip 000000000042e34d sp 00007fffbaed29a0 error 4 in pgpool[400000+139000] [2712166.355798] pgpool[56108]: segfault at 10 ip 000000000042e34d sp 00007fffbaed29a0 error 4 in pgpool[400000+139000] [2712168.840183] pgpool[55776]: segfault at 10 ip 000000000042e34d sp 00007fffbaed29a0 error 4 in pgpool[400000+139000] [2712170.911507] pgpool[56366]: segfault at 10 ip 000000000042e34d sp 00007fffbaed29a0 error 4 in pgpool[400000+139000] |
|
|
"Cannot assign requested address" usually means Pgpool-II runs out local port. Probably you will find tremendous number of "TIME WAIT" state ports. Especially it tends to happen if Pgpool-II every time accepts requests from client it creates tcp connection to PostgreSQL because you turn off connection_cache. I recommend you to turn on connection_cache. In the mean time segfault error needs to be cared. Your Pgpool-II version is a little bit outdated. The latest one is 3.6.6. Between those versions some segfault errors were fixed. It may be worth to try the latest version 3.6.6. |
|
|
The problem with turning on the connection_cache is that we run out of connections. I had found a couple of kernel parameters related to that , but was trying to reproduce first. |
|
|
I think there' s nothing Pgpool-II can do for this issue merely needs adjusting kernel parameters. May close this issue? |
|
|
Couldn't pgpool keep sessions open to the database to reuse (like pgbouncer), rather than re-using sockets ? I have tried setting connection_cache =on, but this didn't help. |
|
|
I don't understand what you mean. How can we reuse sessions to the database without keeping the sockets connecting to the database? |
|
|
I mean from the database point of view. I do a select count(*) from pg_stat_activity and I don't see any sessions that are not active. Whereas if I use pgbouncer it will see a count that matches the configuration after the sessions have been used. |
|
|
Can we close this item? This forum is for solving bugs of Pgpool-II. If you need further discussions on Pgpool-II design, Please use our mailing list. |
|
|
You can close but you may want to note cause of issues in your product requirements. The original issue that this ticket was for was for Kernel parameters not being correct. In particular : net.ipv4.tcp_max_tw_buckets = 8192 net.ipv4.tcp_max_orphans = 8192 |
|
|
You mean we have: net.ipv4.tcp_max_tw_buckets = 8192 net.ipv4.tcp_max_orphans = 8192 somewhere in our documents? |
|
|
If you do not have further comments, I am going close this issue. |
|
|
Yes, you should have some comment about that in the documents along with other kernel recommendations. You can close this ticket. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2017-08-29 03:20 | jeeads | New Issue | |
| 2017-08-29 03:20 | jeeads | File Added: pgpool.conf | |
| 2017-08-31 04:54 | jeeads | Note Added: 0001695 | |
| 2017-09-06 17:15 | t-ishii | Assigned To | => t-ishii |
| 2017-09-06 17:15 | t-ishii | Status | new => assigned |
| 2017-09-06 17:15 | t-ishii | Description Updated | |
| 2017-09-06 17:26 | t-ishii | Note Added: 0001700 | |
| 2017-09-06 17:27 | t-ishii | Status | assigned => feedback |
| 2017-09-07 03:00 | jeeads | Note Added: 0001701 | |
| 2017-09-07 03:00 | jeeads | Status | feedback => assigned |
| 2017-09-13 14:41 | t-ishii | Note Added: 0001707 | |
| 2017-09-13 14:42 | t-ishii | Status | assigned => feedback |
| 2017-09-13 23:07 | jeeads | Note Added: 0001709 | |
| 2017-09-13 23:07 | jeeads | Status | feedback => assigned |
| 2017-09-13 23:19 | t-ishii | Note Added: 0001710 | |
| 2017-09-16 02:37 | jeeads | Note Added: 0001711 | |
| 2017-10-10 10:17 | t-ishii | Note Added: 0001748 | |
| 2017-10-10 10:18 | t-ishii | Status | assigned => feedback |
| 2017-10-11 00:00 | jeeads | Note Added: 0001754 | |
| 2017-10-11 00:00 | jeeads | Status | feedback => assigned |
| 2017-10-11 07:44 | t-ishii | Note Added: 0001755 | |
| 2017-10-17 14:03 | t-ishii | Status | assigned => feedback |
| 2017-10-25 13:01 | t-ishii | Note Added: 0001785 | |
| 2017-10-25 22:57 | jeeads | Note Added: 0001786 | |
| 2017-10-25 22:57 | jeeads | Status | feedback => assigned |
| 2017-11-13 18:02 | t-ishii | Status | assigned => closed |
