[pgpool-general: 4850] Re: Problem shutting down interfaces
Muhammad Usama
m.usama at gmail.com
Wed Aug 3 05:15:43 JST 2016
Hi Guptesh
I am looking into the issue. meanwhile I had recently committed a fix
related to watchdog heartbeat problem, So could you also try building from
the latest source code to verify if you still face the issue. And if the
issue still presists please share the complete pgpool-II log file with
debug enabled. That would help to solve the problem more quickly.
Kind regards
Muhammad Usama
On Tue, Aug 2, 2016 at 9:25 PM, Guptesh Kumar <guptesh.cg4 at gmail.com> wrote:
> Hi Usama,
>
> Please let me know if I can provide any more input which can help you
> debug/reproduce this issue.
> We are considering pgpool(with watchdog) as a high availability solution
> for DB. And this issue is kind of show stopper because gracefully handling
> server crash/network problems is key in a Highly Available solution.
>
> I think this issue is same as Bug
> http://www.pgpool.net/mantisbt/view.php?id=227 .
>
> Looking forward for a solution of this issue.
>
> Thanks & Regards,
> Guptesh
>
> On Sat, Jul 30, 2016 at 12:17 AM, Guptesh Kumar <guptesh.cg4 at gmail.com>
> wrote:
>
>> Hi Usama,
>>
>>
>> I am still facing issues here-
>>
>> I shutdown primary(Node0) and standby(Node1) became primary successfully.
>> When Node0 came up, i recovered postgres0(postgres on Node0). Now Node0
>> is standby and Node1 is primary.
>> Again if I shutdown Node1(now primary), failover is not happening on
>> Node0. Below are the logs after health_check retry count is expired
>>
>>
>>
>> Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4211: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4185: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:02 ]: 2016-07-29 23:50:02: pid 4185: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4312: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4312: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4279: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4279: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4257: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4257: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4230: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4230: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4210: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4210: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4278: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:04 ]: 2016-07-29 23:50:04: pid 4278: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4304: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4304: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4227: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4227: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4276: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4276: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4261: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4261: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4297: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4297: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4204: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4204: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4295: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4295: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4291: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4291: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4228: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:06 ]: 2016-07-29 23:50:06: pid 4228: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4211: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4211: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4185: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:07 ]: 2016-07-29 23:50:07: pid 4185: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4228: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4291: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4261: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4297: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4295: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4227: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4276: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4228: LOG: received
>> degenerate backend request for node_id: 1 from pid [4228]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4261: LOG: received
>> degenerate backend request for node_id: 1 from pid [4261]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4297: LOG: received
>> degenerate backend request for node_id: 1 from pid [4297]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4291: LOG: received
>> degenerate backend request for node_id: 1 from pid [4291]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4295: LOG: received
>> degenerate backend request for node_id: 1 from pid [4295]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4227: LOG: received
>> degenerate backend request for node_id: 1 from pid [4227]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4276: LOG: received
>> degenerate backend request for node_id: 1 from pid [4276]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4204: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4204: LOG: received
>> degenerate backend request for node_id: 1 from pid [4204]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4304: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4304: LOG: received
>> degenerate backend request for node_id: 1 from pid [4304]
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:08 ]: 2016-07-29 23:50:08: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4257: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4257: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4279: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4279: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4230: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4230: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4210: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4210: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4312: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4312: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4278: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4278: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4329: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4217: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4266: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4269: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4289: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4250: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:09 ]: 2016-07-29 23:50:09: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4185: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4185: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4211: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:12 ]: 2016-07-29 23:50:12: pid 4211: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4230: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4230: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4312: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4312: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4279: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4279: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4257: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4257: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4210: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4210: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4278: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:14 ]: 2016-07-29 23:50:14: pid 4278: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4210: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4230: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4312: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4279: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4278: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4312: LOG: received
>> degenerate backend request for node_id: 1 from pid [4312]
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4230: LOG: received
>> degenerate backend request for node_id: 1 from pid [4230]
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4210: LOG: received
>> degenerate backend request for node_id: 1 from pid [4210]
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4279: LOG: received
>> degenerate backend request for node_id: 1 from pid [4279]
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4278: LOG: received
>> degenerate backend request for node_id: 1 from pid [4278]
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4257: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4257: LOG: received
>> degenerate backend request for node_id: 1 from pid [4257]
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:16 ]: 2016-07-29 23:50:16: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4185: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4185: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4211: LOG: trying connecting
>> to PostgreSQL server on "192.168.160.5:5432" by INET socket
>> Jul 29 23:50:17 ]: 2016-07-29 23:50:17: pid 4211: DETAIL: timed out.
>> retrying...
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4295: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4227: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4304: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4228: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:18 ]: 2016-07-29 23:50:18: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4185: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4211: LOG: failed to connect
>> to PostgreSQL server on "192.168.160.5:5432", getsockopt() detected
>> error "Connection timed out"
>> Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4185: LOG: received
>> degenerate backend request for node_id: 1 from pid [4185]
>> Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4211: LOG: received
>> degenerate backend request for node_id: 1 from pid [4211]
>> Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:19 ]: 2016-07-29 23:50:19: pid 4179: LOG: new IPC
>> connection received
>> Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4279: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4279: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4278: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4278: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:25 ]: 2016-07-29 23:50:25: pid 4312: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4278: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4278: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4279: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4279: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4210: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4257: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4230: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:26 ]: 2016-07-29 23:50:25: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: WARNING: degenerate
>> backend set command failed
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: DETAIL: ipc command
>> timeout
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4185: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: FATAL: failed to
>> create a backend connection
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4211: DETAIL: executing
>> failover on backend
>> Jul 29 23:50:29 ]: 2016-07-29 23:50:29: pid 4179: LOG: read from socket
>> failed, remote end closed the connection
>> Jul 29 23:51:13 ]: 2016-07-29 23:51:13: pid 4179: LOG: read from socket
>> failed with error :"Connection reset by peer"
>> Jul 29 23:51:19 ]: 2016-07-29 23:51:19: pid 4179: LOG: read from socket
>> failed with error :"Connection reset by peer"
>>
>>
>> Thanks & Regards,
>> Guptesh
>>
>> On Thu, Jun 16, 2016 at 6:57 PM, Guptesh Kumar <guptesh.cg4 at gmail.com>
>> wrote:
>>
>>> Hi Usama,
>>>
>>> I built from V3_5_STABLE head and tested. The problem is fixed :)
>>>
>>>
>>> Thanks & Regards,
>>> Guptesh
>>>
>>> On Thu, Jun 16, 2016 at 1:14 AM, Muhammad Usama <m.usama at gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> I have fixed the similar problem in pgpool-II 3.5 a few days back and
>>>> it will be part of next point release, which is expected in a few days time
>>>>
>>>>
>>>> http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=6ae42e74d48391b8cfaa97927e00bd65aec68fe8
>>>>
>>>> You can try if building from the V3_5_STABLE head fixes the problem.
>>>>
>>>> Regards
>>>> Muhammad Usama
>>>>
>>>>
>>>> On Wed, Jun 15, 2016 at 11:24 PM, Guptesh Kumar <guptesh.cg4 at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Ricardo,
>>>>>
>>>>> I am facing a similar issue -
>>>>>
>>>>> Node 1 : Master pgpool, Master Postgresql
>>>>> Node 2 : Slave Pgpool, Slave Postgresql
>>>>>
>>>>> When I pull out the power cable of master node(Node 1), the Slave
>>>>> watchdog promotes itself to Master correctly. But the pgpool in NOW master
>>>>> node(Node 2) neither disconnects the Node 1 Postgresql from the pool nor
>>>>> it promotes the Node 2 postgresql as Master(failover does not happen). All
>>>>> my DB connection hangs.
>>>>>
>>>>> I tried various values for health_check parameters but none of them
>>>>> helped.
>>>>>
>>>>> Softwares Used are -
>>>>> pgpool-II-pg94-3.5.2-1pgdg.rhel7.x86_64
>>>>> postgresql94-server-9.4.7-1PGDG.rhel7.x86_64
>>>>>
>>>>> Thanks & Regards,
>>>>> Guptesh
>>>>>
>>>>>
>>>>> On Tue, May 10, 2016 at 11:36 PM, Ricardo Larrañaga <
>>>>> ricardo.larranaga at gmail.com> wrote:
>>>>>
>>>>>> Hi guys:
>>>>>> I have been working on this a little bit, and i encounter this
>>>>>> problem (running pgpool 3.5.2 on a 2 node cluster with streaming
>>>>>> replication, if i shut down the interface of the slave, pgpool on the
>>>>>> master hangs the connections). And i realize that the problem only happens
>>>>>> when i shut down the interface. If i either shut down the processes or use
>>>>>> the firewall to block backend communication, pgpool reacts properly.
>>>>>>
>>>>>> Does anyone have experience with this or have a 2 node cluster with
>>>>>> streaming replication to test it. I am trying to identify what makes the
>>>>>> setup hang, since shutting down the interface is the best way i have to
>>>>>> simulate a node crash. When i shut down the interface, both postgres and
>>>>>> pgpool loose connection, but this is a valid scenario that could happen in
>>>>>> real life (when the whole server crashes).
>>>>>> Thanks a lot.
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>> On Thu, May 5, 2016 at 4:55 PM, Ricardo Larrañaga <
>>>>>> ricardo.larranaga at gmail.com> wrote:
>>>>>>
>>>>>>> Hello guys:
>>>>>>> I am running pgpool 3.5.2 in a 2 node cluster with postgresql-9.5 in
>>>>>>> master-slave and streaming replication mode.
>>>>>>>
>>>>>>> I have been testing failover and failback for a while. While doing
>>>>>>> failover by shutting down the processes, everything looks fine.
>>>>>>>
>>>>>>>
>>>>>>> The one test that is failing is when i shutdown the interface of one
>>>>>>> node.
>>>>>>> Right now i am shutting down the interface of the slave node (both
>>>>>>> pgool slave and postgresql slave).
>>>>>>>
>>>>>>> The problem i am running into is that after doing this, all
>>>>>>> connections to my database (through pgpool) hang. i am testing it with
>>>>>>> psql, and psql just hangs and does not give me an output. When i bring the
>>>>>>> interface of the slave back up, and try to connect to psql again, it looks
>>>>>>> like pgpool never marked the postgresql node as disconnected.
>>>>>>>
>>>>>>> I tried both with and without health check, and also tried with
>>>>>>> different values of health_check_timeout. my connect timeout value is
>>>>>>> default (10 seconds).
>>>>>>>
>>>>>>> Any one has encountered this issue? I just dont see pgpool
>>>>>>> attempting to do any failover. Pgpool is still running though, i can see
>>>>>>> the logs still comming. I just never see an error.
>>>>>>>
>>>>>>> I am NOT using interface monitoring, and i would prefer not to use
>>>>>>> it.
>>>>>>>
>>>>>>> Any pointers on how could i troubleshoot this?
>>>>>>> Thanks.
>>>>>>> Regards
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> pgpool-general mailing list
>>>>>> pgpool-general at pgpool.net
>>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> pgpool-general mailing list
>>>>> pgpool-general at pgpool.net
>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20160803/5de16b00/attachment-0001.html>
More information about the pgpool-general
mailing list