[pgpool-general: 7961] all backend nodes are down, pgpool requires at least one valid node

[martin.quevedo at temiandu.com]

Hi Tatsuo

 

I am having this problem where all of a sudden I get this error

 

all backend nodes are down, pgpool requires at least one valid node

 

after restarting pgpool it all looks good again.

 

 

2021-12-24 08:01:01: pid 4895: LOG:  child process with pid: 67325 exits
with status 256

2021-12-24 08:01:01: pid 4895: LOCATION:  pgpool_main.c:2415

2021-12-24 08:01:01: pid 4895: LOG:  fork a new child process with pid:
17150

2021-12-24 08:01:01: pid 4895: LOCATION:  pgpool_main.c:2545

2021-12-24 08:29:38: pid 67238: FATAL:  pgpool is not accepting any new
connections

2021-12-24 08:29:38: pid 67238: DETAIL:  all backend nodes are down, pgpool
requires at least one valid node

2021-12-24 08:29:38: pid 67238: HINT:  repair the backend nodes and restart
pgpool

2021-12-24 08:29:38: pid 67238: LOCATION:  child.c:1795

2021-12-24 08:29:38: pid 4895: LOG:  child process with pid: 67238 exits
with status 256

2021-12-24 08:29:38: pid 4895: LOCATION:  pgpool_main.c:2415

2021-12-24 08:29:38: pid 4895: LOG:  fork a new child process with pid:
58429

2021-12-24 08:29:38: pid 4895: LOCATION:  pgpool_main.c:2545

2021-12-24 08:29:50: pid 4895: LOG:  shutting down

2021-12-24 08:29:50: pid 4895: LOCATION:  pgpool_main.c:1152

2021-12-24 08:29:50: pid 4895: LOG:  terminating all child processes

2021-12-24 08:29:50: pid 4895: LOCATION:  pgpool_main.c:1161

2021-12-24 08:29:50: pid 4895: LOG:  Pgpool-II system is shutdown

2021-12-24 08:29:50: pid 4895: LOCATION:  pgpool_main.c:1189

2021-12-24 08:29:50: pid 4895: WARNING:  All the DB nodes are in down status
and skip writing status file.

2021-12-24 08:29:50: pid 4895: LOCATION:  pgpool_main.c:3815

2021-12-24 08:29:51: pid 58836: LOG:  health_check_stats_shared_memory_size:
requested size: 12288

2021-12-24 08:29:51: pid 58836: LOCATION:  health_check.c:546

2021-12-24 08:29:51: pid 58836: LOG:  memory cache initialized

2021-12-24 08:29:51: pid 58836: DETAIL:  memcache blocks :64

2021-12-24 08:29:51: pid 58836: LOCATION:  pool_memqcache.c:2061

2021-12-24 08:29:51: pid 58836: LOG:  allocating (144847936) bytes of shared
memory segment

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:3503

2021-12-24 08:29:51: pid 58836: LOG:  allocating shared memory segment of
size: 144847936

2021-12-24 08:29:51: pid 58836: LOCATION:  pool_shmem.c:61

2021-12-24 08:29:51: pid 58836: LOG:  health_check_stats_shared_memory_size:
requested size: 12288

2021-12-24 08:29:51: pid 58836: LOCATION:  health_check.c:546

2021-12-24 08:29:51: pid 58836: LOG:  health_check_stats_shared_memory_size:
requested size: 12288

2021-12-24 08:29:51: pid 58836: LOCATION:  health_check.c:546

2021-12-24 08:29:51: pid 58836: LOG:  memory cache initialized

2021-12-24 08:29:51: pid 58836: DETAIL:  memcache blocks :64

2021-12-24 08:29:51: pid 58836: LOCATION:  pool_memqcache.c:2061

2021-12-24 08:29:51: pid 58836: LOG:  pool_discard_oid_maps: discarded
memqcache oid maps

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:3584

2021-12-24 08:29:51: pid 58836: LOG:  Setting up socket for 0.0.0.0:5432

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:813

2021-12-24 08:29:51: pid 58836: LOG:  Setting up socket for :::5432

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:813

2021-12-24 08:29:51: pid 58836: LOG:  find_primary_node_repeatedly: waiting
for finding a primary node

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:3375 

2021-12-24 08:29:51: pid 58836: LOG:  find_primary_node: primary node is 0

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:3295

2021-12-24 08:29:51: pid 58836: LOG:  find_primary_node: standby node is 1

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:3301

2021-12-24 08:29:51: pid 58903: LOG:  PCP process: 58903 started

2021-12-24 08:29:51: pid 58903: LOCATION:  pcp_child.c:162

2021-12-24 08:29:51: pid 58904: LOG:  process started

2021-12-24 08:29:51: pid 58904: LOCATION:  pgpool_main.c:729

2021-12-24 08:29:51: pid 58905: LOG:  process started

2021-12-24 08:29:51: pid 58905: LOCATION:  pgpool_main.c:729

2021-12-24 08:29:51: pid 58906: LOG:  process started

2021-12-24 08:29:51: pid 58906: LOCATION:  pgpool_main.c:729

2021-12-24 08:29:51: pid 58836: LOG:  pgpool-II successfully started.
version 4.3.0 (tamahomeboshi)

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:490

2021-12-24 08:29:51: pid 58836: LOG:  node status[0]: 1

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:501

2021-12-24 08:29:51: pid 58836: LOG:  node status[1]: 2

2021-12-24 08:29:51: pid 58836: LOCATION:  pgpool_main.c:501

2021-12-24 08:29:54: pid 58901: LOG:  reading message length

2021-12-24 08:29:54: pid 58901: DETAIL:  message length (22) in slot 1 does
not match with slot 0(23)

2021-12-24 08:29:54: pid 58901: LOCATION:  pool_proto_modules.c:4312

2021-12-24 08:29:54: pid 58901: LOG:  pool_reuse_block: blockid: 0

2021-12-24 08:29:54: pid 58901: CONTEXT:  while searching system catalog,
When relcache is missed

2021-12-24 08:29:54: pid 58901: LOCATION:  pool_memqcache.c:2263

 

 

 

Could you give me some pointers where to look?

 

I have 9 servers connected to the same DB pair but only one pgpool at a time
will have this error. I mean if it was a problem of the backend DBs it would
happen to all of them, right?

 

Regards,

Martin 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20211224/a5bde06b/attachment.htm>