<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:tax="http://schemas.microsoft.com/sharepoint/taxonomy/soap/" xmlns:tns="http://schemas.microsoft.com/sharepoint/soap/recordsrepository/" xmlns:spsup="http://microsoft.com/webservices/SharePointPortalServer/UserProfileService" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoPlainText>Hi Tatsuo<o:p></o:p></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'>I am having this problem where all of a sudden I get this error<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'>all backend nodes are down, pgpool requires at least one valid node<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'>after restarting pgpool it all looks good again.<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:01:01: pid 4895: LOG: child process with pid: 67325 exits with status 256<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:01:01: pid 4895: LOCATION: pgpool_main.c:2415<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:01:01: pid 4895: LOG: fork a new child process with pid: 17150<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:01:01: pid 4895: LOCATION: pgpool_main.c:2545<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:38: pid 67238: FATAL: pgpool is not accepting any new connections<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:38: pid 67238: DETAIL: </span><b><span style='color:red'>all backend nodes are down, pgpool requires at least one valid node<o:p></o:p></span></b></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:38: pid 67238: HINT: repair the backend nodes and restart pgpool<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:38: pid 67238: LOCATION: child.c:1795<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:38: pid 4895: LOG: child process with pid: 67238 exits with status 256<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:38: pid 4895: LOCATION: pgpool_main.c:2415<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:38: pid 4895: LOG: fork a new child process with pid: 58429<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:38: pid 4895: LOCATION: pgpool_main.c:2545<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:50: pid 4895: LOG: </span><b><span style='color:red'>shutting down<o:p></o:p></span></b></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:50: pid 4895: LOCATION: pgpool_main.c:1152<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:50: pid 4895: LOG: terminating all child processes<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:50: pid 4895: LOCATION: pgpool_main.c:1161<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:50: pid 4895: LOG: Pgpool-II system is shutdown<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:50: pid 4895: LOCATION: pgpool_main.c:1189<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:50: pid 4895: WARNING: All the DB nodes are in down status and skip writing status file.<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:50: pid 4895: LOCATION: pgpool_main.c:3815<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: health_check_stats_shared_memory_size: requested size: 12288<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: health_check.c:546<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: memory cache initialized<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: DETAIL: memcache blocks :64<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pool_memqcache.c:2061<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: allocating (144847936) bytes of shared memory segment<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:3503<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: allocating shared memory segment of size: 144847936<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pool_shmem.c:61<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: health_check_stats_shared_memory_size: requested size: 12288<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: health_check.c:546<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: health_check_stats_shared_memory_size: requested size: 12288<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: health_check.c:546<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: memory cache initialized<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: DETAIL: memcache blocks :64<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pool_memqcache.c:2061<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: pool_discard_oid_maps: discarded memqcache oid maps<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:3584<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: Setting up socket for 0.0.0.0:5432<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:813<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: Setting up socket for :::5432<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:813<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: find_primary_node_repeatedly: waiting for finding a primary node<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:3375</span> <o:p></o:p></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: find_primary_node: </span><b><span style='color:red'>primary node is 0</span></b><span style='color:black'><o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:3295<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: find_primary_node: </span><b><span style='color:red'>standby node is 1</span></b><span style='color:black'><o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:3301<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58903: LOG: PCP process: 58903 started<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58903: LOCATION: pcp_child.c:162<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58904: LOG: process started<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58904: LOCATION: pgpool_main.c:729<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58905: LOG: process started<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58905: LOCATION: pgpool_main.c:729<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58906: LOG: process started<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58906: LOCATION: pgpool_main.c:729<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: pgpool-II successfully started. version 4.3.0 (tamahomeboshi)<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:490<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: node status[0]: 1<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:501<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOG: node status[1]: 2<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:51: pid 58836: LOCATION: pgpool_main.c:501<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:54: pid 58901: LOG: reading message length<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:54: pid 58901: DETAIL: message length (22) in slot 1 does not match with slot 0(23)<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:54: pid 58901: LOCATION: pool_proto_modules.c:4312<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:54: pid 58901: LOG: pool_reuse_block: blockid: 0<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:54: pid 58901: CONTEXT: while searching system catalog, When relcache is missed<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>2021-12-24 08:29:54: pid 58901: LOCATION: pool_memqcache.c:2263<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'>Could you give me some pointers where to look?<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'>I have 9 servers connected to the same DB pair but only one pgpool at a time will have this error. I mean if it was a problem of the backend DBs it would happen to all of them, right?<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:black'>Regards,<o:p></o:p></span></p><p class=MsoPlainText><span style='color:black'>Martin <o:p></o:p></span></p></div></body></html>