<div dir="ltr"><div><br></div><div>Hi,</div><div>We've made some progress here and I'd like to get your feedback regarding the fix/workaround. Basically, after some research (incl. google/chatgpt etc), we came across this information:</div><div><br></div><div><h4>🧠 1. <code>systemd-logind</code> manages user sessions</h4>
<p>In Ubuntu 24.04 (and since ~15.04), systemd manages user sessions. When a user session ends (e.g. when a background process or login shell finishes), the following happens:</p>
<ul><li>
<p>The session scope (<code>session-XXXX.scope</code>) is stopped.</p>
</li><li>
<p>The user manager (<code>user@UID.service</code>) is also stopped <strong>unless lingering is enabled</strong>.</p>
</li></ul>
<h4>🧨 If no processes remain and the user is not lingering:</h4><ul><li>
<p>Shared memory and <strong>System V IPC resources</strong> (like semaphores) <strong>may be destroyed</strong> automatically if no one is using them.</p>
</li><li>
<p>If a process (like <code>pgpool2</code> or <code>PostgreSQL</code>) relied on those semaphores, you may later see:</p><pre class="gmail-overflow-visible!"><div class="gmail-contain-inline-size gmail-rounded-md gmail-border-[0.5px] gmail-border-token-border-medium gmail-relative gmail-bg-token-sidebar-surface-primary"><div class="gmail-overflow-y-auto gmail-p-4" dir="ltr"><code class="gmail-whitespace-pre!"><span><span>failed </span><span><span class="gmail-hljs-keyword">to</span></span><span> </span><span><span class="gmail-hljs-keyword">lock</span></span><span> semaphore</span></span></code></div></div></pre></li></ul></div><div>This is a strong sign that the user session ended and IPC resources (like semaphores) were destroyed.</div><div><br></div><div>This does seem to be happening here, as I see these in /var/log/syslog around that time we discussed in last posting, just prior to semaphore errors showing in pgpool logs:</div><div>2025-06-02T12:45:17.173486+02:00 db-replica3 systemd[1]: user@3000.service: Deactivated successfully.<br>2025-06-02T12:45:17.173997+02:00 db-replica3 systemd[1]: Stopped user@3000.service - User Manager for UID 3000.<br>2025-06-02T12:45:17.240314+02:00 db-replica3 systemd[1]: Stopping user-runtime-dir@3000.service - User Runtime Directory /run/user/3000...<br>2025-06-02T12:45:17.248017+02:00 db-replica3 systemd[1]: run-user-3000.mount: Deactivated successfully.<br>2025-06-02T12:45:17.249107+02:00 db-replica3 systemd[1]: user-runtime-dir@3000.service: Deactivated successfully.<br>2025-06-02T12:45:17.249353+02:00 db-replica3 systemd[1]: Stopped user-runtime-dir@3000.service - User Runtime Directory /run/user/3000.<br>2025-06-02T12:45:17.251672+02:00 db-replica3 systemd[1]: Removed slice user-3000.slice - User Slice of UID 3000.<br>2025-06-02T12:45:23.581108+02:00 db-replica3 strace[1692927]: [pid 1693495] 12:45:23.580360 semtimedop(41, [{sem_num=6, sem_op=-1, sem_flg=SEM_UNDO}], 1, NULL) = -1 EINVAL (Invalid argument)</div><div><br></div><div>It also advises to add Type=forking to the [Service] section of the pgpool.service file, which we don't currently have, I think I posted it earlier, but just in case here is relevant part of our service file:</div><div><br>[Unit]<br>Description=Pgpool-II<br>After=syslog.target network.target postgresql.service<br>Wants=postgresql.service<br><br>[Service]<br>User=postgres<br>Group=postgres<br><br>EnvironmentFile=-/etc/default/pgpool2<br><br>ExecStart=/usr/sbin/pgpool -f /etc/pgpool2/pgpool.conf $OPTS<br>ExecStop=/usr/sbin/pgpool -f /etc/pgpool2/pgpool.conf $STOP_OPTS stop<br>ExecReload=/usr/sbin/pgpool -f /etc/pgpool2/pgpool.conf reload</div><div><br></div><div>-------------------</div><div>One bit I read that is different is the Type=forking and adding -n is required (as you can see above, we don't have that right now:</div><div>
<li><p><code>User=postgres</code>: Assumes <code>pgpool</code> should run as the <code>postgres</code> user. Adjust if different.</p>
</li><li>
<p><code>ExecStart=/usr/sbin/pgpool -n</code>: Use the <strong>correct path and arguments</strong> for your installation.</p></li><li>
<p><code>pgpool</code> must be configured to <strong>not daemonize</strong> (<code>-n</code> flag), or systemd will lose track of it.</p>
</li><li>
<p>You can add environment variables for config files, ports, etc.</p></li>-----------------------</div><div><br></div><div>Anyways, our sysadmins suggested changing the user postgres to have Linger option on:</div><div>$ sudo loginctl show-user postgres -p Linger<br>Linger=yes<br></div><div>Once we did that, the semaphore issue disappeared, it's been running for over an hour now and ipcs shows them:</div><div>$ ipcs -s<br><br>------ Semaphore Arrays --------<br>key semid owner perms nsems <br>0x00000000 6 postgres 600 8 </div><div><br></div><div><br></div><div>So, what do you think about this fix? I read that letting the user have linger on is not generally recommended, but that seems to be the only way to keep the semaphores from disappearing right now...</div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, Jun 3, 2025 at 2:30 AM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org">ishii@postgresql.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks for the strace log.<br>
<br>
1756952 16:45:14.448431 semtimedop(46, [{sem_num=6, sem_op=1, sem_flg=SEM_UNDO}], 1, NULL) = 0<br>
1756952 16:45:24.482757 semtimedop(46, [{sem_num=6, sem_op=-1, sem_flg=SEM_UNDO}], 1, NULL) = -1 EINVAL (Invalid argument)<br>
<br>
So the semaphore was removed between 16:45:14.448431 and<br>
16:45:24.482757. I was expecting "semctl(46, 0, IPC_RMID, NULL) = 0"<br>
(which removes semaphore) was executed there, but actually it was<br>
not. Nothing was recorded in the period.<br>
<br>
I think this definitely means that a process which is not a child of<br>
pgpool executed semctl(46, 0, IPC_RMID, NULL) (or ipcrm command). Do<br>
you have any idea such a program other than pgpool?<br>
<br>
> Here is the strace result, hope this helps in any way, it takes about 10<br>
> minutes for the semaphors to 'go away'...<br>
> <br>
> <br>
> On Mon, Jun 2, 2025 at 2:53 PM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>> wrote:<br>
> <br>
>> You have 4 pgpool nodes and you issue the queries through one of<br>
>> pgpools. Suppose you issued the queries through node 0, the "status"<br>
>> field is only updated on node 0. On other 3 nodes the status remains<br>
>> "waiting". However the status on all 4 nodes will be synched when the<br>
>> status becomes "down", or becomes "up" and the former status is<br>
>> "down".<br>
>><br>
>> > That's where I wonder if this is working properly, because there seem to<br>
>> be<br>
>> > clients/connections happening, I ran the backend status command this is<br>
>> > what it showed:<br>
>> ><br>
>> > node_id | hostname | port | status | role | select_cnt |<br>
>> > insert_cnt | update_cnt | delete_cnt | ddl_cnt | other_cnt | panic_cnt |<br>
>> > fatal_cnt | error_cnt<br>
>> ><br>
>> ><br>
>> ---------+------------------------+------+--------+---------+------------+------------+------------+------------+---------+-----------+-----------+-----------+-----------<br>
>> ><br>
>> > 0 | db-primarypriv | 5434 | up | primary | 7552 | 47<br>
>> > | 332 | 27 | 0 | 13678 | 0 | 0<br>
>> |<br>
>> > 6<br>
>> ><br>
>> > 1 | db-replica1priv | 5434 | up | standby | 6817 | 0<br>
>> > | 0 | 0 | 0 | 13682 | 0 | 0<br>
>> |<br>
>> > 0<br>
>> ><br>
>> > 2 | db-replica2priv | 5434 | up | standby | 6514 | 0<br>
>> > | 0 | 0 | 0 | 13693 | 0 | 0<br>
>> |<br>
>> > 1<br>
>> ><br>
>> > 3 | db-replica3priv | 5434 | up | standby | 6910 | 0<br>
>> > | 0 | 0 | 0 | 13679 | 0 | 0<br>
>> |<br>
>> > 1<br>
>> ><br>
>> > (4 rows)<br>
>> ><br>
>> > So plenty of queries supposedly happening on them, but the nodes still<br>
>> show<br>
>> > status 1. What gives?<br>
>> ><br>
>> ><br>
>> > On Mon, Jun 2, 2025 at 1:43 PM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> wrote:<br>
>> ><br>
>> >> > Hi, another thing I noticed - the status of all those nodes in<br>
>> >> > pcp_node_info seems to be wrong, for example:<br>
>> >> > $ pcp_node_info -h localhost -p 9898 -U pcpadmin --verbose -n 3<br>
>> >> > Hostname : db-replica2<br>
>> >> > Port : 5434<br>
>> >> > Status : 1<br>
>> >> > Weight : 0.250000<br>
>> >> > Status Name : waiting<br>
>> >> > Backend Status Name : up<br>
>> >> > Role : standby<br>
>> >> > Backend Role : standby<br>
>> >> > Replication Delay : 0<br>
>> >> > Replication State : streaming<br>
>> >> > Replication Sync State : async<br>
>> >> > Last Status Change : 2025-06-02 12:31:46<br>
>> >><br>
>> >> Are you saying this is wrong?<br>
>> >><br>
>> >> > Status : 1<br>
>> >><br>
>> >> Seems normal to me because it just says no client has connected to the<br>
>> >> pgpool server until now.<br>
>> >><br>
>> >> > Do you think this is because of the semaphore problem, or does it<br>
>> suggest<br>
>> >> > something else completely?<br>
>> >> ><br>
>> >> > On Mon, Jun 2, 2025 at 12:59 PM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> >> wrote:<br>
>> >> ><br>
>> >> >> > Hi, I changed the pgpool.service file for startup as you said,<br>
>> >> replaced<br>
>> >> >> my<br>
>> >> >> > start with this:<br>
>> >> >> > ExecStart=strace -f -tt --trace=ipc /usr/sbin/pgpool -f<br>
>> >> >> > /etc/pgpool2/pgpool.conf $OPTS<br>
>> >> >> > I'm attaching the log from replica again, but I'm not sure this<br>
>> >> actually<br>
>> >> >> > worked (as in, it doesn't look much different, to me at least).<br>
>> >> >><br>
>> >> >> Yeah. It seems there's strace output in the log.<br>
>> >> >> Can you try "-o" of strace, which instructs strace a file name to<br>
>> print<br>
>> >> >> the output. e.g.<br>
>> >> >><br>
>> >> >> ExecStart=strace -f -tt --trace=ipc -o /tmp/strace.out<br>
>> /usr/sbin/pgpool<br>
>> >> -f<br>
>> >> >> /etc/pgpool2/pgpool.conf $OPTS<br>
>> >> >><br>
>> >> >> > On Mon, Jun 2, 2025 at 11:47 AM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a><br>
>> ><br>
>> >> >> wrote:<br>
>> >> >> ><br>
>> >> >> >> > Hi,<br>
>> >> >> >> > I tried the strace as you shown but it didn't really output<br>
>> >> anything:<br>
>> >> >> >> ><br>
>> >> >> >> > $ sudo strace -p 1662563 -f -tt --trace=ipc<br>
>> >> >> >> > strace: Process 1662563 attached<br>
>> >> >> >> ><br>
>> >> >> >> > ^Cstrace: Process 1662563 detached. <--- this is me getting<br>
>> out<br>
>> >> of<br>
>> >> >> it,<br>
>> >> >> >> > since I saw in another window the semaphores were gone already,<br>
>> >> >> without<br>
>> >> >> >> > anything outputting here. Unless something's wrong with the<br>
>> >> syntax of<br>
>> >> >> >> the<br>
>> >> >> >> > command, it didn't really output anything. I changed the log<br>
>> line<br>
>> >> you<br>
>> >> >> >> > suggested as well.<br>
>> >> >> >> ><br>
>> >> >> >> > Also, FYI - I tried to downgrade to a lower (4.3.7) version as<br>
>> it<br>
>> >> was<br>
>> >> >> >> > available in my package repository, and the same semaphore<br>
>> version<br>
>> >> >> >> persists,<br>
>> >> >> >><br>
>> >> >> >> Okay. I think you need to start pgpool from strace. I mean,<br>
>> modifiy<br>
>> >> >> >> pgpool.service from:<br>
>> >> >> >> ExecStart=/usr/sbin/pgpool -f /etc/pgpool2/pgpool.conf $OPTS<br>
>> >> >> >> to:<br>
>> >> >> >> ExecStart=strace -f -tt --trace=ipc /usr/sbin/pgpool -f<br>
>> >> >> >> /etc/pgpool2/pgpool.conf $OPTS<br>
>> >> >> >><br>
>> >> >> >> > On Mon, Jun 2, 2025 at 2:40 AM Tatsuo Ishii <<br>
>> <a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> >> >> >> wrote:<br>
>> >> >> >> ><br>
>> >> >> >> >> > Hi,<br>
>> >> >> >> >> > Not sure I can patch the source, as it was installed via apt<br>
>> get<br>
>> >> >> and<br>
>> >> >> >> not<br>
>> >> >> >> >> > compiled from source. If that is doable to a package-built<br>
>> one<br>
>> >> and<br>
>> >> >> >> you<br>
>> >> >> >> >> can<br>
>> >> >> >> >> > give me some instructions, I can certainly try.<br>
>> >> >> >> >><br>
>> >> >> >> >> Unfortunately I'm not familiar with debian packaging.<br>
>> >> >> >> >><br>
>> >> >> >> >> > As for the pgpool log, I'm attaching it, hopefully it will go<br>
>> >> >> through,<br>
>> >> >> >> >> it's<br>
>> >> >> >> >> > about 5mb as I enabled higher debug options.<br>
>> >> >> >> >><br>
>> >> >> >> >> It was sent as a direct mail to me. Not sure who/what removed<br>
>> the<br>
>> >> >> >> >> semaphore. I observe following from the log anyway:<br>
>> >> >> >> >><br>
>> >> >> >> >> - pgpool started on 16:10:21. At that time the semaphore<br>
>> >> definitely<br>
>> >> >> >> >> existed.<br>
>> >> >> >> >> - Until 16:15:13, everything worked great.<br>
>> >> >> >> >> - On 16:15:23, the problem started.<br>
>> >> >> >> >><br>
>> >> >> >> >> So it seems the semaphore was removed by someone between<br>
>> 16:15:13<br>
>> >> and<br>
>> >> >> >> >> 16:15:23.<br>
>> >> >> >> >><br>
>> >> >> >> >> Unfortunately the removal of semaphore was not recorded in the<br>
>> >> log.<br>
>> >> >> To<br>
>> >> >> >> >> determine who removed the semaphore, can you please attach<br>
>> strace<br>
>> >> to<br>
>> >> >> >> >> pgpool main and share the log? This time debug log is not<br>
>> >> necessary.<br>
>> >> >> >> >><br>
>> >> >> >> >> - Find the pgpool parent process pid (in the debug log it was<br>
>> >> >> 1483510)<br>
>> >> >> >> >> - Become root (attaching strace usually requires root<br>
>> privilege)<br>
>> >> >> >> >> - Attach strace to the process $pid<br>
>> >> >> >> >><br>
>> >> >> >> >> strace -p $pid -f -tt --trace=ipc<br>
>> >> >> >> >><br>
>> >> >> >> >> -f: (follow the child process of pgpool main)<br>
>> >> >> >> >> -tt: add timestamp with milliseconds<br>
>> >> >> >> >> --trace=ipc: limit the tracing system call to ipc to reduce the<br>
>> >> log<br>
>> >> >> >> >><br>
>> >> >> >> >> BTW, I recommend yo to set log_line_prefix to something like<br>
>> this<br>
>> >> so<br>
>> >> >> >> >> that you can take milliseconds log (%m) and process application<br>
>> >> name<br>
>> >> >> >> >> (%a). They are usable in 4.6.<br>
>> >> >> >> >><br>
>> >> >> >> >> log_line_prefix = '%m: %a pid %p: '<br>
>> >> >> >> >><br>
>> >> >> >> >> > On Sun, Jun 1, 2025 at 4:20 PM Tatsuo Ishii <<br>
>> >> <a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> >> >> >> >> wrote:<br>
>> >> >> >> >> ><br>
>> >> >> >> >> >> > To your points/questions:<br>
>> >> >> >> >> >> > 1) version is 4.6.1 - the source I'm using (via apt get)<br>
>> >> doesn't<br>
>> >> >> >> have<br>
>> >> >> >> >> >> 4.6.2<br>
>> >> >> >> >> >> > yet<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> Ok.<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> > 2) ipcs -s shows... nothing, literally. Checked all<br>
>> nodes,<br>
>> >> i.e.<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > postgres@db-replica3:~$ ipcs -s<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > ------ Semaphore Arrays --------<br>
>> >> >> >> >> >> > key semid owner perms nsems<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > Also, it kinda seems like "failed to unlock semaphore"<br>
>> shows<br>
>> >> >> during<br>
>> >> >> >> >> >> startup<br>
>> >> >> >> >> >> > of pgpool and while it's running, but maybe I'm not<br>
>> reading<br>
>> >> the<br>
>> >> >> log<br>
>> >> >> >> >> >> > properly.<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> So pgpool lost semaphore soon after starting. Can you share<br>
>> >> full<br>
>> >> >> >> >> >> pgpool log before "failed to unlock semaphore" appears?<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> > 3) regarding heartbeats, my pgpool.conf has it roughly<br>
>> like<br>
>> >> >> this:<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > heartbeat_hostname0 = 'db-primary-priv'<br>
>> >> >> >> >> >> > heartbeat_port0 = 9694<br>
>> >> >> >> >> >> > heartbeat_device0 = ''<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > heartbeat_hostname1 = 'db-replica1-priv'<br>
>> >> >> >> >> >> > heartbeat_port1 = 9694<br>
>> >> >> >> >> >> > heartbeat_device1 = ''<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > heartbeat_hostname2 = 'db-replica2-priv'<br>
>> >> >> >> >> >> > heartbeat_port2 = 9694<br>
>> >> >> >> >> >> > heartbeat_device2 = ''<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > heartbeat_hostname3 = 'db-replica3-priv'<br>
>> >> >> >> >> >> > heartbeat_port3 = 9694<br>
>> >> >> >> >> >> > heartbeat_device3 = ''<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> Looks normal.<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> I have checked the pgpool source code and found suspicious<br>
>> >> >> >> >> >> code. before 4.6, pgpool heartbeart receiver process binds<br>
>> to<br>
>> >> any<br>
>> >> >> >> >> >> address on the host. However from 4.6 pgpool binds to only<br>
>> loop<br>
>> >> >> back<br>
>> >> >> >> >> >> address, which is plain wrong. Is it possible for you to<br>
>> patch<br>
>> >> the<br>
>> >> >> >> >> >> pgpool source code and build pgpool binary?<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> > So it seems likely that the inability to lock/unlock<br>
>> >> semaphores<br>
>> >> >> is<br>
>> >> >> >> the<br>
>> >> >> >> >> >> > problem, question is why and how do I resolve it? :(<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> Currently I have no idea. That's why I am requesting the<br>
>> pgpool<br>
>> >> >> log.<br>
>> >> >> >> >> >><br>
>> >> >> >> >> >> > I'll c/p parts of my pgpool.service file as well, in case<br>
>> you<br>
>> >> >> may<br>
>> >> >> >> see<br>
>> >> >> >> >> >> > something 'off' in it?<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > [Service]<br>
>> >> >> >> >> >> > User=postgres<br>
>> >> >> >> >> >> > Group=postgres<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > EnvironmentFile=-/etc/default/pgpool2<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > ExecStart=/usr/sbin/pgpool -f /etc/pgpool2/pgpool.conf<br>
>> $OPTS<br>
>> >> >> >> >> >> > ExecStop=/usr/sbin/pgpool -f /etc/pgpool2/pgpool.conf<br>
>> >> $STOP_OPTS<br>
>> >> >> >> stop<br>
>> >> >> >> >> >> > ExecReload=/usr/sbin/pgpool -f /etc/pgpool2/pgpool.conf<br>
>> >> reload<br>
>> >> >> >> >> >> > LimitNOFILE=65536:65536<br>
>> >> >> >> >> >> > LimitNPROC=16384:16384<br>
>> >> >> >> >> >> > LimitSTACK=10485760:33554432<br>
>> >> >> >> >> >> > LimitMEMLOCK=infinity<br>
>> >> >> >> >> >> > LimitDATA=infinity<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > Thanks in advance.<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> > On Sun, Jun 1, 2025 at 2:06 PM Tatsuo Ishii <<br>
>> >> >> <a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> >> >> >> >> >> wrote:<br>
>> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > Hi,<br>
>> >> >> >> >> >> >> > So long story short, I'm migrating a PG13 cluster from<br>
>> >> RHEL<br>
>> >> >> to<br>
>> >> >> >> >> Ubuntu<br>
>> >> >> >> >> >> >> > 24.04. In the process, the pgpool version is also being<br>
>> >> >> >> upgraded to<br>
>> >> >> >> >> >> 4.6<br>
>> >> >> >> >> >> >> > (from 4.2).<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> What is the exact version of 4.6? I am asking because<br>
>> 4.6.0<br>
>> >> has<br>
>> >> >> >> >> >> >> vulnerability.<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >><br>
>> >> >> >> >><br>
>> >> >> >><br>
>> >> >><br>
>> >><br>
>> <a href="https://pgpool.net/mediawiki/index.php/Main_Page#Pgpool-II_4.6.1.2C_4.5.7.2C_4.4.12.2C_4.3.15_and_4.2.22_officially_released_.282025.2F05.2F15.29" rel="noreferrer" target="_blank">https://pgpool.net/mediawiki/index.php/Main_Page#Pgpool-II_4.6.1.2C_4.5.7.2C_4.4.12.2C_4.3.15_and_4.2.22_officially_released_.282025.2F05.2F15.29</a><br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> Also 4.6.1 (and 4.6.0) have an issue with IPv6.<br>
>> >> >> >> >> >> >><br>
>> >> <a href="https://www.pgpool.net/docs/latest/en/html/release-4-6-2.html" rel="noreferrer" target="_blank">https://www.pgpool.net/docs/latest/en/html/release-4-6-2.html</a><br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> If possible, I recommend to upgrade to 4.6.2.<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> > There is also repmgr running on this 4 node cluster, 1<br>
>> >> >> primary<br>
>> >> >> >> >> >> >> > and 3 replicas.<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> I am not famililar with repmgr, and I ignore this part.<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> > All the migration seems to have completed successfully,<br>
>> >> >> >> >> >> >> > but I've ran into a weird problem with pgpool. I have<br>
>> >> quite<br>
>> >> >> a<br>
>> >> >> >> few<br>
>> >> >> >> >> >> errors<br>
>> >> >> >> >> >> >> > that look like the below entries, and it seems to "come<br>
>> >> and<br>
>> >> >> go"<br>
>> >> >> >> >> when<br>
>> >> >> >> >> >> >> > checking pgpool log. The process itself, seems to<br>
>> work (I<br>
>> >> >> >> think),<br>
>> >> >> >> >> but<br>
>> >> >> >> >> >> >> > every now and then, these errors (warnings?) show up:<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628043: DEBUG: watchdog<br>
>> >> heartbeat:<br>
>> >> >> >> send<br>
>> >> >> >> >> >> >> heartbeat<br>
>> >> >> >> >> >> >> > signal to db-replica3-priv:9694<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628043: LOCATION:<br>
>> >> >> wd_heartbeat.c:771<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: LOG: exit handler<br>
>> called<br>
>> >> >> >> >> (signal: 2)<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: LOCATION:<br>
>> >> >> pgpool_main.c:1272<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > *2025-06-01 09:11:00: pid 627958: WARNING: failed to<br>
>> lock<br>
>> >> >> >> >> >> >> > semaphore2025-06-01 09:11:00: pid 627958: DETAIL:<br>
>> Invalid<br>
>> >> >> >> >> >> >> argument*2025-06-01<br>
>> >> >> >> >> >> >> > 09:11:00: pid 627958: LOCATION: pool_sema.c:129<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: WARNING: failed to<br>
>> >> unlock<br>
>> >> >> >> >> semaphore<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: DETAIL: Invalid<br>
>> argument<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: LOCATION:<br>
>> >> pool_sema.c:202<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: LOG: shutting down by<br>
>> >> >> signal 2<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: LOCATION:<br>
>> >> >> pgpool_main.c:1324<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: LOG: terminating all<br>
>> >> child<br>
>> >> >> >> >> processes<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 627958: LOCATION:<br>
>> >> >> pgpool_main.c:1334<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628047: WARNING: failed to<br>
>> lock<br>
>> >> >> >> semaphore<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628047: DETAIL: Invalid<br>
>> argument<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628047: LOCATION:<br>
>> >> pool_sema.c:129<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628042: WARNING: failed to<br>
>> lock<br>
>> >> >> >> semaphore<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628042: DETAIL: Invalid<br>
>> argument<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628042: LOCATION:<br>
>> >> pool_sema.c:129<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628039: WARNING: failed to<br>
>> lock<br>
>> >> >> >> semaphore<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:00: pid 628039: DETAIL: Invalid<br>
>> argument<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> Can you share the result of "ipcs -s" so that I can<br>
>> confirm<br>
>> >> >> pgpool<br>
>> >> >> >> >> >> >> properly allocates the semaphore.<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> By the way do you see "WARNING: failed to unlock<br>
>> semaphore"<br>
>> >> >> only<br>
>> >> >> >> when<br>
>> >> >> >> >> >> >> pgpool is shutting down?<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> > Also, earlier in the log, I see errors about being<br>
>> unable<br>
>> >> to<br>
>> >> >> >> >> create a<br>
>> >> >> >> >> >> >> > socket, for some reason it's trying (and failing) to<br>
>> >> create<br>
>> >> >> >> socket<br>
>> >> >> >> >> on<br>
>> >> >> >> >> >> >> > localhost (<a href="http://127.0.0.1:9694" rel="noreferrer" target="_blank">127.0.0.1:9694</a>) and "creating socket on<br>
>> >> >> ::1:9694".<br>
>> >> >> >> >> Are<br>
>> >> >> >> >> >> >> these<br>
>> >> >> >> >> >> >> > "ok to ignore"?<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> First of all "creating socket on ::1:9694" looks unusual<br>
>> >> >> because<br>
>> >> >> >> it<br>
>> >> >> >> >> >> >> means that heartbeat_hostname is set to "::1", which is<br>
>> the<br>
>> >> >> IPv6<br>
>> >> >> >> loop<br>
>> >> >> >> >> >> >> back address. heartbeat_hostname is expected to be set<br>
>> IPv4<br>
>> >> or<br>
>> >> >> >> IPv6<br>
>> >> >> >> >> >> >> inet addesss (not loop back address).<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> For now, I would like to wait for your reply regarding<br>
>> >> these my<br>
>> >> >> >> >> >> >> questions above.<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> Best regards,<br>
>> >> >> >> >> >> >> --<br>
>> >> >> >> >> >> >> Tatsuo Ishii<br>
>> >> >> >> >> >> >> SRA OSS K.K.<br>
>> >> >> >> >> >> >> English: <a href="http://www.sraoss.co.jp/index_en/" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en/</a><br>
>> >> >> >> >> >> >> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: DETAIL: creating<br>
>> socket<br>
>> >> on<br>
>> >> >> >> >> >> >> <a href="http://127.0.0.1:9694" rel="noreferrer" target="_blank">127.0.0.1:9694</a><br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: LOCATION:<br>
>> >> >> wd_heartbeat.c:385<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: LOG: set SO_REUSEPORT<br>
>> >> >> option<br>
>> >> >> >> to<br>
>> >> >> >> >> the<br>
>> >> >> >> >> >> >> socket<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: LOCATION:<br>
>> >> >> wd_heartbeat.c:891<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: LOG: creating<br>
>> watchdog<br>
>> >> >> >> heartbeat<br>
>> >> >> >> >> >> >> receive<br>
>> >> >> >> >> >> >> > socket.<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: DETAIL: creating<br>
>> socket<br>
>> >> on<br>
>> >> >> >> >> ::1:9694<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: LOCATION:<br>
>> >> >> wd_heartbeat.c:385<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: LOG: failed to create<br>
>> >> >> watchdog<br>
>> >> >> >> >> >> >> heartbeat<br>
>> >> >> >> >> >> >> > receive socket. retrying...<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: DETAIL: bind failed<br>
>> with<br>
>> >> >> >> reason:<br>
>> >> >> >> >> >> >> "Cannot<br>
>> >> >> >> >> >> >> > assign requested address"<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634772: LOCATION:<br>
>> >> >> wd_heartbeat.c:394<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: DEBUG:<br>
>> >> >> >> >> verify_backend_node_status:<br>
>> >> >> >> >> >> >> > true_primary 0<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: LOCATION:<br>
>> >> >> pgpool_main.c:2704<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: DEBUG: node<br>
>> status[0]: 1<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: LOCATION:<br>
>> >> >> >> pool_worker_child.c:267<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: DEBUG: node<br>
>> status[1]: 2<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: LOCATION:<br>
>> >> >> >> pool_worker_child.c:267<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: DEBUG: node<br>
>> status[2]: 2<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: LOCATION:<br>
>> >> >> >> pool_worker_child.c:267<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: DEBUG: node<br>
>> status[3]: 2<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: LOCATION:<br>
>> >> >> >> pool_worker_child.c:267<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: DEBUG:<br>
>> >> >> >> >> >> pool_release_follow_primary_lock<br>
>> >> >> >> >> >> >> > called<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:22: pid 635274: LOCATION:<br>
>> >> >> pgpool_main.c:4106<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: LOG: set SO_REUSEPORT<br>
>> >> >> option<br>
>> >> >> >> to<br>
>> >> >> >> >> the<br>
>> >> >> >> >> >> >> socket<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: LOCATION:<br>
>> >> >> wd_heartbeat.c:891<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: LOG: creating<br>
>> watchdog<br>
>> >> >> >> heartbeat<br>
>> >> >> >> >> >> >> receive<br>
>> >> >> >> >> >> >> > socket.<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: DETAIL: creating<br>
>> socket<br>
>> >> on<br>
>> >> >> >> >> >> >> <a href="http://127.0.0.1:9694" rel="noreferrer" target="_blank">127.0.0.1:9694</a><br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: LOCATION:<br>
>> >> >> wd_heartbeat.c:385<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: LOG: set SO_REUSEPORT<br>
>> >> >> option<br>
>> >> >> >> to<br>
>> >> >> >> >> the<br>
>> >> >> >> >> >> >> socket<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: LOCATION:<br>
>> >> >> wd_heartbeat.c:891<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: LOG: creating<br>
>> watchdog<br>
>> >> >> >> heartbeat<br>
>> >> >> >> >> >> >> receive<br>
>> >> >> >> >> >> >> > socket.<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > *2025-06-01 09:11:23: pid 634765: DETAIL: creating<br>
>> >> socket on<br>
>> >> >> >> >> >> >> > ::1:96942025-06-01 09:11:23: pid 634765: LOCATION:<br>
>> >> >> >> >> >> >> > wd_heartbeat.c:3852025-06-01 09:11:23: pid 634765:<br>
>> LOG:<br>
>> >> >> >> failed to<br>
>> >> >> >> >> >> >> create<br>
>> >> >> >> >> >> >> > watchdog heartbeat receive socket.<br>
>> retrying...*2025-06-01<br>
>> >> >> >> 09:11:23:<br>
>> >> >> >> >> >> pid<br>
>> >> >> >> >> >> >> > 634765: DETAIL: bind failed with reason: "Cannot<br>
>> assign<br>
>> >> >> >> requested<br>
>> >> >> >> >> >> >> address"<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634765: LOCATION:<br>
>> >> >> wd_heartbeat.c:394<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634767: LOG: set SO_REUSEPORT<br>
>> >> >> option<br>
>> >> >> >> to<br>
>> >> >> >> >> the<br>
>> >> >> >> >> >> >> socket<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634767: LOCATION:<br>
>> >> >> wd_heartbeat.c:891<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634767: LOG: creating socket<br>
>> for<br>
>> >> >> >> sending<br>
>> >> >> >> >> >> >> heartbeat<br>
>> >> >> >> >> >> >> > 2025-06-01 09:11:23: pid 634767: DETAIL: set<br>
>> SO_REUSEPORT<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > Other odd errors I see in the logs later on are when<br>
>> >> pgpool<br>
>> >> >> >> tries<br>
>> >> >> >> >> to<br>
>> >> >> >> >> >> run<br>
>> >> >> >> >> >> >> > escalation script - can't quite find where it's<br>
>> failing to<br>
>> >> >> find<br>
>> >> >> >> >> ipv4<br>
>> >> >> >> >> >> >> > address:<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:15: pid 395780: LOG: Pgpool-II parent<br>
>> >> >> process<br>
>> >> >> >> >> >> received<br>
>> >> >> >> >> >> >> > SIGUSR1<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:15: pid 395780: LOG: Pgpool-II parent<br>
>> >> >> process<br>
>> >> >> >> >> >> received<br>
>> >> >> >> >> >> >> > watchdog quorum change signal from watchdog<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:15: pid 396973: LOG: watchdog:<br>
>> >> escalation<br>
>> >> >> >> started<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:15: pid 395786: LOG: Setting failover<br>
>> >> >> command<br>
>> >> >> >> >> >> timeout<br>
>> >> >> >> >> >> >> to<br>
>> >> >> >> >> >> >> > 135<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:15: pid 395780: LOG: watchdog cluster<br>
>> >> now<br>
>> >> >> >> holds<br>
>> >> >> >> >> the<br>
>> >> >> >> >> >> >> quorum<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:15: pid 395780: DETAIL: updating the<br>
>> >> state<br>
>> >> >> of<br>
>> >> >> >> >> >> >> quarantine<br>
>> >> >> >> >> >> >> > backend nodes<br>
>> >> >> >> >> >> >> > Sun Jun 1 12:07:15 AM CEST 2025<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > *escalation.sh: - remove VIP IP from all nodesNode:<br>
>> >> >> >> >> >> db-primary-privError:<br>
>> >> >> >> >> >> >> > ipv4: Address not found.Node: db-replica1-privError:<br>
>> ipv4:<br>
>> >> >> >> Address<br>
>> >> >> >> >> not<br>
>> >> >> >> >> >> >> > found.Node: db-replica2-privError: ipv4: Address not<br>
>> >> >> found.Node:<br>
>> >> >> >> >> >> >> > db-replica3-privError: ipv4: Address not<br>
>> found.*2025-06-01<br>
>> >> >> >> >> 00:07:20:<br>
>> >> >> >> >> >> pid<br>
>> >> >> >> >> >> >> > 396973: LOG: watchdog escalation successful<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:23: pid 396973: LOG: successfully<br>
>> >> acquired<br>
>> >> >> the<br>
>> >> >> >> >> >> delegate<br>
>> >> >> >> >> >> >> > IP:"192.168.77.130"<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:23: pid 396973: DETAIL: 'if_up_cmd'<br>
>> >> >> returned<br>
>> >> >> >> with<br>
>> >> >> >> >> >> >> success<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:23: pid 395786: LOG: watchdog<br>
>> escalation<br>
>> >> >> >> process<br>
>> >> >> >> >> >> with<br>
>> >> >> >> >> >> >> > pid: 396973 exit with SUCCESS.<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:27: pid 395962: WARNING: failed to<br>
>> lock<br>
>> >> >> >> semaphore<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:27: pid 395962: DETAIL: Invalid<br>
>> argument<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:27: pid 395962: WARNING: failed to<br>
>> >> unlock<br>
>> >> >> >> >> semaphore<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:27: pid 395962: DETAIL: Invalid<br>
>> argument<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:28: pid 396924: WARNING: failed to<br>
>> lock<br>
>> >> >> >> semaphore<br>
>> >> >> >> >> >> >> > 2025-06-01 00:07:28: pid 396924: DETAIL: Invalid<br>
>> argument<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > The way I started was I copied pgpool.conf from the<br>
>> "old"<br>
>> >> and<br>
>> >> >> >> >> working<br>
>> >> >> >> >> >> >> > pgpool 4.2 configuration and modified it slightly,<br>
>> though<br>
>> >> >> >> nothing<br>
>> >> >> >> >> >> >> > major,changed hostnames and i.e. delegate_ip variable<br>
>> name<br>
>> >> >> was<br>
>> >> >> >> >> changed<br>
>> >> >> >> >> >> >> > from delegate_IP. Also added this variable as without<br>
>> >> it, I<br>
>> >> >> >> didn't<br>
>> >> >> >> >> >> see a<br>
>> >> >> >> >> >> >> > socket created for port 9898:<br>
>> >> >> >> >> >> >> > unix_socket_directories = '/var/run/postgresql'<br>
>> >> >> >> >> >> >> > I had to change network interfaces as well, in the<br>
>> RHEL it<br>
>> >> >> was<br>
>> >> >> >> >> using<br>
>> >> >> >> >> >> eth0<br>
>> >> >> >> >> >> >> > where on Ubuntu it's ens18, so I changed lines like:<br>
>> >> >> >> >> >> >> > /usr/bin/sudo /sbin/ip addr add $_IP_$/23 dev ens18<br>
>> label<br>
>> >> >> >> ens18:0<br>
>> >> >> >> >> >> >> > Other thing I had to change was path to arping as on<br>
>> >> Ubuntu<br>
>> >> >> 24<br>
>> >> >> >> >> it's in<br>
>> >> >> >> >> >> >> > /usr/bin (not sbin)<br>
>> >> >> >> >> >> >> > arping_cmd = '/usr/bin/sudo /usr/bin/arping -U $_IP_$<br>
>> -w<br>
>> >> 1<br>
>> >> >> -I<br>
>> >> >> >> >> ens18'<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > The other thing I'm confused about, I have two network<br>
>> >> >> >> interfaces<br>
>> >> >> >> >> on<br>
>> >> >> >> >> >> each<br>
>> >> >> >> >> >> >> > host, i.e.<br>
>> >> >> >> >> >> >> > <a href="http://192.168.77.135/23" rel="noreferrer" target="_blank">192.168.77.135/23</a> brd 192.168.77.255 scope global<br>
>> ens18<br>
>> >> >> >> >> >> >> > and<br>
>> >> >> >> >> >> >> > inet <a href="http://192.168.78.135/24" rel="noreferrer" target="_blank">192.168.78.135/24</a> brd 192.168.78.255 scope global<br>
>> >> ens19<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > In pgpool.conf, it is referencing the "priv" hosts<br>
>> (i.e.<br>
>> >> >> >> >> >> >> > heartbeat_hostname2 = 'db-replica2-priv'), which<br>
>> >> >> theoretically<br>
>> >> >> >> >> should<br>
>> >> >> >> >> >> be<br>
>> >> >> >> >> >> >> on<br>
>> >> >> >> >> >> >> > ens19 network interface as shown above, but in my old<br>
>> >> >> >> pgpool.conf,<br>
>> >> >> >> >> it<br>
>> >> >> >> >> >> was<br>
>> >> >> >> >> >> >> > only referencing eth0 on RHEL, even though "priv" hosts<br>
>> >> were<br>
>> >> >> on<br>
>> >> >> >> >> eth1.<br>
>> >> >> >> >> >> >> I'm<br>
>> >> >> >> >> >> >> > wondering if that needs to be changed in Ubuntu or it<br>
>> >> makes<br>
>> >> >> no<br>
>> >> >> >> >> >> difference<br>
>> >> >> >> >> >> >> > here?<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > Right now, I can't quite figure out whether it is<br>
>> >> something<br>
>> >> >> >> with<br>
>> >> >> >> >> >> >> > pgpool.conf or with the Ubuntu settings and the<br>
>> semaphore<br>
>> >> >> error<br>
>> >> >> >> is<br>
>> >> >> >> >> >> quite<br>
>> >> >> >> >> >> >> > cryptic. If you need me to post some parts of my<br>
>> >> >> pgpool.conf,<br>
>> >> >> >> >> please<br>
>> >> >> >> >> >> >> > advise which. I've seen an older version of the FAQ<br>
>> that<br>
>> >> >> said<br>
>> >> >> >> >> repmgr<br>
>> >> >> >> >> >> and<br>
>> >> >> >> >> >> >> > pgpool don't work together but then I saw mention that<br>
>> >> this<br>
>> >> >> was<br>
>> >> >> >> no<br>
>> >> >> >> >> >> longer<br>
>> >> >> >> >> >> >> > the case.<br>
>> >> >> >> >> >> >> ><br>
>> >> >> >> >> >> >> > I would much appreciate any help you can provide in<br>
>> >> solving<br>
>> >> >> this<br>
>> >> >> >> >> >> problem.<br>
>> >> >> >> >> >> >><br>
>> >> >> >> >> >><br>
>> >> >> >> >><br>
>> >> >> >><br>
>> >> >><br>
>> >><br>
>> _______________________________________________<br>
>> pgpool-general mailing list<br>
>> <a href="mailto:pgpool-general@pgpool.net" target="_blank">pgpool-general@pgpool.net</a><br>
>> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
>><br>
</blockquote></div>