View Issue Details

IDProjectCategoryView StatusLast Update
0000668Pgpool-IIBugpublic2020-12-09 15:50
Reportersyarif.thalib Assigned Topengbo  
PriorityhighSeveritymajorReproducibilityalways
Status feedbackResolutionopen 
PlatformLinuxOSCentosOS Version7.9.2009
Product Version4.1.5 
Summary0000668: Virtual IP not starting up due to failed to create watchdog heartbeat receive socket
DescriptionHI Support,

We've configured 2 nodes pg cluster ( postgres version 12.5) with pgpool refer to this link https://www.pgpool.net/docs/latest/en/html/example-cluster.html.
Here the ip configuration :
Node 1
ip local : 192.168.168.3
ip private p2p : 100.10.10.1

Node 2
ip local : 192.168.168.4
ip private p2p : 100.10.10.2

ip virtual : 192.168.168.13.

when we tried to bringup vip ip using systemctl start pgpool. vip ip is failed to start up due to below error

Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [8-1] 2020-12-02 18:44:53: pid 18255: LOG: failed to create watchdog heartbeat receive socket.
Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [8-2] 2020-12-02 18:44:53: pid 18255: DETAIL: setsockopt(SO_BINDTODEVICE) requies root privilege
Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [9-1] 2020-12-02 18:44:53: pid 18255: LOG: set SO_REUSEPORT option to the socket
Dec 2 18:44:53 postgredbprod01 pgpool[18255]: [10-1] 2020-12-02 18:44:53: pid 18255: LOG: creating watchdog he
Additional InformationHowever, when we tried manually bringup vip ip using postgres user is successfully

+++ Before +++
-bash-4.2$ ifconfig -a | grep -i inet
        inet 192.168.168.3 netmask 255.255.255.224 broadcast 192.168.168.31
        inet6 fe80::2e71:12d9:f32a:54f0 prefixlen 64 scopeid 0x20<link>
        inet 100.10.10.1 netmask 255.255.255.252 broadcast 100.10.10.3
        inet6 fe80::bbdc:5a75:d8f9:a458 prefixlen 64 scopeid 0x20<link>
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>

++ Bringup vip manually ++
-bash-4.2$ /usr/bin/sudo /var/lib/pgsql/sbin/ip addr add 192.168.168.13/27 dev ens192 label ens192:0

++ After ++
-bash-4.2$ ifconfig -a | grep -i inet
        inet 192.168.168.3 netmask 255.255.255.224 broadcast 192.168.168.31
        inet6 fe80::2e71:12d9:f32a:54f0 prefixlen 64 scopeid 0x20<link>
        inet 192.168.168.13 netmask 255.255.255.224 broadcast 0.0.0.0
        inet 100.10.10.1 netmask 255.255.255.252 broadcast 100.10.10.3
        inet6 fe80::bbdc:5a75:d8f9:a458 prefixlen 64 scopeid 0x20<link>
        inet 127.0.0.1 netmask 255.0.0.0
        inet6 ::1 prefixlen 128 scopeid 0x10<host>
TagsNo tags attached.

Activities

syarif.thalib

2020-12-02 21:18

reporter  

pgpool.log.log (99,692 bytes)
pool_hba_node2.conf (3,722 bytes)
pgpool_node2.conf (44,627 bytes)
pgpool_node1.conf (44,665 bytes)
pool_hba_node1.conf.txt (3,844 bytes)   
# pgpool Client Authentication Configuration File
# ===============================================
#
# The format rule in this file follows the rules in the PostgreSQL
# Administrator's Guide. Refer to chapter "Client Authentication" for a
# complete description.  A short synopsis follows.
#
# This file controls: which hosts are allowed to connect, how clients
# are authenticated, which user names they can use, which databases they
# can access.  Records take one of these forms:
#
# local      DATABASE  USER  METHOD  [OPTION]
# host       DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
# hostssl    DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
# hostnossl  DATABASE  USER  ADDRESS  METHOD  [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
# The first field is the connection type: "local" is a Unix-domain
# socket, "host" is either a plain or SSL-encrypted TCP/IP socket.
# "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a
# plain TCP/IP socket.
#
# DATABASE can be "all", "sameuser", a database name, or a comma-separated
# list thereof. Note that "samegroup" like in PostgreSQL's pg_hba.conf
# file is not supported, since pgpool does not know which group a user
# belongs to. Also note that the database specified here may not exist in
# the backend PostgreSQL. pgpool will authenticate based on the database's
# name, not based on whether it exists or not.
#
# USER can be "all", a user name, or a comma-separated list thereof.  In
# both the DATABASE and USER fields you can also write a file name prefixed
# with "@" to include names from a separate file. Note that a group name
# prefixed with "+" like in PostgreSQL's pg_hba.conf file is not supported
# because of the same reason as "samegroup" token. Also note that a user
# name specified here may not exist in the backend PostgreSQL. pgpool will
# authenticate based on the user's name, not based on whether he/she exists.
#
# CIDR-ADDRESS specifies the set of hosts the record matches.
# It is made up of an IP address and a CIDR mask that is an integer
# (between 0 and 32 (IPv4) that specifies the number of significant bits in
# the mask.  Alternatively, you can write an IP address and netmask in
# separate columns to specify the set of hosts.
#
# METHOD can be "trust", "reject", "md5" , "scram-sha-256" or "pam".
# Note that "pam" sends passwords in clear text.
#
# OPTION is the name of the PAM service. Default service name is "pgpool"
#
# Database and user names containing spaces, commas, quotes and other special
# characters must be quoted. Quoting one of the keywords "all" or "sameuser"
# makes the name lose its special character, and just match a database or
# username with that name.
#
# This file is read on pgpool startup.  If you edit the file on a running
# system, you have to restart the pgpool  for the changes to take effect.

# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records. In that case you will also need to make pgpool listen
# on a non-local interface via the listen_addresses configuration parameter.
#

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD

# "local" is for Unix domain socket connections only
local   all         all                               trust
# IPv4 local connections:
host    all         all         127.0.0.1/32          trust
host    all         all         ::1/128               trust
#host    all         all       0.0.0.0/0              scram-sha-256
####main config###
host    all         pgpool      0.0.0.0/0             scram-sha-256
host    all         postgres    0.0.0.0/0             scram-sha-256
host    all         all         0.0.0.0/0             scram-sha-256
pool_hba_node1.conf.txt (3,844 bytes)   

syarif.thalib

2020-12-02 21:55

reporter   ~0003634

We are installing pgpool using yum repository https://www.pgpool.net/yum/rpms/4.1/redhat/rhel-7-x86_64/pgpool-II-release-4.1-2.noarch.rpm.

Found doc from github related pgpool setsockopt(SO_BINDTODEVICE) requies root privilege error ( https://github.com/mbell8903/pgpool-II/blob/master/src/watchdog/wd_heartbeat.c)

Seems, pgpool do a verification check uid must be 0 ( root), else setsockopt(SO_BINDTODEVICE) requires root privilege error will shown. Thus, we are unable to bringup vip using postgres user even have sudoer.

#if defined(SO_BINDTODEVICE)
        {
            if (geteuid() == 0) /* check root privileges */
            {
                struct ifreq i;
                strlcpy(i.ifr_name, hb_if->if_name, sizeof(i.ifr_name));

                if (setsockopt(sock, SOL_SOCKET, SO_BINDTODEVICE, &i, sizeof(i)) == -1)
                {
                    close(sock);
                    ereport(ERROR,
                        (errmsg("failed to create watchdog heartbeat sender socket"),
                             errdetail("setsockopt(SO_BINDTODEVICE) failed with reason: \"%s\"", strerror(errno))));

                }
                ereport(LOG,
                    (errmsg("creating socket for sending heartbeat"),
                         errdetail("bind send socket to device: %s", i.ifr_name)));
            }
            else
                ereport(LOG,
                    (errmsg("creating socket for sending heartbeat"),
                         errdetail("setsockopt(SO_BINDTODEVICE) requires root privilege"))); --> We are hitting this condition
        }
#else
        ereport(LOG,
            (errmsg("creating socket for sending heartbeat"),
                 errdetail("setsockopt(SO_BINDTODEVICE) is not available on this platform")));


Is there any steps / way to solve this issue?

pengbo

2020-12-04 09:15

developer   ~0003637

Thank you for reporting this issue.

Yes. postgres user doesn't have the privilege to bind socket to a particular device.
I'm asking other developer who implemented wacthdog for the solution.

pengbo

2020-12-09 15:47

developer   ~0003645

Last edited: 2020-12-09 15:50

> Virtual IP not starting up due to failed to create watchdog heartbeat receive socket

To use "heartbeat_device" option root privilege is mandatory.
But it should not cause the problem with starting up VIP.
It only throws the log message when bind to device fails and keeps using the socket.


You are using 2 pgpool nodes and only one pgpool is started?
In this case, the quorum does not exist.

As mentioned in the documentation:
https://www.pgpool.net/docs/latest/en/html/runtime-watchdog-config.html#CONFIG-WATCHDOG-VIP-CONTROL
"VIP will not be brought up in case the quorum does not exist."

To bring up VIP in case that the quorum does not exist, you need to enable "enable_consensus_with_half_votes" parameter.
https://www.pgpool.net/docs/latest/en/html/runtime-watchdog-config.html#GUC-ENABLE-CONSENSUS-WITH-HALF-VOTES

  enable_consensus_with_half_votes = on

Issue History

Date Modified Username Field Change
2020-12-02 21:18 syarif.thalib New Issue
2020-12-02 21:18 syarif.thalib File Added: pgpool.log.log
2020-12-02 21:18 syarif.thalib File Added: pool_hba_node2.conf
2020-12-02 21:18 syarif.thalib File Added: pgpool_node2.conf
2020-12-02 21:18 syarif.thalib File Added: pgpool_node1.conf
2020-12-02 21:18 syarif.thalib File Added: pool_hba_node1.conf.txt
2020-12-02 21:55 syarif.thalib Note Added: 0003634
2020-12-04 09:05 pengbo Assigned To => pengbo
2020-12-04 09:05 pengbo Status new => assigned
2020-12-04 09:05 pengbo Description Updated
2020-12-04 09:05 pengbo Additional Information Updated
2020-12-04 09:15 pengbo Note Added: 0003637
2020-12-04 09:15 pengbo Status assigned => feedback
2020-12-09 15:47 pengbo Note Added: 0003645
2020-12-09 15:50 pengbo Note Edited: 0003645