View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000740Pgpool-IIBugpublic2021-12-04 01:212022-05-23 17:12
Reportervenkat_rj Assigned Tot-ishii  
PriorityurgentSeveritymajorReproducibilityalways
Status closedResolutionopen 
PlatformLinuxOSRed Hat Enterprise Linux ServerOS Version7.3
Product Version4.0.6 
Summary0000740: ssl_ciphers are not honored and pgpool port allow other vulnerable non-compliant ciphers as per sslscan tool
DescriptionAs part of security scans one of our customer found that pgpool port is allowing other vulnerable non-compliant ciphers other then the configured ciphers in the pgpool.conf.

Note Postgres port does not allow non-compliant ciphers, but pgpool port dose as per sslscan tool.

We had set to 4 ciphers in the config file i.e. 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384'

Here is the sslscan tool link https://github.com/rbsec/sslscan

Here are the sslscan logs for both Postgres and pgpool port gathered from our system.

-------------------------------------------------------------------
Postgres Port:
-------------------------------------------------------------------
Command:
   sslscan --starttls-psql <MOCKED_HOSTNAME>:5432

Example:
   sslscan --starttls-psql mylax51.unx.com:5432
-------------------------------------------------------------------
 sslscan Log:
-------------------------------------------------------------------
Version: 2.0.10-static
OpenSSL 1.1.1m-dev xx XXX xxxx

Connected to <MOKED_IP_ADDRESS>

Testing SSL server <MOCKED_HOSTNAME> on port 5432 using SNI name <MOCKED_HOSTNAME>

  SSL/TLS Protocols:
SSLv2 disabled
SSLv3 disabled
TLSv1.0 disabled
TLSv1.1 disabled
TLSv1.2 enabled
TLSv1.3 disabled

  TLS Fallback SCSV:
Server supports TLS Fallback SCSV

  TLS renegotiation:
Session renegotiation not supported

  TLS Compression:
Compression disabled

  Heartbleed:
TLSv1.2 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.2 128 bits ECDHE-RSA-AES128-GCM-SHA256 Curve P-256 DHE 256
Accepted TLSv1.2 256 bits ECDHE-RSA-AES256-GCM-SHA384 Curve P-256 DHE 256

  Server Key Exchange Group(s):
TLSv1.2 128 bits secp256r1 (NIST P-256)

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength: 2048

Subject: <MOCKED_HOSTNAME>.unx.com
Altnames: DNS:localhost, DNS:localhost4, DNS:localhost4.localdomain4, DNS:localhost6, DNS:localhost6.localdomain6, DNS:localhost.localdomain, DNS:<MOCKED_HOSTNAME>.unx.com, DNS:<MOCKED_HOSTNAME>.unx.com, IP Address:<MOKED_IP_ADDRESS>, IP Address:FD07:AF30:A3FC:7928:250:56FF:FE8C:48B0, IP Address:127.0.0.1
Issuer: SAS VIYA Intermediate CA

Not valid before: Oct 18 16:26:10 2021 GMT
Not valid after: Oct 16 01:26:40 2028 GMT

-------------------------------------------------------------------
pgpool Port:
-------------------------------------------------------------------
Command:
   sslscan --starttls-psql <MOCKED_HOSTNAME>:5431

Example:
   sslscan --starttls-psql mylax51.unx.com:5431
-------------------------------------------------------------------
 sslscan Log:
-------------------------------------------------------------------
Version: 2.0.10-static
OpenSSL 1.1.1m-dev xx XXX xxxx

Connected to <MOKED_IP_ADDRESS>

Testing SSL server <MOKED_IP_ADDRESS> on port 5431 using SNI name <MOKED_IP_ADDRESS>

  SSL/TLS Protocols:
SSLv2 disabled
SSLv3 disabled
TLSv1.0 enabled
TLSv1.1 enabled
TLSv1.2 enabled
TLSv1.3 disabled

  TLS Fallback SCSV:
Server supports TLS Fallback SCSV

  TLS renegotiation:
Secure session renegotiation supported

  TLS Compression:
Compression disabled

  Heartbleed:
TLSv1.2 not vulnerable to heartbleed
TLSv1.1 not vulnerable to heartbleed
TLSv1.0 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.2 256 bits AES256-GCM-SHA384
Accepted TLSv1.2 128 bits AES128-GCM-SHA256
Accepted TLSv1.2 256 bits AES256-SHA256
Accepted TLSv1.2 128 bits AES128-SHA256
Accepted TLSv1.2 256 bits AES256-SHA
Accepted TLSv1.2 256 bits CAMELLIA256-SHA
Accepted TLSv1.2 128 bits AES128-SHA
Accepted TLSv1.2 128 bits SEED-SHA
Accepted TLSv1.2 128 bits CAMELLIA128-SHA
Accepted TLSv1.2 128 bits RC4-SHA
Accepted TLSv1.2 128 bits RC4-MD5
Accepted TLSv1.2 112 bits DES-CBC3-SHA
Preferred TLSv1.1 256 bits AES256-SHA
Accepted TLSv1.1 256 bits CAMELLIA256-SHA
Accepted TLSv1.1 128 bits AES128-SHA
Accepted TLSv1.1 128 bits SEED-SHA
Accepted TLSv1.1 128 bits CAMELLIA128-SHA
Accepted TLSv1.1 128 bits IDEA-CBC-SHA
Accepted TLSv1.1 128 bits RC4-SHA
Accepted TLSv1.1 128 bits RC4-MD5
Accepted TLSv1.1 112 bits DES-CBC3-SHA
Preferred TLSv1.0 256 bits AES256-SHA
Accepted TLSv1.0 256 bits CAMELLIA256-SHA
Accepted TLSv1.0 128 bits AES128-SHA
Accepted TLSv1.0 128 bits SEED-SHA
Accepted TLSv1.0 128 bits CAMELLIA128-SHA
Accepted TLSv1.0 128 bits IDEA-CBC-SHA
Accepted TLSv1.0 128 bits RC4-SHA
Accepted TLSv1.0 128 bits RC4-MD5
Accepted TLSv1.0 112 bits DES-CBC3-SHA

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength: 2048

Subject: <MOKED_IP_ADDRESS>.unx.com
Altnames: DNS:localhost, DNS:localhost4, DNS:localhost4.localdomain4, DNS:localhost6, DNS:localhost6.localdomain6, DNS:localhost.localdomain, DNS:<MOKED_IP_ADDRESS>.unx.com, DNS:<MOKED_IP_ADDRESS>.unx.com, IP Address:<MOKED_IP_ADDRESS>, IP Address:FD07:AF30:A3FC:7928:250:56FF:FE8C:48B0, IP Address:127.0.0.1
Issuer: SAS VIYA Intermediate CA

Not valid before: Oct 18 16:31:34 2021 GMT
Not valid after: Oct 16 01:32:04 2028 GMT








-------------------------------------------------------------------
Current SSL settings:
-------------------------------------------------------------------
ssl_ciphers = 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384' # Added through Consul-template

ssl_prefer_server_ciphers = on # Added through Consul-template

ssl_cert = '<PATH_MOCKED>/pgpool0/cert.pem' # Added through Consul-template

ssl_key = ''<PATH_MOCKED>/pgpool0/key.pem' # Added through Consul-template

ssl_ca_cert = '<PATH_MOCKED>/cacerts/trustedcerts.pem' # Added through Consul-template

ssl_ca_cert_dir = '<PATH_MOCKED>/cacerts' # Added through Consul-template
ssl = true # Added through Consul-template
Steps To Reproduce
-1 Install sslscan tool on pgpool machine
    Here is the link:
         https://github.com/rbsec/sslscan

-2 Execute the sslscan tool on pgpool port
    Command:
         sslscan --starttls-psql <MOCKED_HOSTNAME>:5431

Please replace
   from <MOCKED_HOSTNAME> to your machine hostname
   from 5431 to your pgpool port no in case if it's different.
in the above Command

Example:
   sslscan --starttls-psql mylax51.unx.com:5431

-3 Look for section 'Supported Server Cipher(s):' in the Command output
    Currently we see following:
    Supported Server Cipher(s):
       Preferred TLSv1.2 256 bits AES256-GCM-SHA384
       Accepted TLSv1.2 128 bits AES128-GCM-SHA256
       Accepted TLSv1.2 256 bits AES256-SHA256
       Accepted TLSv1.2 128 bits AES128-SHA256
       Accepted TLSv1.2 256 bits AES256-SHA
       Accepted TLSv1.2 256 bits CAMELLIA256-SHA
       Accepted TLSv1.2 128 bits AES128-SHA
       Accepted TLSv1.2 128 bits SEED-SHA
       Accepted TLSv1.2 128 bits CAMELLIA128-SHA
       Accepted TLSv1.2 128 bits RC4-SHA
       Accepted TLSv1.2 128 bits RC4-MD5
       Accepted TLSv1.2 112 bits DES-CBC3-SHA
       Preferred TLSv1.1 256 bits AES256-SHA
       Accepted TLSv1.1 256 bits CAMELLIA256-SHA
       Accepted TLSv1.1 128 bits AES128-SHA
       Accepted TLSv1.1 128 bits SEED-SHA
       Accepted TLSv1.1 128 bits CAMELLIA128-SHA
       Accepted TLSv1.1 128 bits IDEA-CBC-SHA
       Accepted TLSv1.1 128 bits RC4-SHA
       Accepted TLSv1.1 128 bits RC4-MD5
       Accepted TLSv1.1 112 bits DES-CBC3-SHA
       Preferred TLSv1.0 256 bits AES256-SHA
       Accepted TLSv1.0 256 bits CAMELLIA256-SHA
       Accepted TLSv1.0 128 bits AES128-SHA
       Accepted TLSv1.0 128 bits SEED-SHA
       Accepted TLSv1.0 128 bits CAMELLIA128-SHA
       Accepted TLSv1.0 128 bits IDEA-CBC-SHA
       Accepted TLSv1.0 128 bits RC4-SHA
       Accepted TLSv1.0 128 bits RC4-MD5
       Accepted TLSv1.0 112 bits DES-CBC3-SHA


Tagsciphers, ssl

Activities

t-ishii

2021-12-04 06:43

developer   ~0003962

Last edited: 2021-12-04 06:43

 Can you try with Pgpool-II 4.0.9 or later? This should have been fixed in the version:
https://www.pgpool.net/docs/latest/en/html/release-4-0-9.html

venkat_rj

2021-12-04 07:52

reporter   ~0003963

t-ishii,
Thank you so much, we are planning to take 4.0.16.
I will confirm whether it's fixed or not, Please keep this ticket open.
Regards,
Venkat

venkat_rj

2021-12-15 02:46

reporter   ~0003965

We took pgpool version 4.0.16 and upgraded pgpool-II from 4.0.6 to 4.0.16.

Now pgpool starts, but it does not allowing any connections though.

We see below errors in the pgpool log and I'm also listing OpenSSL versions in case it matters.

OpenSSL versions:

    pgpool RPM build system:
       openssl-devel-1.0.1e-58.el6_10.x86_64

    pgpool deployed system:
        OpenSSL 1.0.2k-fips 26 Jan 2017

pgpool log:
$ head pgp-n 50 pgpool_20211214_005340.log
2021-12-14 00:54:04: pid 6705: LOG: Backend status file /opt/config/pgpool0/pgpool_status discarded
2021-12-14 00:54:04: pid 6705: LOG: Setting up socket for 0.0.0.0:5431
2021-12-14 00:54:04: pid 6705: LOG: Setting up socket for :::5431
2021-12-14 00:54:05: pid 6705: LOG: find_primary_node_repeatedly: waiting for finding a primary node
2021-12-14 00:54:05: pid 6705: LOG: find_primary_node: primary node is 0
2021-12-14 00:54:05: pid 7784: LOG: process started
2021-12-14 00:54:05: pid 7785: LOG: process started
2021-12-14 00:54:05: pid 6705: LOG: pgpool-II successfully started. version 4.0.16 (torokiboshi)
2021-12-14 00:54:05: pid 6705: LOG: node status[0]: 1
2021-12-14 00:54:07: pid 7777: LOG: pool_ssl: "SSL_accept": "no shared cipher"
2021-12-14 00:54:07: pid 7759: FATAL: client authentication failed
2021-12-14 00:54:07: pid 7759: DETAIL: no pool_hba.conf entry for host "NNN.NNN.NNN.NNN", user "maskedUserName", database "postgres", SSL off
2021-12-14 00:54:07: pid 7759: HINT: see pgpool log for details
2021-12-14 00:54:07: pid 6705: LOG: child process with pid: 7759 exits with status 512
2021-12-14 00:54:07: pid 6705: LOG: fork a new child process with pid: 8036
2021-12-14 00:54:08: pid 7777: LOG: pool_ssl: "SSL_accept": "no shared cipher"
2021-12-14 00:54:08: pid 7777: FATAL: client authentication failed
2021-12-14 00:54:08: pid 7777: DETAIL: no pool_hba.conf entry for host "NNN.NNN.NNN.NNN", user "maskedUserName", database "postgres", SSL off
2021-12-14 00:54:08: pid 7777: HINT: see pgpool log for details
2021-12-14 00:54:08: pid 6705: LOG: child process with pid: 7777 exits with status 512
2021-12-14 00:54:08: pid 6705: LOG: fork a new child process with pid: 8164
2021-12-14 00:54:09: pid 8164: LOG: pool_ssl: "SSL_accept": "no shared cipher"

venkat_rj

2021-12-15 06:48

reporter   ~0003966

Here is the ciphers list currently set for reference,

ssl_ciphers = 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384' # Added through Consul-template

t-ishii

2021-12-15 09:21

developer   ~0003967

Works for me with the ssl_ciphers you set.
psql -p 11000 -h localhost test
psql (14.1)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

$ openssl version
OpenSSL 1.1.1f 31 Mar 2020

This is Ubuntu 20.

venkat_rj

2021-12-16 01:22

reporter   ~0003968

I tested the same command on my environment.

It works with Postgres port.

But it fails with pgpool port and I see an error i.e. "psql: SSL error: sslv3 alert handshake failure"

Do you think it's related to OpenSSL version?

Log:

#------------------------------------------------------
# Postgres connection test
#------------------------------------------------------

$ $PSQL -p 5432 -h localhost postgres -c '\conninfo'
You are connected to database "postgres" as user "MaskedUsername" on host "localhost" at port "5432".
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256, bits: 128, compression: off)

#------------------------------------------------------
# pgpool connection test
#------------------------------------------------------

$ $PSQL -p 5431 -h localhost postgres -c '\conninfo'
psql: SSL error: sslv3 alert handshake failure
FATAL: client authentication failed
DETAIL: no pool_hba.conf entry for host "::1", user "MaskedUsername", database "postgres", SSL off
HINT: see pgpool log for details

t-ishii

2021-12-16 09:32

developer   ~0003969

Not sure but:

2021-12-14 00:54:07: pid 7777: LOG: pool_ssl: "SSL_accept": "no shared cipher"

It seems this suggests that the supported cyphers in the client side are not overlap in the pgpool side.
I suspect this because when the client succeeded in connecting to PostgreSQL it used ECDHE-RSA-AES128-GCM-SHA256, but it is not in the supported cypher list for pgpool.

venkat_rj

2021-12-17 08:38

reporter   ~0003970

How to resolve this issue?
Do we have to change any pgpool config for SSL parameters?
Do we have to rebuild pgpool-II RPM with different OpenSSL version?

Please help.

t-ishii

2021-12-17 09:29

developer   ~0003971

I am not an OpenSSL expert but I guess you can start with disabling ssl_ciphers in pgpool.conf and see if psql can connect to pgpool. If succeed, psql should show the cipher which pgpool accepts, and you can set it to ssl_ciphers.

t-ishii

2021-12-17 09:37

developer   ~0003972

What does sslscan show for pgpool port?

venkat_rj

2021-12-18 00:08

reporter   ~0003973

Thank you.

Here are the sslscan log for both ports.

Interesting thing I noticed is that pgpool port sslscan log shows most of the SSL/TLS Protocols are disabled.

I'm not sure it's due to pgpool server starts, but does not allow any connections that might be causing sslscan tool to report that SSL/TLS Protocols are disabled or something really an issue that might be manifesting here. Just a thought.

Logs:

#----------------------------------------------------------------
# Postgres port: 5432 version: 11.14
#----------------------------------------------------------------

$ /home/centos/tools/sslscan/sslscan --verbose --starttls-psql <MASKED_HOSTNAME>:5432
Version: 2.0.10-6-gaa0f6c2-static
OpenSSL 1.1.1m-dev xx XXX xxxx

Connected to 10.121.40.195

Some servers will fail to response to SSLv3 ciphers over STARTTLS
If your scan hangs, try using the --tlsall option

Testing SSL server <MASKED_HOSTNAME> on port 5432 using SNI name <MASKED_HOSTNAME>

  SSL/TLS Protocols:
SSLv2 disabled
SSLv3 disabled
TLSv1.0 disabled
TLSv1.1 disabled
TLSv1.2 enabled
TLSv1.3 disabled

  TLS Fallback SCSV:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Server supports TLS Fallback SCSV

  TLS renegotiation:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
use_unsafe_renegotiation_op
Session renegotiation not supported

  TLS Compression:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Compression disabled

  Heartbleed:
TLSv1.2 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.2 128 bits ECDHE-RSA-AES128-GCM-SHA256 Curve P-256 DHE 256
Accepted TLSv1.2 256 bits ECDHE-RSA-AES256-GCM-SHA384 Curve P-256 DHE 256
SSL_get_error(ssl, cipherStatus) said: 1

  Server Key Exchange Group(s):
TLSv1.2 128 bits secp256r1 (NIST P-256)

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength: 2048

Subject: <MASKED_HOSTNAME>.unx.com
Altnames: DNS:localhost, DNS:localhost4, DNS:localhost4.localdomain4, DNS:localhost6, DNS:localhost6.localdomain6, DNS:localhost.localdomain, DNS:<MASKED_HOSTNAME>.unx.com, DNS:<MASKED_HOSTNAME>.unx.com, IP Address:10.121.40.195, IP Address:FD07:AF30:A3FC:7928:250:56FF:FE8C:48B0, IP Address:127.0.0.1
Issuer: <MASKED_USERNAME> Intermediate CA

Not valid before: Dec 14 05:50:42 2021 GMT
Not valid after: Oct 24 00:51:12 2028 GMT
<MASKED_USERNAME>@<MASKED_HOSTNAME>: /opt/<MASKED_USERNAME>//config/data/<MASKED_USERNAME>datasvrc/postgres/pgpool0



#--------------------------------------------------------------------------
# pgpool port: 5431 and version: 4.0.16
#--------------------------------------------------------------------------

$ /home/centos/tools/sslscan/sslscan --verbose --starttls-psql <MASKED_HOSTNAME>:5431
Version: 2.0.10-6-gaa0f6c2-static
OpenSSL 1.1.1m-dev xx XXX xxxx

Connected to 10.121.40.195

Some servers will fail to response to SSLv3 ciphers over STARTTLS
If your scan hangs, try using the --tlsall option

Testing SSL server <MASKED_HOSTNAME> on port 5431 using SNI name <MASKED_HOSTNAME>

  SSL/TLS Protocols:
SSLv2 disabled
SSLv3 disabled
TLSv1.0 disabled
TLSv1.1 disabled
TLSv1.2 disabled
TLSv1.3 disabled

  TLS Fallback SCSV:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Connection failed - unable to determine TLS Fallback SCSV support

  TLS renegotiation:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
use_unsafe_renegotiation_op
Session renegotiation not supported

  TLS Compression:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Compression disabled

  Heartbleed:

  Supported Server Cipher(s):
Certificate information cannot be retrieved.

t-ishii

2021-12-20 10:16

developer   ~0003974

One of the possibilities is, server.key. In my case I used the server.* In the source tree,
src/test/regression/tests/023.ssl_connection/server.*
For your convenience, server.key is attached. These files are generated as described in README file.
--------------------------------------------------------------------------------------------------------------------------------------------------
The sample server.key and server.crt was created by using following commands:

openssl req -new -text -out server.req
openssl rsa -in privkey.pem -out server.key
rm privkey.pem
openssl req -x509 -days 3650 -in server.req -text -key server.key -out server.crt
server.key (1,679 bytes)    
-----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA9OMw8AniQw6hZy4777WIHk6pBvrylIv6Twqf4FdFlSzBGB0h
bchcKgWUC8ZJxZeIrIi6c/uBKOvhzX+d++DFDe81zBK4dAuh6WXUGTibqOHF79Xx
nM+K3r/612z514UQ25/pA+l+8YHenfOzo28ZMci7McDnfeprApghfML1nQ18hStd
gXxxdEKJq1wxrBn9wgv75cKyVBVkQJy/7dKxu3X56dhns05jxD70iiyHUCfdIpfx
9yZJhwNVCKTSRCHhmjOClgmzCGW8qDr0ZPNgYtCY2YIMG762nCLof+brINlxZyzV
FIOq8Td1mExoHZX+H+zLmxec9JIi2rt4RvPefQIDAQABAoIBAD6EfaraKxxJcOUh
hYWlx3FNTZONnz5TGfzxzmz8erQhr84TKcSYIQdNU0VKQu0hyW+anFcdvxSOW6AV
02RJNqVfC1Hk+ZgOnDA0odgqfnq34MtgyATrax2Az24N0R63Rt16zocEJjdLm2Sh
oZu4sirmfvutrquTm+wWoH30W7XTCrd17thq8+5VBumXLW4sr40PWXcfPC1Od7JH
utYZ0lGxr89fhuPpnpLw7V67EdslwmYl8avhzrUlzkXPqufZ33XdWFi/IXoNjMZ2
MFgZFNKHv682nqjrrodyn4iXEzMSj4z4QLTNxoH79HWvCb2HjUpEWTNrfdu3LNCZ
FahqHmECgYEA/5MmEEB2GNlhaMP+NMK/3x9RCfDfleqEdExf0KA6TuISNYUgeEeB
dfczCN8WtxyDMor6fvwINxEAHX64/CompE/ya+Z25VdBQMcLpNVGS01aDdrqWN5B
4qG8OyK3+eAjCnvyCzWsvvpK1I5u65q6+Q+Cw1wQaxYzhsZkEazQ0OkCgYEA9Ut9
m74RMGudjkvJK7JIPLDLGEsko+yrh8IuoGn/wLpUJwmkZFQu8HRqdddMVjwQi4CN
IDScbvyH0uE2yN+gg/BB0eBIyhfrI3xl6FJIQBnI0/7wmb2U82OT0LS18h9n8dAK
+mONcR3zL7XDe8xy1qTS0jjd1QT1QS4YLGPCxHUCgYEAowftCgT00NkqaDhOWr24
w84oVd0P44QcRkvJ+z2atGNGFln74n5KuUOdjJUy2lAX6Q/6xzJi0y3HEwmZW1JQ
IBTXobj8M0Q73eSbKuTZ2INZZOk3AMWW5ckiV97H2V//OlrihgARWCo1ve22GBk2
GFaqpZB+8LDS4bCAeT3yXrECgYEA5rZ3USonLry5d2JOt5u7F+JNU+8xakErYMhC
ZLzuQY6/oewOxBLuB1nn3CiBc0aRZTSnCFiTnkxFUBJmHe9AIXiz37wtmm9+yWSy
0R27ORdHbiYGlQPcekP5fr7Jtw7VDHraKIHEQlWiKwix8dntVXe3luTHuRktuH2r
XO0D/xUCgYEAlQtXyGSl/taUjKyfvfxlFDtpOwBAwUj2CsNzeUd2/5aWkqUrtYz8
JNCPgSLPKDhLpavH0vUEmftF3uDVPxvMQ4JG9MQ7meHgL7AZmtKdU0VI+Av9xiJe
d7A1x6o88gv1TqvGRit2qRxNOT0mzhDcXuR2EIQqUav45NyBokSo9xw=
-----END RSA PRIVATE KEY-----
server.key (1,679 bytes)   

venkat_rj

2021-12-23 06:30

reporter   ~0003976

Are the server.key and server.crt files are it related to ssl_key and ssl_cert parameters in the pgpool.conf?

We do have an internal tool which generates the certs when we deploy the software and also every time we stop & start the Postgres/pgpool cluster.

I don't have access to the tool internals and have to wait for the developer to see it matches with your openssl commands. They are on Xmas break and will be back on 4/Jan/2022.

My current SSL/Certs parameter LOG:

#-------------------------------------------------------
$ egrep 'cert|key' pgpool.conf
#-------------------------------------------------------
#ssl_key = './server.key'
                                   # Path to the SSL private key file
#ssl_cert = './server.cert'
                                   # Path to the SSL public certificate file
#ssl_ca_cert = ''
                                   # containing CA root certificate(s)
#ssl_ca_cert_dir = ''
                                   # Directory containing CA root certificate(s)
wd_authkey = ''
                                    # Authentication key for watchdog communication
ssl_cert = '/opt/config/etc/SSCFW/tls/certs/datasvrc/postgres/pgpool0/cert.pem' # Added through Consul-template
ssl_key = '/opt/config/etc/SSCFW/private/datasvrc/postgres/pgpool0/key.pem' # Added through Consul-template
ssl_ca_cert = '/opt/config/etc/SSCFW/cacerts/trustedcerts.pem' # Added through Consul-template
ssl_ca_cert_dir = '/opt/config/etc/SSCFW/cacerts' # Added through Consul-template

venkat_rj

2022-01-05 07:56

reporter   ~0003980

I did follow up with internal TLS expert about the cert creation and we do follow the same steps you had mentioned.

As per him, cert is not an issue here and It might be issue with values set for ssl_ciphers.

Looks like it is, Please see my tests and its results below.

I tired following settings for both Postgres and pgpool with older version (4.0.6) to make sure it works, before I can try with new version 4.0.16.

But I noticed lots of errors in the pgpool log before it can stable later.

#------------------------------------------------------
# Test1 - Set both Postgres & pgpool
# Results: Works
#------------------------------------------------------
Both Postgres & pgpool set to:
   ssl_ciphers = 'TLSv1.2:!aNULL' # allowed SSL ciphers

#------------------------------------------------------
# Test2 - Set Postgres, but make pgpool value empty
# Results: Does NOT work
#------------------------------------------------------
Postgres:
    ssl_ciphers = 'TLSv1.2:!aNULL' # allowed SSL ciphers

pgpool:
   ssl_ciphers = ' ' # allowed SSL ciphers

#------------------------------------------------------
# Test3 - Set Postgres, but pgpool to default value
# Results: Works
#------------------------------------------------------
Postgres:
    ssl_ciphers = 'TLSv1.2:!aNULL' # allowed SSL ciphers
pgpool:
    ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'


Interesting thing I noticed with sslscan tool with above ssl_ciphers settings and here are the logs.

#---------------------------------
# pgpool: 4.0.16
# - sslscan Logs:
#---------------------------------

#-----------------------------------------
# Test1:
# Both Postgres & pgpoll are set to:
# ssl_ciphers = 'TLSv1.2:!aNULL'
#-----------------------------------------


Version: 2.0.10-6-gaa0f6c2-static
OpenSSL 1.1.1m-dev xx XXX xxxx

Connected to Masked.IP.Address

Some servers will fail to response to SSLv3 ciphers over STARTTLS
If your scan hangs, try using the --tlsall option

Testing SSL server Masked.Host.Name on port 5431 using SNI name Masked.Host.Name

  SSL/TLS Protocols:
SSLv2 disabled
SSLv3 disabled
TLSv1.0 disabled
TLSv1.1 disabled
TLSv1.2 enabled
TLSv1.3 disabled

  TLS Fallback SCSV:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Server supports TLS Fallback SCSV

  TLS renegotiation:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
use_unsafe_renegotiation_op
Session renegotiation not supported

  TLS Compression:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Compression disabled

  Heartbleed:
TLSv1.2 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.2 256 bits AES256-GCM-SHA384
Accepted TLSv1.2 256 bits AES256-SHA256
Accepted TLSv1.2 128 bits AES128-GCM-SHA256
Accepted TLSv1.2 128 bits AES128-SHA256
Accepted TLSv1.2 0 bits NULL-SHA256
SSL_get_error(ssl, cipherStatus) said: 1

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength: 2048

Subject: Masked.Host.Name.unx.com
Altnames: DNS:localhost, DNS:localhost4, DNS:localhost4.localdomain4, DNS:localhost6, DNS:localhost6.localdomain6, DNS:localhost.localdomain, DNS:Masked.Host.Name.unx.com, DNS:Masked.Host.Name.unx.com, IP Address:Masked.IP.Address, IP Address:FD07:AF30:A3FC:7928:250:56FF:FE8C:48B0, IP Address:127.0.0.1
Issuer: SVA Intermediate CA

Not valid before: Jan 4 21:38:39 2022 GMT
Not valid after: Dec 15 17:39:09 2028 GMT


#--------------------------------------------------------------
# Test3:
# Postgres:
# ssl_ciphers = 'TLSv1.2:!aNULL' # allowed SSL ciphers
# pgpool:
# ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'
#--------------------------------------------------------------

/home/centos/tools/sslscan/sslscan --verbose --starttls-psql Masked.Host.Name:5431
Version: 2.0.10-6-gaa0f6c2-static
OpenSSL 1.1.1m-dev xx XXX xxxx

Connected to Masked.IP.Address

Some servers will fail to response to SSLv3 ciphers over STARTTLS
If your scan hangs, try using the --tlsall option

Testing SSL server Masked.Host.Name on port 5431 using SNI name Masked.Host.Name

  SSL/TLS Protocols:
SSLv2 disabled
SSLv3 disabled
TLSv1.0 enabled
TLSv1.1 enabled
TLSv1.2 enabled
TLSv1.3 disabled

  TLS Fallback SCSV:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Server supports TLS Fallback SCSV

  TLS renegotiation:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
use_unsafe_renegotiation_op
Attempting secure_renegotiation_support()
Secure session renegotiation supported

  TLS Compression:
OpenSSL OpenSSL 1.1.1m-dev xx XXX xxxx looks like version 0.9.8m or later; I will try SSL_OP to enable renegotiation
Compression disabled

  Heartbleed:
TLSv1.2 not vulnerable to heartbleed
TLSv1.1 not vulnerable to heartbleed
TLSv1.0 not vulnerable to heartbleed

  Supported Server Cipher(s):
Preferred TLSv1.2 256 bits AES256-GCM-SHA384
Accepted TLSv1.2 256 bits AES256-SHA256
Accepted TLSv1.2 256 bits AES256-SHA
Accepted TLSv1.2 256 bits CAMELLIA256-SHA
Accepted TLSv1.2 128 bits AES128-GCM-SHA256
Accepted TLSv1.2 128 bits AES128-SHA256
Accepted TLSv1.2 128 bits AES128-SHA
Accepted TLSv1.2 128 bits CAMELLIA128-SHA
Accepted TLSv1.2 128 bits SEED-SHA
Accepted TLSv1.2 128 bits RC4-SHA
Accepted TLSv1.2 128 bits RC4-MD5
Accepted TLSv1.2 112 bits DES-CBC3-SHA
SSL_get_error(ssl, cipherStatus) said: 1
Preferred TLSv1.1 256 bits AES256-SHA
Accepted TLSv1.1 256 bits CAMELLIA256-SHA
Accepted TLSv1.1 128 bits AES128-SHA
Accepted TLSv1.1 128 bits CAMELLIA128-SHA
Accepted TLSv1.1 128 bits SEED-SHA
Accepted TLSv1.1 128 bits IDEA-CBC-SHA
Accepted TLSv1.1 128 bits RC4-SHA
Accepted TLSv1.1 128 bits RC4-MD5
Accepted TLSv1.1 112 bits DES-CBC3-SHA
SSL_get_error(ssl, cipherStatus) said: 1
Preferred TLSv1.0 256 bits AES256-SHA
Accepted TLSv1.0 256 bits CAMELLIA256-SHA
Accepted TLSv1.0 128 bits AES128-SHA
Accepted TLSv1.0 128 bits CAMELLIA128-SHA
Accepted TLSv1.0 128 bits SEED-SHA
Accepted TLSv1.0 128 bits IDEA-CBC-SHA
Accepted TLSv1.0 128 bits RC4-SHA
Accepted TLSv1.0 128 bits RC4-MD5
Accepted TLSv1.0 112 bits DES-CBC3-SHA
SSL_get_error(ssl, cipherStatus) said: 1

  SSL Certificate:
Signature Algorithm: sha256WithRSAEncryption
RSA Key Strength: 2048

Subject: Masked.Host.Name.unx.com
Altnames: DNS:localhost, DNS:localhost4, DNS:localhost4.localdomain4, DNS:localhost6, DNS:localhost6.localdomain6, DNS:localhost.localdomain, DNS:Masked.Host.Name.unx.com, DNS:Masked.Host.Name.unx.com, IP Address:Masked.IP.Address, IP Address:FD07:AF30:A3FC:7928:250:56FF:FE8C:48B0, IP Address:127.0.0.1
Issuer: SVA Intermediate CA

Not valid before: Jan 4 22:16:59 2022 GMT
Not valid after: Dec 15 17:17:29 2028 GMT


#--------------------------------------------------------------
# Errors:
# pgpool_20220104_163908.log
#--------------------------------------------------------------

2022-01-04 16:39:30: pid 4474: LOG: Backend status file /opt/sva/config/var/log/datasvrc/postgres/pgpool0/pgpool_status discarded
2022-01-04 16:39:30: pid 4474: LOG: Setting up socket for 0.0.0.0:5431
2022-01-04 16:39:30: pid 4474: LOG: Setting up socket for :::5431
2022-01-04 16:39:31: pid 4474: LOG: find_primary_node_repeatedly: waiting for finding a primary node
2022-01-04 16:39:31: pid 4474: LOG: find_primary_node: primary node is 0
2022-01-04 16:39:31: pid 5593: LOG: process started
2022-01-04 16:39:31: pid 5594: LOG: process started
2022-01-04 16:39:31: pid 4474: LOG: pgpool-II successfully started. version 4.0.16 (torokiboshi)
2022-01-04 16:39:31: pid 4474: LOG: node status[0]: 1
2022-01-04 16:39:46: pid 5592: LOG: forked new pcp worker, pid=9082 socket=7
2022-01-04 16:39:46: pid 9082: LOG: received failback request for node_id: 0 from pid [9082]
2022-01-04 16:39:46: pid 9082: LOG: signal_user1_to_parent_with_reason(0)
2022-01-04 16:39:46: pid 5592: LOG: PCP process with pid: 9082 exit with SUCCESS.
2022-01-04 16:39:46: pid 5592: LOG: PCP process with pid: 9082 exits with status 0
2022-01-04 16:39:46: pid 4474: LOG: Pgpool-II parent process received SIGUSR1
2022-01-04 16:39:46: pid 4474: LOG: Pgpool-II parent process has received failover request
2022-01-04 16:39:46: pid 4474: LOG: invalid failback request, status: [2] of node id : 0 is invalid for failback
2022-01-04 16:40:57: pid 5500: LOG: pool_ssl: "SSL_read": "ssl handshake failure"
2022-01-04 16:40:57: pid 5500: ERROR: unable to read data from frontend
2022-01-04 16:40:57: pid 5500: DETAIL: socket read failed with error "Success"
2022-01-04 16:40:57: pid 5391: LOG: pool_ssl: "SSL_accept": "unknown protocol"
2022-01-04 16:40:57: pid 5391: ERROR: failed while reading startup packet
2022-01-04 16:40:57: pid 5391: DETAIL: incorrect packet length (83918847)
2022-01-04 16:40:57: pid 5559: LOG: pool_ssl: "SSL_accept": "wrong version number"
2022-01-04 16:40:57: pid 5587: LOG: pool_ssl: "SSL_accept": "no shared cipher"
2022-01-04 16:40:57: pid 5587: ERROR: unable to read data from frontend
2022-01-04 16:40:57: pid 5587: DETAIL: socket read failed with error "Connection reset by peer"
2022-01-04 16:40:57: pid 5559: LOG: pool_ssl: "SSL_accept": "no shared cipher"
2022-01-04 16:40:57: pid 5559: ERROR: unable to read data from frontend
2022-01-04 16:40:57: pid 5559: DETAIL: socket read failed with error "Connection reset by peer"
2022-01-04 16:40:57: pid 5587: LOG: pool_ssl: "SSL_accept": "no SSL error reported"
2022-01-04 16:40:57: pid 5559: LOG: pool_ssl: "SSL_accept": "no shared cipher"
2022-01-04 16:40:57: pid 5559: ERROR: unable to read data from frontend
2022-01-04 16:40:57: pid 5559: DETAIL: socket read failed with error "Connection reset by peer"
2022-01-04 16:40:57: pid 5353: LOG: pool_ssl: "SSL_read": "no SSL error reported"
2022-01-04 16:40:57: pid 5353: ERROR: unable to read data from frontend
2022-01-04 16:40:57: pid 5353: DETAIL: socket read failed with error "Success"
2022-01-04 16:40:57: pid 5531: LOG: pool_ssl: "SSL_accept": "inappropriate fallback"
2022-01-04 16:40:57: pid 5559: LOG: pool_ssl: "SSL_accept": "no shared cipher"
2022-01-04 16:40:57: pid 5559: LOG: pool_ssl: "SSL_accept": "no shared cipher"
2022-01-04 16:40:57: pid 4681: LOG: pool_ssl: "SSL_accept": "no shared cipher"
2022-01-04 16:40:57: pid 4681: ERROR: failed while reading startup packet
2022-01-04 16:40:57: pid 4681: DETAIL: incorrect packet length (402850556)
2022-01-04 16:40:57: pid 4681: LOG: pool_ssl: "SSL_read": "no SSL error reported"
2022-01-04 16:40:57: pid 4681: ERROR: unable to read data from frontend
2022-01-04 16:40:57: pid 4681: DETAIL: socket read failed with error "Success"
2022-01-04 16:40:57: pid 5011: LOG: pool_ssl: "SSL_read": "no SSL error reported"
2022-01-04 16:40:57: pid 5011: ERROR: unable to read data from frontend
2022-01-04 16:40:57: pid 5011: DETAIL: socket read failed with error "Success"
2022-01-04 16:40:57: pid 5559: LOG: pool_ssl: "SSL_read": "no SSL error reported"
2022-01-04 16:40:57: pid 5559: ERROR: unable to read data from frontend
2022-01-04 16:40:57: pid 5559: DETAIL: socket read failed with error "Success"
2022-01-04 16:40:57: pid 5011: LOG: pool_ssl: "SSL_read": "no SSL error reported"
...
...
...
2022-01-04 16:42:42: pid 5539: LOG: pool_ssl: "SSL_read": "no SSL error reported"
2022-01-04 16:42:42: pid 5539: ERROR: unable to read data from frontend
2022-01-04 16:42:42: pid 5539: DETAIL: socket read failed with error "Success"
2022-01-04 16:44:42: pid 4474: LOG: child process with pid: 5581 exits with status 256
2022-01-04 16:44:42: pid 4474: LOG: fork a new child process with pid: 23352
2022-01-04 16:44:46: pid 4474: LOG: child process with pid: 5560 exits with status 256

t-ishii

2022-01-10 17:54

developer   ~0003982

I tried with following with Pgpool-II 4.0.17/PostgreSQL 14.0 on Ubuntu20 and saw no errors in pgpool log.

#--------------------------------------------------------------
# Test3:
# Postgres:
# ssl_ciphers = 'TLSv1.2:!aNULL' # allowed SSL ciphers
# pgpool:
# ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL'
#--------------------------------------------------------------

t-ishii

2022-05-19 11:29

developer   ~0004038

May I close this issue?

t-ishii

2022-05-23 17:11

developer   ~0004046

Per comment at: https://www.pgpool.net/mantisbt/view.php?id=712#c4042
I'm going to close this issue.

Issue History

Date Modified Username Field Change
2021-12-04 01:21 venkat_rj New Issue
2021-12-04 01:21 venkat_rj Tag Attached: ciphers
2021-12-04 01:21 venkat_rj Tag Attached: ssl
2021-12-04 06:43 t-ishii Note Added: 0003962
2021-12-04 06:43 t-ishii Note Edited: 0003962
2021-12-04 06:44 t-ishii Assigned To => t-ishii
2021-12-04 06:44 t-ishii Status new => assigned
2021-12-04 06:44 t-ishii Status assigned => feedback
2021-12-04 07:52 venkat_rj Note Added: 0003963
2021-12-04 07:52 venkat_rj Status feedback => assigned
2021-12-08 19:45 t-ishii Status assigned => feedback
2021-12-15 02:46 venkat_rj Note Added: 0003965
2021-12-15 02:46 venkat_rj Status feedback => assigned
2021-12-15 06:48 venkat_rj Note Added: 0003966
2021-12-15 09:21 t-ishii Note Added: 0003967
2021-12-16 01:22 venkat_rj Note Added: 0003968
2021-12-16 09:32 t-ishii Note Added: 0003969
2021-12-17 08:38 venkat_rj Note Added: 0003970
2021-12-17 09:29 t-ishii Note Added: 0003971
2021-12-17 09:37 t-ishii Note Added: 0003972
2021-12-18 00:08 venkat_rj Note Added: 0003973
2021-12-20 10:16 t-ishii Note Added: 0003974
2021-12-20 10:16 t-ishii File Added: server.key
2021-12-23 06:30 venkat_rj Note Added: 0003976
2022-01-05 07:56 venkat_rj Note Added: 0003980
2022-01-10 17:54 t-ishii Note Added: 0003982
2022-05-19 11:29 t-ishii Note Added: 0004038
2022-05-23 17:11 t-ishii Note Added: 0004046
2022-05-23 17:12 t-ishii Status assigned => closed