[pgpool-general: 6971] Re: Keberos issues

[Bo Peng]

On Mon, 6 Apr 2020 03:57:44 +0200
Oliver Freyermuth <freyermuth at physik.uni-bonn.de> wrote:

> Dear Bo,
> 
> Am 06.04.20 um 01:55 schrieb Bo Peng:
> > Hello,
> > 
> > On Fri, 27 Mar 2020 16:47:00 +0100
> > Oliver Freyermuth <freyermuth at physik.uni-bonn.de> wrote:
> > 
> >> Dear Bo,
> >>
> >> thanks for your reply!
> >>
> >> Am 27.03.20 um 07:24 schrieb Bo Peng:
> >>> Hello,
> >>>
> >>> On Fri, 20 Mar 2020 09:57:23 +0100
> >>> Oliver Freyermuth <freyermuth at physik.uni-bonn.de> wrote:
> >>>
> >>>> Dear PGPoolers,
> >>>>
> >>>> after setting up PGPool[0] with md5 auth and having the pool_hba correctly set up,
> >>>> and login working fine, I encountered one client which could not log in with the following message
> >>>> (just using the CLI psql tool):
> >>>> --------------------
> >>>> $ psql -l "host=my_pgpool_node port=5432 dbname=testdb user=testuser"
> >>>> psql: error: could not connect to server: server closed the connection unexpectedly
> >>>>           This probably means the server terminated abnormally
> >>>>           before or while processing the request.
> >>>> --------------------
> >>>> So the client does not see any underlying cause.
> >>>>
> >>>> Checking with strace, I found the client seems to try Kerberos 5 first, and apparently never falls back to password auth
> >>>> (it does not prompt for a password).
> >>>>
> >>>> Checking the pgpool-II debug logs:
> >>>> --------------------
> >>>> Mar 20 09:46:34 pgsql pgpool[25144]: [949-1] 2020-03-20 09:46:34: pid 25144: LOG:  new connection received
> >>>> Mar 20 09:46:34 pgsql pgpool[25144]: [949-2] 2020-03-20 09:46:34: pid 25144: DETAIL:  connecting host=client_ip port=54946
> >>>> Mar 20 09:46:34 pgsql pgpool[25144]: [949-3] 2020-03-20 09:46:34: pid 25144: LOCATION:  child.c:2166
> >>>> Mar 20 09:46:34 pgsql pgpool[25144]: [951-1] 2020-03-20 09:46:34: pid 25144: FATAL:  client authentication failed
> >>>> Mar 20 09:46:34 pgsql pgpool[25144]: [951-2] 2020-03-20 09:46:34: pid 25144: DETAIL:  no pool_hba.conf entry for host "client_ip", user "", database "", SSL off
> >>>> Mar 20 09:46:34 pgsql pgpool[25144]: [951-3] 2020-03-20 09:46:34: pid 25144: HINT:  see pgpool log for details
> >>>> --------------------
> >>>> it's quite curious that "user" and "database" are empty.
> >>>>
> >>>> Reading through the code, notably check_hba() and ClientAuthentication() in src/auth/pool_hba.c,
> >>>> my best guess is that a client connecting with Kerberos is implicitly rejected and not asked to retry with another method.
> >>>> Sadly, there seems to be no way to disallow psql from trying KRB5 (and it prefers it if possible).
> >>>>
> >>>> So the only approach I've found for the moment is to destroy KRB credentials to force psql to use MD5 auth,
> >>>> that does indeed work, but any client who is capable of KRB5 auth will silently fail to authenticate with MD5 against PGPool
> >>>> since the connection is dropped before it can try.
> >>>>
> >>>> I'd interpret this as a bug, but before opening an issue, wanted to ask here if this problem is known
> >>>> and / or my understanding is correct.
> >>>>
> >>>> Cheers,
> >>>> 	Oliver
> >>>>
> >>>> [0] Version: pgpool-II-pg11-4.0.8-1.pgdg.rhel7 (on CentOS 7)
> >>>>
> >>>
> >>> I think it is a existing issue on CentOS6.
> >>>
> >>> [FAQ]
> >>> https://www.pgpool.net/mediawiki/index.php/FAQ#Connection_failed_in_CentOS6
> >>>
> >>> I tried on CentOS7 but I couldn't reproduce this issue.
> >>> Could you share your pgpool.conf, pool_hba.conf and pg_hba.conf?
> >>>
> >>
> >> It seems I forgot to mention what was used on the client. The client is using the PostgreSQL 12 client tools, compiled with GSSAPI / Kerberos 5 support
> >> (it's a Gentoo Linux system).
> >>
> >> I performed the following tests now:
> >> - Trying "export PGGSSENCMODE=disable" on that client.
> >>    Indeed, that worked around the problem!
> >> - Trying to reproduce with a CentOS 7 client. I couldn't.
> >> - Trying to reproduce with a Ubuntu 18.04 client. I couldn't.
> >>
> >> Since this means the server configuration is not the cause, I did not (yet) include the config files, I can still provide them if you want.
> >>
> >> So I agree, this should be the FAQ entry you linked. Since Gentoo uses a rather "vanilla" PostgreSQL build, I checked which patches Debian / Ubuntu and CentOS 7 apply,
> >> but could not find anything related.
> >> So the cause likely is that old CentOS 6 and rather modern Gentoo both try GSSAPI / Kerberos 5 first. I wonder if we can pinpoint the actual cause of that ―
> >> the only difference I see right now is that Gentoo is already at PostgreSQL 12, and already ships the 1.17 Kerberos libraries.
> >>
> >> Are you aware of any other components involved?
> >> I can certainly pinpoint this if I have an idea of where to start looking. Another solution would probably be if pgpool denied GSSAPI and caused the client to rather use password auth
> >> (if that is possible?).
> > 
> > No.
> > Because Pgpool-II doesn't support GSSAPI, you need to disable GSSAPI using "export PGGSSENCMODE=disable".
> 
> I see. While that certainly works for me, I find it strange that CentOS 7 and Ubuntu 18.04 (which both should support GSSAPI, no?) so not need this explicitly,
> while both old CentOS 6 and modern Gentoo need it. 
> Is there something obvious I am missing? 
> 
> Also, is there a way (protocol-wise) for a server to deny GSSAPI and force clients to fall back to other techniques to prevent this? 

Sorry for late response.

I think it is caused by differences between GSS on CentOS 6/7/8.
So far I don't know a other way to prevent this issue.
 
> Cheers and thanks,
> 	Oliver
> 
> >  
> >> Cheers,
> >> 	Oliver
> >>
> > 
> > 
> 


-- 
Bo Peng <pengbo at sraoss.co.jp>
SRA OSS, Inc. Japan