[Pgpool-general] "FATAL: invalid frontend message type 4"

[Tatsuo Ishii]

>  From what I've seen, only a PANIC error means that the postgres server
> is down. I think a FATAL error should trigger a failover only if it's
> received in response to an authentication attempt.
> 
>  About the "type 4" message I think my client/frontend is sending the
> corrupt messages (I still don't know why though) and pgpool is just
> forwarding them to the backends. I know this would mean that it's not a
> pgpool error, but I'm attaching a patch which would make sure pgpool
> doesn't forward any unrecognized message to its backends.

Thanks. Patch committed to CVS HEAD and 2.2-STATBLE tree.
--
Tatsuo Ishii
SRA OSS, Inc. Japan

>  It checks the kind of message received from the frontend before calling
> SimpleForwardToBackend(), only for messages that should be expected
> there (no startup, ssl, password ('p'), or cancel messages).
> 
>  If it receives an unrecognized message, pgpool will log a "
> "ProcessFrontendResponse: unknown message type" error (this is the
> current behavior). 
> 
> 
>  Regards
> 
> 
> On Sun, 2010-01-03 at 21:11 +0900, Tatsuo Ishii wrote:
> > I looked into this but couldn't find why pgpool-II sent 4 to
> > PostgreSQL. Any test case to reproduce the error please.
> > 
> > In the mean time I start to think that pgpool-II should not trigger
> > failover if failed to read data from backend. Looking into PostgreSQL
> > I found lots of places where FATAL errors are generated and PostgreSQL
> > exits. If it exits, pgpool-II will detect a read error from socket
> > that will cause failover. I feel this behavior is too fragile. I'm
> > looking forward to tweak pool_stream.c so that it does not failover.
> > Comments?
> > --
> > Tatsuo Ishii
> > SRA OSS, Inc. Japan
> > 
> > >  pgpool-II version is 2.2.4 compiled against libpq 8.1.15, and postgres
> > > version is 8.1.15.
> > > 
> > >  Both run on Debian etch amd64.
> > > 
> > >  I haven't been able to create a test case yet, but I've been able to
> > > verify that it always happens after a COMMIT (like my previous example)
> > > or a BEGIN.
> > > 
> > > 
> > >  Regards
> > > 
> > > On Wed, 2009-12-30 at 17:41 +0900, Tatsuo Ishii wrote:
> > > > PostgreSQL, pgpool-II version?
> > > > 
> > > > Do you have self contained test case?
> > > > --
> > > > Tatsuo Ishii
> > > > SRA OSS, Inc. Japan
> > > > 
> > > > >  On this same issue, I found that pgpool detected an error just before
> > > > > failing a backend as a response to a COMMIT (this is on a different
> > > > > setup than the one from my previous post). This is the relevant part
> > > > > from the pgpool log:
> > > > > 
> > > > > 
> > > > > 2009-12-26 00:13:04 DEBUG: pid 16329: ReadyForQuery: transaction state:
> > > > > T
> > > > > 2009-12-26 00:13:04 DEBUG: pid 16329: do_command: Query: COMMIT
> > > > > 2009-12-26 00:13:04 DEBUG: pid 16329: wait_for_query_response: waiting
> > > > > for backend 1 completing the query
> > > > > 2009-12-26 00:13:04 DEBUG: pid 16329: detect_error: kind: E
> > > > > 2009-12-26 00:13:04 DEBUG: pid 16329: do_command: kind: E
> > > > > 2009-12-26 00:13:04 ERROR: pid 16329: pool_read: read failed (Connection
> > > > > reset by peer)
> > > > > 2009-12-26 00:13:04 LOG:   pid 16329: notice_backend_error: 1 fail over
> > > > > request from pid 16329
> > > > > 
> > > > > 
> > > > >  I can see the commit on the postgres log of the failed backend,
> > > > > followed by the "invalid frontend message type 4" which closes the
> > > > > session and the connection to the client. That would explain why the
> > > > > pool_read call on pgpool fails, but I still can't figure out why does
> > > > > the backend receive that invalid message.
> > > > > 
> > > > > 
> > > > > 
> > > > >  Regards
> > > > > 
> > > > > 
> > > > > On Thu, Dec 24, 2009 at 11:38 AM, Xavier Noguer <xnoguer at antica.cl>
> > > > > wrote:
> > > > > >
> > > > > >  Hello,
> > > > > >
> > > > > >  I encountered a problem with a pgpool installation. Pgpool failed and
> > > > > > detached a backend, but I don't know the reason because that
> > > > > particular
> > > > > > pgpool server is not writing to its log (problem with cronolog, and I
> > > > > > can't restart pgpool since its a live system).
> > > > > >
> > > > > >  What I did find was a suspicious message in the postgres log of the
> > > > > > failed backend. Here are the relevant lines of the log:
> > > > > >
> > > > > >
> > > > > > 2009-12-23 15:07:20 CLST, sess LOG:  duration: 39.809 ms  statement:
> > > > > > EXECUTE <unnamed>  [PREPARE:  COMMIT]
> > > > > > 2009-12-23 15:07:20 CLST, sess FATAL:  invalid frontend message type 4
> > > > > > 2009-12-23 15:07:20 CLST, sess LOG:  unexpected EOF on client
> > > > > connection
> > > > > > 2009-12-23 15:07:20 CLST, sess LOG:  unexpected EOF on client
> > > > > connection
> > > > > > 2009-12-23 15:07:20 CLST, sess LOG:  unexpected EOF on client
> > > > > connection
> > > > > > 2009-12-23 15:07:20 CLST, sess LOG:  unexpected EOF on client
> > > > > connection
> > > > > > 2009-12-23 15:07:20 CLST, sess LOG:  unexpected EOF on client
> > > > > connection
> > > > > > 2009-12-23 15:07:20 CLST, sess LOG:  unexpected EOF on client
> > > > > connection
> > > > > > 2009-12-23 15:07:20 CLST, sess LOG:  unexpected EOF on client
> > > > > connection
> > > > > >
> > > > > >
> > > > > >  Since this happened at the same time as the failing of the backend
> > > > > I'm
> > > > > > assuming this condition made postgres close the connection to pgpool
> > > > > and
> > > > > > so triggered the failing and detaching of the backend.
> > > > > >
> > > > > >  Does anybody know what message type 4 is ?
> > > > > >
> > > > > >
> > > > > >  Regards
> > > > > >
> > > > > >
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > Pgpool-general mailing list
> > > > > Pgpool-general at pgfoundry.org
> > > > > http://pgfoundry.org/mailman/listinfo/pgpool-general
> > >