[pgpool-hackers: 1522] Re: Proposal: minimize process restart when fail over occurs

Tue Apr 26 17:49:48 JST 2016

> On Tue, Apr 26, 2016 at 1:35 PM, Tatsuo Ishii <ishii at postgresql.org> wrote:
> 
>> From: Ahsan Hadi <ahsan.hadi at enterprisedb.com>
>> Subject: Re: [pgpool-hackers: 1504] Re: Proposal: minimize process restart
>> when fail over occurs
>> Date: Tue, 26 Apr 2016 13:10:48 +0500
>> Message-ID: <
>> CAGX_dr1RLL2MwcKdfN+Vi-8xAzm+Hsoy36Nqrwqc+nr88C+2Fg at mail.gmail.com>
>>
>> > On Tue, Apr 19, 2016 at 4:20 PM, Ahsan Hadi <ahsan.hadi at enterprisedb.com
>> >
>> > wrote:
>> >
>> >> Hi Ishii.
>> >>
>> >> As per my understanding, the goal of this feature is to avoid doing the
>> >> following in-case a stand-by node failure :
>> >>
>> >> a) disconnect all the client connections
>> >> b) not accept new client connection until timeout
>> >>
>> >> The objective is continue serving new client connection and continue
>> >> serving existing connection (unless they were connected to the failed
>> node)
>> >> in-case of a failover.
>> >>
>> >> Is my understanding correct?
>>
>> My recent committ tried to address a), not b).
>>
> 
> okay...do you believe both a and b will be done before the feature freeze
> date for pgpool II 3.6?

My plan is addressing b) in 3.6 as well. However, I haven't looked
into b) yet. So the problem could be harder than I thought...

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

>> > Hi Ishii-san,
>> >
>> > Will this feature serve the above objectives?
>> >
>> > -- Ahsan
>> >
>> >
>> >>
>> >> -- Ahsan
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Sun, Apr 17, 2016 at 2:15 PM, Tatsuo Ishii <ishii at postgresql.org>
>> >> wrote:
>> >>
>> >>> Ok, I have succeeded in not restart child process when certain
>> >>> conditions are met.
>> >>>
>> >>>  - Streaming replication mode
>> >>>  - pcp_detach_node is used
>> >>>  - does not use the load balance node (that means the process does not
>> >>>    issue queries to the load balance node)
>> >>>  - the node is not primary node
>> >>>
>> >>> At this point this just enhance allow following use cases (we assume
>> >>> that pcp_detach_node detaches DB node N):
>> >>>
>> >>> 1) Lucky users connecting to the database server N are not affected by
>> >>>    the pcp_detach_node.
>> >>>
>> >>> 2) Planned DB shutdown. For demonstration purpose, I use pgbench -C.
>> >>>
>> >>>    - start pgbench -C
>> >>>    - change pgpool.conf to change the weight to 0 for backend N.
>> >>>    - pgpool reload
>> >>>    - pcp_detach_node N
>> >>>    - pgbench happily continues the benchmark
>> >>>
>> >>> Probably #2 is practically useful.
>> >>>
>> >>> I think we could expand this to certain cases such as PostgreSQL
>> >>> is shutdown by admin. Will continue to work on this.
>> >>> --
>> >>> Tatsuo Ishii
>> >>> SRA OSS, Inc. Japan
>> >>> English: http://www.sraoss.co.jp/index_en.php
>> >>> Japanese:http://www.sraoss.co.jp
>> >>>
>> >>> > I have moved forward a little bit with this. At this point I have
>> just
>> >>> > a created necessary infrastructure to deal with the goal. See
>> >>> > [pgpool-committers: 3127] for more details.
>> >>> >
>> >>> > Best regards,
>> >>> > --
>> >>> > Tatsuo Ishii
>> >>> > SRA OSS, Inc. Japan
>> >>> > English: http://www.sraoss.co.jp/index_en.php
>> >>> > Japanese:http://www.sraoss.co.jp
>> >>> >
>> >>> >> So this is a proposal for pgpool-II 3.6.
>> >>> >>
>> >>> >> I already did some discussion on this:
>> >>> >>
>> >>> >> From: Tatsuo Ishii <ishii at postgresql.org>
>> >>> >> Subject: [pgpool-hackers: 1413] Item #11, torward pgpool-II 3.6
>> >>> >> Date: Fri, 19 Feb 2016 12:03:12 +0900 (JST)
>> >>> >> Message-ID: <
>> 20160219.120312.816223524770393776.t-ishii at sraoss.co.jp>
>> >>> >>
>> >>> >> Here is a more or less formal proposal which is replacing it.
>> >>> >>
>> >>> >> Goal:
>> >>> >>
>> >>> >> Currently pgpool-II kills all child process when fail over (or
>> switch
>> >>> >> over by pcp_detach_node) occurs. Of course this leads to
>> disconnecting
>> >>> >> of all existing client connections because the peer process which
>> >>> >> client is connecting is gone. This proposal is seeking a way to
>> >>> >> minimize such session disconnections.
>> >>> >>
>> >>> >> o Precondition:
>> >>> >>
>> >>> >> I assume this proposal is for streaming replication mode only. Maybe
>> >>> >> we could expand this for other modes in the future. I also assume
>> the
>> >>> >> broken server is not primary.
>> >>> >>
>> >>> >> o Consideration:
>> >>> >>
>> >>> >> What is the reason why we need to kill child process? Basically the
>> >>> >> problem is the retry in the TCP/IP stack layer when the connection
>> >>> >> goes wrong, for example, the network cable is pulled out. In this
>> case
>> >>> >> the only way to stop the retry is restarting the process.
>> >>> >>
>> >>> >> There are several chances where we could avoid the restarting:
>> >>> >>
>> >>> >> 1) Knowing that we are not dealing with a fail over caused by the
>> >>> >> cabling problem. There are at least two cases we know the problem is
>> >>> >> not a cabling:
>> >>> >>
>> >>> >>  a) the fail over is triggered by pcp_detach_node.
>> >>> >>
>> >>> >>  b) the fail over is triggered by posmaster shutdown.
>> >>> >>
>> >>> >> For other cases we need to find a way to know that the problem is a
>> >>> >> cabling or not. Currently we use timeout to detect such that
>> >>> >> situation. So if we could know if the timeout is occurred or not,
>> then
>> >>> >> we could know the problem is a cabling or not.
>> >>> >>
>> >>> >> 2) Once we succeed in #1, next thing we need to do is, whether a
>> >>> >> session in question is using the broken server. This is fairly easy
>> >>> >> because we already have the info on shared memory. If the session
>> uses
>> >>> >> the broken server, then we need to restart the process. No way.
>> Other
>> >>> >> case we just close a connection to the broken backend (if any).
>> >>> >>
>> >>> >> o Things we need to do:
>> >>> >>
>> >>> >> - Invent a way to know if the fail over request is created by
>> >>> >>   pcp_detach_node. Probably we add a new flag to the fail over
>> request
>> >>> >>   packet to indicate whether the origin of the request is
>> >>> >>   pcp_detach_node or not.
>> >>> >>
>> >>> >> - The same technique above can be used for the admin PostgreSQL
>> >>> >>   shutdown case.
>> >>> >>
>> >>> >> - Create a API to deal with connections using the broken server.
>> >>> >>
>> >>> >> o What are the benefit once above proposal is implemented?
>> >>> >>
>> >>> >> - If conditions below are met, the user session can be survives
>> after
>> >>> fail over.
>> >>> >>
>> >>> >>  - Operated in streaming replication mode
>> >>> >>
>> >>> >>  - The failed server is not primary
>> >>> >>
>> >>> >>  - The session does not connect to the broken broken standby server
>> >>> >>
>> >>> >> Comments, opinions?
>> >>> >> --
>> >>> >> Tatsuo Ishii
>> >>> >> SRA OSS, Inc. Japan
>> >>> >> English: http://www.sraoss.co.jp/index_en.php
>> >>> >> Japanese:http://www.sraoss.co.jp
>> >>> >> _______________________________________________
>> >>> >> pgpool-hackers mailing list
>> >>> >> pgpool-hackers at pgpool.net
>> >>> >> http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>> >>> > _______________________________________________
>> >>> > pgpool-hackers mailing list
>> >>> > pgpool-hackers at pgpool.net
>> >>> > http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>> >>> _______________________________________________
>> >>> pgpool-hackers mailing list
>> >>> pgpool-hackers at pgpool.net
>> >>> http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Ahsan Hadi
>> >> Snr Director Product Development
>> >> EnterpriseDB Corporation
>> >> The Enterprise Postgres Company
>> >>
>> >> Phone: +92-51-8358874
>> >> Mobile: +92-333-5162114
>> >>
>> >> Website: www.enterprisedb.com
>> >> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>> >> Follow us on Twitter: http://www.twitter.com/enterprisedb
>> >>
>> >> This e-mail message (and any attachment) is intended for the use of the
>> >> individual or entity to whom it is addressed. This message contains
>> >> information from EnterpriseDB Corporation that may be privileged,
>> >> confidential, or exempt from disclosure under applicable law. If you are
>> >> not the intended recipient or authorized to receive this for the
>> intended
>> >> recipient, any use, dissemination, distribution, retention, archiving,
>> or
>> >> copying of this communication is strictly prohibited. If you have
>> received
>> >> this e-mail in error, please notify the sender immediately by reply
>> e-mail
>> >> and delete this message.
>> >>
>> >
>> >
>> >
>> > --
>> > Ahsan Hadi
>> > Snr Director Product Development
>> > EnterpriseDB Corporation
>> > The Enterprise Postgres Company
>> >
>> > Phone: +92-51-8358874
>> > Mobile: +92-333-5162114
>> >
>> > Website: www.enterprisedb.com
>> > EnterpriseDB Blog: http://blogs.enterprisedb.com/
>> > Follow us on Twitter: http://www.twitter.com/enterprisedb
>> >
>> > This e-mail message (and any attachment) is intended for the use of the
>> > individual or entity to whom it is addressed. This message contains
>> > information from EnterpriseDB Corporation that may be privileged,
>> > confidential, or exempt from disclosure under applicable law. If you are
>> > not the intended recipient or authorized to receive this for the intended
>> > recipient, any use, dissemination, distribution, retention, archiving, or
>> > copying of this communication is strictly prohibited. If you have
>> received
>> > this e-mail in error, please notify the sender immediately by reply
>> e-mail
>> > and delete this message.
>>
> 
> 
> 
> -- 
> Ahsan Hadi
> Snr Director Product Development
> EnterpriseDB Corporation
> The Enterprise Postgres Company
> 
> Phone: +92-51-8358874
> Mobile: +92-333-5162114
> 
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
> 
> This e-mail message (and any attachment) is intended for the use of the
> individual or entity to whom it is addressed. This message contains
> information from EnterpriseDB Corporation that may be privileged,
> confidential, or exempt from disclosure under applicable law. If you are
> not the intended recipient or authorized to receive this for the intended
> recipient, any use, dissemination, distribution, retention, archiving, or
> copying of this communication is strictly prohibited. If you have received
> this e-mail in error, please notify the sender immediately by reply e-mail
> and delete this message.