[pgpool-hackers: 1518] Re: Proposal: minimize process restart when fail over occurs

Ahsan Hadi ahsan.hadi at enterprisedb.com
Tue Apr 26 17:10:48 JST 2016


On Tue, Apr 19, 2016 at 4:20 PM, Ahsan Hadi <ahsan.hadi at enterprisedb.com>
wrote:

> Hi Ishii.
>
> As per my understanding, the goal of this feature is to avoid doing the
> following in-case a stand-by node failure :
>
> a) disconnect all the client connections
> b) not accept new client connection until timeout
>
> The objective is continue serving new client connection and continue
> serving existing connection (unless they were connected to the failed node)
> in-case of a failover.
>
> Is my understanding correct?
>

Hi Ishii-san,

Will this feature serve the above objectives?

-- Ahsan


>
> -- Ahsan
>
>
>
>
>
> On Sun, Apr 17, 2016 at 2:15 PM, Tatsuo Ishii <ishii at postgresql.org>
> wrote:
>
>> Ok, I have succeeded in not restart child process when certain
>> conditions are met.
>>
>>  - Streaming replication mode
>>  - pcp_detach_node is used
>>  - does not use the load balance node (that means the process does not
>>    issue queries to the load balance node)
>>  - the node is not primary node
>>
>> At this point this just enhance allow following use cases (we assume
>> that pcp_detach_node detaches DB node N):
>>
>> 1) Lucky users connecting to the database server N are not affected by
>>    the pcp_detach_node.
>>
>> 2) Planned DB shutdown. For demonstration purpose, I use pgbench -C.
>>
>>    - start pgbench -C
>>    - change pgpool.conf to change the weight to 0 for backend N.
>>    - pgpool reload
>>    - pcp_detach_node N
>>    - pgbench happily continues the benchmark
>>
>> Probably #2 is practically useful.
>>
>> I think we could expand this to certain cases such as PostgreSQL
>> is shutdown by admin. Will continue to work on this.
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese:http://www.sraoss.co.jp
>>
>> > I have moved forward a little bit with this. At this point I have just
>> > a created necessary infrastructure to deal with the goal. See
>> > [pgpool-committers: 3127] for more details.
>> >
>> > Best regards,
>> > --
>> > Tatsuo Ishii
>> > SRA OSS, Inc. Japan
>> > English: http://www.sraoss.co.jp/index_en.php
>> > Japanese:http://www.sraoss.co.jp
>> >
>> >> So this is a proposal for pgpool-II 3.6.
>> >>
>> >> I already did some discussion on this:
>> >>
>> >> From: Tatsuo Ishii <ishii at postgresql.org>
>> >> Subject: [pgpool-hackers: 1413] Item #11, torward pgpool-II 3.6
>> >> Date: Fri, 19 Feb 2016 12:03:12 +0900 (JST)
>> >> Message-ID: <20160219.120312.816223524770393776.t-ishii at sraoss.co.jp>
>> >>
>> >> Here is a more or less formal proposal which is replacing it.
>> >>
>> >> Goal:
>> >>
>> >> Currently pgpool-II kills all child process when fail over (or switch
>> >> over by pcp_detach_node) occurs. Of course this leads to disconnecting
>> >> of all existing client connections because the peer process which
>> >> client is connecting is gone. This proposal is seeking a way to
>> >> minimize such session disconnections.
>> >>
>> >> o Precondition:
>> >>
>> >> I assume this proposal is for streaming replication mode only. Maybe
>> >> we could expand this for other modes in the future. I also assume the
>> >> broken server is not primary.
>> >>
>> >> o Consideration:
>> >>
>> >> What is the reason why we need to kill child process? Basically the
>> >> problem is the retry in the TCP/IP stack layer when the connection
>> >> goes wrong, for example, the network cable is pulled out. In this case
>> >> the only way to stop the retry is restarting the process.
>> >>
>> >> There are several chances where we could avoid the restarting:
>> >>
>> >> 1) Knowing that we are not dealing with a fail over caused by the
>> >> cabling problem. There are at least two cases we know the problem is
>> >> not a cabling:
>> >>
>> >>  a) the fail over is triggered by pcp_detach_node.
>> >>
>> >>  b) the fail over is triggered by posmaster shutdown.
>> >>
>> >> For other cases we need to find a way to know that the problem is a
>> >> cabling or not. Currently we use timeout to detect such that
>> >> situation. So if we could know if the timeout is occurred or not, then
>> >> we could know the problem is a cabling or not.
>> >>
>> >> 2) Once we succeed in #1, next thing we need to do is, whether a
>> >> session in question is using the broken server. This is fairly easy
>> >> because we already have the info on shared memory. If the session uses
>> >> the broken server, then we need to restart the process. No way. Other
>> >> case we just close a connection to the broken backend (if any).
>> >>
>> >> o Things we need to do:
>> >>
>> >> - Invent a way to know if the fail over request is created by
>> >>   pcp_detach_node. Probably we add a new flag to the fail over request
>> >>   packet to indicate whether the origin of the request is
>> >>   pcp_detach_node or not.
>> >>
>> >> - The same technique above can be used for the admin PostgreSQL
>> >>   shutdown case.
>> >>
>> >> - Create a API to deal with connections using the broken server.
>> >>
>> >> o What are the benefit once above proposal is implemented?
>> >>
>> >> - If conditions below are met, the user session can be survives after
>> fail over.
>> >>
>> >>  - Operated in streaming replication mode
>> >>
>> >>  - The failed server is not primary
>> >>
>> >>  - The session does not connect to the broken broken standby server
>> >>
>> >> Comments, opinions?
>> >> --
>> >> Tatsuo Ishii
>> >> SRA OSS, Inc. Japan
>> >> English: http://www.sraoss.co.jp/index_en.php
>> >> Japanese:http://www.sraoss.co.jp
>> >> _______________________________________________
>> >> pgpool-hackers mailing list
>> >> pgpool-hackers at pgpool.net
>> >> http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>> > _______________________________________________
>> > pgpool-hackers mailing list
>> > pgpool-hackers at pgpool.net
>> > http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>> _______________________________________________
>> pgpool-hackers mailing list
>> pgpool-hackers at pgpool.net
>> http://www.pgpool.net/mailman/listinfo/pgpool-hackers
>>
>
>
>
> --
> Ahsan Hadi
> Snr Director Product Development
> EnterpriseDB Corporation
> The Enterprise Postgres Company
>
> Phone: +92-51-8358874
> Mobile: +92-333-5162114
>
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
>
> This e-mail message (and any attachment) is intended for the use of the
> individual or entity to whom it is addressed. This message contains
> information from EnterpriseDB Corporation that may be privileged,
> confidential, or exempt from disclosure under applicable law. If you are
> not the intended recipient or authorized to receive this for the intended
> recipient, any use, dissemination, distribution, retention, archiving, or
> copying of this communication is strictly prohibited. If you have received
> this e-mail in error, please notify the sender immediately by reply e-mail
> and delete this message.
>



-- 
Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-hackers/attachments/20160426/335f688f/attachment-0001.html>


More information about the pgpool-hackers mailing list