[pgpool-hackers: 3910] Re: Patch: Move auto_failback_interval in to BackendInfo, and update it any time the backend state is set to CON_DOWN
Tatsuo Ishii
ishii at sraoss.co.jp
Wed Jun 2 08:55:18 JST 2021
Note that I have already implemented follow_primay_lock and pushed to
the repository.
https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=455f00dd5f5b7b94bd91aa0b6b40aab21dceabb9
> Hi,
>
> I am just coming back to this work now after some time on other projects.
>
> I think there are several proposals around improving auto_failback in this thread:
> 1) my patch
> 2) Ishii-san’s patch to check follow_primary_count == 0
> 3) Ishii-san’s proposal to implement a lock to avoid the window where follow_primary might run after checking follow_primary_count
>
> My understanding is we think 1+2 are good, and we can look at 3 if there is still a problem - or perhaps we plan to look at 3 as a future improvement, to avoid a potential problem?
>
> Would you like me to test 1+2?
>
>> On 10/05/2021, at 7:16 PM, Takuma Hoshiai <hoshiai.takuma at nttcom.co.jp> wrote:
>>
>> Hi,
>>
>> On 2021/05/05 16:03, Tatsuo Ishii wrote:
>>>>> On 27/04/2021, at 10:18 AM, Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>>>>>
>>>>> Hi Nathan,
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Sorry about that! I dragged them from the vscode file list directly to Mail - I suspect that that doesn’t work when using remote editing..!
>>>>>>
>>>>>> I have attached the files now - does that work?
>>>>>
>>>>> Yes! I will look into the patches. Hoshiai-san, can you please look
>>>>> into the patches as well because you are the original author of the
>>>>> feature.
>>>>
>>>> Hi!
>>>>
>>>> I was wondering if you had time to look at these patches yet? :-)
>>>>
>>>> No rush - just making sure it doesn’t get missed!
>>> I just have started to look into your patch. Also I was able to
>>> reproduce the problem.
>>> 1) create 3-node streaming replication cluster.
>>> pgpool_setup -n 3
>>> Enable auto_failback and set health_check_period to 1 so that
>>> auto_failback runs more aggressively.
>>> auto_failback = on
>>> health_check_period0 = 1
>>> health_check_period1 = 1
>>> health_check_period2 = 1
>>> start the whole system.
>>> 2) detach node 0 (which is primary)
>>> 3) node 3 becomes down and PostgreSQL won't start
>>> psql -p 11000 -c "show pool_nodes" test
>>> node_id | hostname | port | status | pg_status | lb_weight | role | pg_role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change
>>> ---------+----------+-------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
>>> 0 | /tmp | 11002 | up | up | 0.333333 | standby | standby | 0 | true | 0 | streaming | async | 2021-05-05 14:10:38
>>> 1 | /tmp | 11003 | up | up | 0.333333 | primary | primary | 0 | false | 0 | | | 2021-05-05 14:10:25
>>> 2 | /tmp | 11004 | down | down | 0.333333 | standby | unknown | 0 | false | 0 | | | 2021-05-05 14:10:38
>>> (3 rows)
>>> The cause of the problem is a race condition between the auto failback
>>> and follow primary as you and Hoshiai-san suggested. Here are some
>>> extraction from the pgpool.log.
>>> $ egrep "degeneration|failback" log/pgpool.log|grep -v child
>>> 2021-05-05 14:10:22: main pid 28630: LOG: starting degeneration. shutdown host /tmp(11002)
>>> 2021-05-05 14:10:25: main pid 28630: LOG: starting follow degeneration. shutdown host /tmp(11002)
>>> 2021-05-05 14:10:25: main pid 28630: LOG: starting follow degeneration. shutdown host /tmp(11004) -- #1
>>> 2021-05-05 14:10:25: health_check2 pid 28673: LOG: request auto failback, node id:2 -- #2
>>> 2021-05-05 14:10:25: health_check2 pid 28673: LOG: received failback request for node_id: 2 from pid [28673]
>>> 2021-05-05 14:10:35: main pid 28630: LOG: failback done. reconnect host /tmp(11004)
>>> 2021-05-05 14:10:35: main pid 28630: LOG: failback done. reconnect host /tmp(11002) -- #3
>>> 2021-05-05 14:10:36: pcp_child pid 29035: LOG: starting recovering node 2
>>> 2021-05-05 14:10:36: pcp_child pid 29035: ERROR: node recovery failed, node id: 2 is alive -- #4
>>> 2021-05-05 14:10:38: child pid 29070: LOG: failed to connect to PostgreSQL server by unix domain socket
>>> 2021-05-05 14:10:38: child pid 29070: DETAIL: executing failover on backend
>>> 2021-05-05 14:10:38: main pid 28630: LOG: Pgpool-II parent process has received failover request
>>> 2021-05-05 14:10:38: main pid 28630: LOG: starting degeneration. shutdown host /tmp(11004) -- #5
>>> 1) Follow primary started to shutdown node 2. At this point the
>>> backend node 2 was still running.
>>> 2) auto failback found that backend is still alive and send failback
>>> request for node 2.
>>> 3) pgpool main process reported that node 2 was back. But actual
>>> failback had not done and continued by follow primary command.
>>> 4) follow primary command for node 2 failed because auto failback set
>>> the status of node 2 to "up".
>>> 5) Node 2 PostgreSQL was down and health check detected it. Node 2
>>> status became down.
>>> So if auto failback did not run at #2, the follow primary should have
>>> been succeeded.
>>> BTW accidently I and a user found similar situation: conflicting
>>> concurrent run of detach_false_primary and follow primary command:
>>> https://www.pgpool.net/pipermail/pgpool-general/2021-April/007583.html
>>> In the discussion I proposed a patch to prevent the concurrent run of
>>> detach_false_primary and follow primary command. I think we can apply
>>> the method to auto_failback as well. Attached is the patch to
>>> implement it on top of the patch I posted here for the master branch:
>>> https://www.pgpool.net/pipermail/pgpool-general/2021-April/007594.html
>>> This patch actually has a small window between here:
>>> if (check_failback && !Req_info->switching && slot &&
>>> Req_info->follow_primary_count == 0)
>>> and here:
>>> ereport(LOG,
>>> (errmsg("request auto failback, node id:%d", node)));
>>> /* get current time to use auto_faliback_interval */
>>> now = time(NULL);
>>> auto_failback_interval = now + pool_config->auto_failback_interval;
>>> send_failback_request(node, true, REQ_DETAIL_CONFIRMED);
>>> because after checking Req_info->follow_primary_count, follow primary
>>> might start just after this. I think the window and probably is
>>> harmless in the wild. If you think it's not so small, we could do an
>>> exclusive lock like in detach_false_primary to plug the window.
>>> Also we have found that detach_false_primary should only run on the
>>> leader watchdog node. Probably we should consider this for
>>> auto_failback too.
>>
>> I have started to look this patch too. But I have failed pgool_setup
>> command in latest master branch with auto_failback_fixes-master.patch.
>> This cause is researching now (It may be my environment is bad).
>>
>> As far as I can see, auto_failback_fixes-master.patch is good.
>> And I think that ishii-san's suggestion makes this patch better.
>>
>>
>>
>>> Best regards,
>>> --
>>> Tatsuo Ishii
>>> SRA OSS, Inc. Japan
>>> English: http://www.sraoss.co.jp/index_en.php <http://www.sraoss.co.jp/index_en.php>
>>> Japanese:http://www.sraoss.co.jp <http://www.sraoss.co.jp/>
>>> _______________________________________________
>>> pgpool-hackers mailing list
>>> pgpool-hackers at pgpool.net <mailto:pgpool-hackers at pgpool.net>
>>> http://www.pgpool.net/mailman/listinfo/pgpool-hackers <http://www.pgpool.net/mailman/listinfo/pgpool-hackers>
>>
>> Best Regards,
>>
>> --
>> Takuma Hoshiai <hoshiai.takuma at nttcom.co.jp <mailto:hoshiai.takuma at nttcom.co.jp>>
>
More information about the pgpool-hackers
mailing list