[pgpool-general: 7481] Re: Promote specific backend to primary in streaming replication

[Nathan Ward]

Hi,

> On 6/04/2021, at 5:41 PM, Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
> 
> Hi Nathan,
> 
>>> OK - this sounds great. I can have a stab at getting something like that
>>> written.
>>> 
>>> So, in terms of implementation this sounds like:
>>> 1) New pcp command to set a next primary backend ID. I'm not sure how this
>>> would be stored, perhaps update the parsed configuration state (is this
>>> pool_config), so it could also be in the configuration file? Or, it could
>>> update a new flag on bkinfo?
>>> 2) Modify get_next_main_node to include a check for the above, and check
>>> that it is a valid backend and not the current (failed/detached) primary
>>> backend.
>>> 
>>> Is modifying get_next_main_node() appropriate here? My thinking is that
>>> doing this means we can continue to use the existing scripts, which use %m
>>> and %H and so on.
>>> Perhaps this is best implemented as a backend "primary-ship" priority -
>>> i.e. set a priority for each backend (via config and pcp) to become the
>>> primary, as a uint8 on bkinfo, then in get_next_main_node sort first by
>>> priority then by node id.
>> 
>> Thank you for the great ideas. I need to think about these. Will be
>> back tomorrow.
> 
> I think we don't need create new pcp command. Instead, we can have new
> parameter something like:
> 
> backend_promote_priority0 = 0
> backend_promote_priority1 = 1
> backend_promote_priority2 = 2
> 
> The higher number, the higher priority. For example suppose node 0 is
> current primary. If node 0 goes down, node 2 will be chosen because it
> has the highest priority 2 (if node 2 is already down, node 1 will be
> chosen). The selected node info will be set to %m, %H, %r of failover
> script as you said.
> 
> If these parameters are not specified, we will chose the youngest live
> node, which is currently Pgpool is doing.
> 
> If we want to change the priority, pgpool reload should work.
> 
> As you already pointed out, the advantage of this way is, existing
> failover script can be used to without any change.

Thanks for your time on this so far!

Would it be possible to allow these to be changed at runtime without a config change + reload? Perhaps a more generic pcp command, to allow any “reloadable” configuration item to be changed? I’m not sure how many of these there are, and if any of them require deeper changes. Maybe we need a 3rd “class” of configuration item, to allow it to be changed without changing config.

We (and I am sure many others) use config management tools to keep our config in sync. In particular, we use Puppet. I have a module which I’m hoping to be able to publish soon, in fact.

Usually, environments which use config management like this have policies to not modify configuration which is managed.

Switching to a different backend would mean we have to manually update that managed configuration (on the pgpool primary, presumably) and reload, and then run pcp_detach_node against the primary backend. In the time between updating the configuration, and detaching the node, our automation might run and undo our changes which would mean we may get an outcome we don’t expect.
An alternative would be to update the configuration in our management system and deploy it - but in many environments making these changes can be quite involved. I don’t think it’s good to require operational changes (i.e. “move the primary function to this backend”) to require configuration updates.

A work-around to this would be to have an include directive in the configuration and include a local, non-managed file for these sorts of temporary changes, but pgpool doesn’t have this at the moment I believe?

> As of implementation details, I think we need to have new member in
> POOL_REQUEST_INFO structure.  Currently we have main_node_id and
> primary_node_id on it. I think we should leave them as they are and
> have new member something like
> "new_primary_candidate". get_next_main_node() will calculate the value
> for new_primary_candidate depending on pgpool.conf. Finally we pass
> the new_primary_candidate info to trigger_failover_command's
> new_main_node parameter to handle failover.
> 
> Also those priority info should be added to BackendInfo structure.

That sounds great - as it would allow the %m and %H and so on to be used to convey the new_primary_candidate.

--
Nathan Ward

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20210406/e8fef46b/attachment-0001.htm>