[pgpool-general: 3904] Re: pgpool doesn't see the new master on failover

[Muhammad Usama]

Hi

On Thu, Jul 30, 2015 at 5:51 PM, Maksym Filenko <
maksym.filenko at kreditech.com> wrote:

> Hey Muhammad,
>
> thank you for your prompt reply!
>
> Any ideas how could it happen? I've got following in my pgpool.conf:
>
> backend_hostname0 = 'db1.tx.dev.mydomain'
> backend_port0 = 5432
> backend_weight0 = 1
> backend_data_directory0 = '/var/lib/postgresql/9.4/main'
> backend_flag0 = 'ALLOW_TO_FAILOVER'
>
> backend_hostname1 = 'db2.tx.dev.mydomain'
> backend_port1 = 5432
> backend_weight1 = 1
> backend_data_directory1 = '/var/lib/postgresql/9.4/main'
> backend_flag1 = 'ALLOW_TO_FAILOVER'
>
> fail_over_on_backend_error = on
>
> Name: db1.tx.dev.mydomain
> Address: 10.200.12.33
>
> Name: db2.tx.dev.mydomain
> Address: 10.200.12.34
>

It is strange your pgpool conf file seems to have different servers
for backend_hostname0
and backend_hostname1. Try to make sure that pgpool is loading the same
config file which has these settings.



>
> Could you also please point a line from logs which leads you to this
> conclusion?
>

in *pgpool-2.log* file, Line no 11 to 14 for backend_hostname0 settings and
line 26 to 29 for backend_hostname1. Both hostnames and port numbers are
same* (hostname = db1.tx.dev.mydomain and port 5432)*

See below the respective extracts from the pgpool-2. log file

11 2015-07-29 18:00:00 DEBUG: pid 2311: key: backend_hostname0

 12 2015-07-29 18:00:00 DEBUG: pid 2311: value: 'db1.tx.dev.mydomain' kind:
4

 13 2015-07-29 18:00:00 DEBUG: pid 2311: key: backend_port0
 14 2015-07-29 18:00:00 DEBUG: pid 2311: value: 5432 kind: 2
...

26 2015-07-29 18:00:00 DEBUG: pid 2311: key: backend_hostname1

  27 2015-07-29 18:00:00 DEBUG: pid 2311: value: 'db1.tx.dev.mydomain'
kind: 4

  28 2015-07-29 18:00:00 DEBUG: pid 2311: key: backend_port1

  29 2015-07-29 18:00:00 DEBUG: pid 2311: value: 5432 kind: 2
Same is the case with  *pgpool-1.log *

Thanks
Muhammad Usama


> Thanks a lot!
>
> Regards,
> Max
>
> On Thu, Jul 30, 2015 at 12:36 PM, Muhammad Usama <m.usama at gmail.com>
> wrote:
>
>> Hi Max
>>
>> Thanks for sharing the logs. From the log it seems like both backend0 and
>> backend1 are pointing to the same PostgreSQL server, so when that PG server
>> goes down(which is actually both master and slave as per the
>> configuration). pgpool-II is left with no valid PostgreSQL backend, As a
>> result no new master node gets selected and eventually the failover.sh gets
>> empty values for new master hostname (%H) and database cluster path (%R)
>> arguments.
>>
>> Thanks and regards
>> Muhammad Usama
>>
>> On Wed, Jul 29, 2015 at 9:28 PM, Maksym Filenko <
>> maksym.filenko at kreditech.com> wrote:
>>
>>> Hey Muhammad,
>>>
>>> thank you for your reply!
>>>
>>> Please, find gzipped logs attached below. pgpool-1 is for master,
>>> pgpool-2 is for standby. Master Postgres node was shutted down at 18:01:01.
>>> You can see a failover.sh call with insufficient parameters. It should be
>>> something like '/srv/pgsql/bin/failover.sh 0 db2.tx.dev.mydomain' (new
>>> master hostname).
>>>
>>> Again, thank you!
>>>
>>> On Wed, Jul 29, 2015 at 10:53 AM, Muhammad Usama <m.usama at gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> Can you please share the pgpool-II log with debug enabled.
>>>>
>>>> Thanks
>>>> Best regards
>>>> Muhammad Usama
>>>>
>>>>
>>>> On Mon, Jul 27, 2015 at 8:18 PM, Maksym Filenko <
>>>> maksym.filenko at kreditech.com> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> I've set up exactly this configuration:
>>>>> http://www.pgpool.net/pgpool-web/contrib_docs/watchdog_master_slave_3.3/en.html
>>>>>
>>>>> I have two instances of pgpool-II version 3.3.4 (tokakiboshi) in front
>>>>> of two PostgreSQL 9.4.3.
>>>>>
>>>>> I have following in my pgpool.conf:
>>>>>
>>>>> failover_command = '/var/lib/postgresql/9.4/main/failover.sh %d %P %H
>>>>> %R'
>>>>>
>>>>> When master goes down and failover occurs, pgpool passes nothing to
>>>>> failover.sh as the new master hostname (%H) and database cluster path (%R):
>>>>>
>>>>> $ grep 'failover' /var/log/messages
>>>>> failover_handler: no valid DB node found
>>>>> execute command: /var/lib/postgresql/9.4/main/failover.sh 0 0
>>>>> failover: set new primary node: -1
>>>>>
>>>>> If the slave fails, everything works as expected:
>>>>>
>>>>> $ grep 'failover' /var/log/messages
>>>>> execute command: /var/lib/postgresql/9.4/main/failover.sh 1 0
>>>>> db1.example.com /var/lib/postgresql/9.4/main
>>>>> failover: set new primary node: 0
>>>>> failover: set new master node: 0
>>>>>
>>>>> Looks like pgpool doesn't see the slave to promote it.
>>>>>
>>>>> Any clue would be extremely appreciated!
>>>>>
>>>>> Best regards,
>>>>> Max
>>>>>
>>>>> _______________________________________________
>>>>> pgpool-general mailing list
>>>>> pgpool-general at pgpool.net
>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20150730/b41bc942/attachment-0001.html>