[pgpool-general: 6021] Re: wrong arguments for online recovery script
Benjamin Firl
bf at wisit.com
Thu Apr 5 19:23:17 JST 2018
I'll try to explain it with an example from our system:
*
*/~$ psql -U postgres -p 5433 -c "show pool_nodes" | grep -v unused//
// node_id | hostname | port | status | lb_weight | role | select_cnt
| load_balance_node | replication_delay//
//---------+-----------+------+--------+-----------+--------+------------+-------------------+-------------------//
// 14 | 10.0.1.14 | 5432 | down | 0.200000 | slave | 2035
| false | 0//
// 15 | 10.0.1.15 | 5432 | down | 0.200000 | slave | 2181
| false | 0//
// 19 | 10.0.1.19 | 5432 | down | 0.200000 | slave | 1834
| false | 0//
// 20 | 10.0.1.20 | 5432 | up | 0.200000 | master | 3997772
| true | 0//
// 21 | 10.0.1.21 | 5432 | up | 0.200000 | slave | 3781854
| false | 0/
We create and delete nodes automatically depending on our current
resource needs. As you can see, we generate the node ID from the IP
address of the system.
Node IDs 14, 15 and 19 were used before, but are not even in the
configuration file anymore. Node 20 and 21 are the only ones active.
If we now add a new node, we write it to the configuration file with its
generated node ID and start the online recovery process. If this ID is
e.g. 22, everything works fine. But if the ID is e.g. 1 in this case,
the recovery scripts receives the wrong parameters as shown.
Hope that makes things clearer.
Kind regards
Benjamin Firl
On 04/05/2018 11:01 AM, Tatsuo Ishii wrote:
> Can you share the exact step to reproduce the problem?
>
> "a new lowest node id is added" is a little bit unclear to me.
>
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp
>
>> Hallo,
>>
>>
>> I tested with the patched 3.7.2, still the same problem. I also tested
>> again with 3.6.7, this version also shows the same behaviour.
>>
>> Also I am now able to reproduce it relatively easy, it really happens
>> every time a new lowest node id is added. So if there is anything else I
>> can try, configuration options or something, I will gladly do it.
>>
>>
>> Kind regards
>> Benjamin Firl
>>
>>
>>
>> On 03/29/2018 10:21 AM, Tatsuo Ishii wrote:
>>> Still I suspect the bug is only with 3.7.2. Recently we releasd a
>>> fixed version of 3.7.2 RPM (actually we just reverted a commit which
>>> caused the bug). If you like, you could try the RPM.
>>>
>>> Or you install from the source code, you could patch 3.7.2 with this commit:
>>> https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commit;h=e8baa3f0ee1a24f833cfa6cc5d0104b5cefe7b04
>>>
>>> Best regards,
>>> --
>>> Tatsuo Ishii
>>> SRA OSS, Inc. Japan
>>> English: http://www.sraoss.co.jp/index_en.php
>>> Japanese:http://www.sraoss.co.jp
>>>
>>>> Hi,
>>>>
>>>>
>>>> is there any news on this topic? Any information that I can provide to help?
>>>> Thanks again for your help.
>>>>
>>>>
>>>> Kind regards
>>>> Benjamin Firl
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 03/02/2018 09:33 AM, Benjamin Firl wrote:
>>>>> We have seen problems with online recovery in pgpool-II version 3.6.7
>>>>> (subaruboshi). But we updated to 3.7.2 before we ware able to reproduce
>>>>> and analyze it. So I suspect the same thing happened in 3.6.7, but I am
>>>>> not 100% sure.
>>>>> Adding a new lowest node id is a relatively rare case in our setup.
>>>>>
>>>>>
>>>>> Kind regards
>>>>> Benjamin Firl
>>>>>
>>>>>
>>>>>
>>>>> On 03/02/2018 12:16 AM, Tatsuo Ishii wrote:
>>>>>> I suspect it's a bug with 3.7.2. Have you seen similar issue with
>>>>>> other Pgpool-II versions?
>>>>>>
>>>>>> Best regards,
>>>>>> --
>>>>>> Tatsuo Ishii
>>>>>> SRA OSS, Inc. Japan
>>>>>> English: http://www.sraoss.co.jp/index_en.php
>>>>>> Japanese:http://www.sraoss.co.jp
>>>>>>
>>>>>>> our script for online recovery stage1 seems to get wrong arguments
>>>>>>> sometimes and thus is failing.
>>>>>>>
>>>>>>> We log the arguments, so on a successful run it prints something like
>>>>>>> that on the master node:
>>>>>>>
>>>>>>> recovery_1st_stage.sh
>>>>>>> all args: /var/lib/postgresql/9.5/main 10.0.1.25
>>>>>>> /var/lib/postgresql/9.5/main 5432
>>>>>>> data: /var/lib/postgresql/9.5/main
>>>>>>> recovery target: 10.0.1.25
>>>>>>> recovery data: /var/lib/postgresql/9.5/main
>>>>>>>
>>>>>>>
>>>>>>> If recovery fails, we get this:
>>>>>>>
>>>>>>> recovery_1st_stage.sh
>>>>>>> all args: /var/lib/postgresql/9.5/main localhost 5432
>>>>>>> data: /var/lib/postgresql/9.5/main
>>>>>>> recovery target: localhost
>>>>>>> recovery data: 5432
>>>>>>> unable to recover myself....exiting
>>>>>>> 2018-03-01 14:31:59 CET [10517-1] postgres at template1 ERROR:
>>>>>>> pgpool_recovery failed
>>>>>>> 2018-03-01 14:31:59 CET [10517-2] postgres at template1 STATEMENT: SELECT
>>>>>>> pgpool_recovery('recovery_1st_stage.sh', 'localhost', '', '5432')
>>>>>>>
>>>>>>>
>>>>>>> This seems to happen, if the new node has a lower id than the master
>>>>>>> node. But normally adding a new node with lower id should not be a
>>>>>>> problem, am I correct?
>>>>>>> We are using PostgreSQL 9.5.10 and pgpool-II version 3.7.2 (amefuriboshi).
>>>>>>> Thanks for your help.
>>>>>>>
>>>>>>>
>>>>>>> Kind regards
>>>>>>> Benjamin Firl
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>
>>>>>>> +++ Jetzt neu Wissensmanagement für Netzwerke +++
>>>>>>>
>>>>>>> +++ www.knodge.de +++
>>>>>>>
>>>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> www.wisit.com
>>>>>>>
>>>>>>> www.knodge.de
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> wisit media GmbH
>>>>>>>
>>>>>>> Ehrenbergstr. 11
>>>>>>>
>>>>>>> D-98693 Ilmenau
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> wisit media GmbH, Ehrenbergstr. 11, D-98693 Ilmenau
>>>>>>>
>>>>>>> Registergericht Jena HRB 512472
>>>>>>>
>>>>>>> Geschaeftsfuehrung: Dipl. Ing. Markus Duelli
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich
>>>>>>>
>>>>>>> erhalten haben, informieren Sie bitte sofort den Absender und vernichten
>>>>>>>
>>>>>>> Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
>>>>>>>
>>>>>>> dieser E-Mail ist nicht gestattet.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> pgpool-general mailing list
>>>>>>> pgpool-general at pgpool.net
>>>>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>>>> --
>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>> +++ Jetzt neu Wissensmanagement für Netzwerke +++
>>>>
>>>> +++ www.knodge.de +++
>>>>
>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> www.wisit.com
>>>>
>>>> www.knodge.de
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> wisit media GmbH
>>>>
>>>> Ehrenbergstr. 11
>>>>
>>>> D-98693 Ilmenau
>>>>
>>>> ---------------------------------------------------------------------------
>>>>
>>>> -
>>>>
>>>> wisit media GmbH, Ehrenbergstr. 11, D-98693 Ilmenau
>>>>
>>>> Registergericht Jena HRB 512472
>>>>
>>>> Geschaeftsfuehrung: Dipl. Ing. Markus Duelli
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich
>>>>
>>>> erhalten haben, informieren Sie bitte sofort den Absender und vernichten
>>>>
>>>> Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
>>>>
>>>> dieser E-Mail ist nicht gestattet.
>>>>
>>>> _______________________________________________
>>>> pgpool-general mailing list
>>>> pgpool-general at pgpool.net
>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>> --
>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> +++ Jetzt neu Wissensmanagement für Netzwerke +++
>>
>> +++ www.knodge.de +++
>>
>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>> --
>>
>> www.wisit.com
>>
>> www.knodge.de
>>
>>
>>
>>
>>
>> wisit media GmbH
>>
>> Ehrenbergstr. 11
>>
>> D-98693 Ilmenau
>>
>> ---------------------------------------------------------------------------
>>
>> -
>>
>> wisit media GmbH, Ehrenbergstr. 11, D-98693 Ilmenau
>>
>> Registergericht Jena HRB 512472
>>
>> Geschaeftsfuehrung: Dipl. Ing. Markus Duelli
>>
>>
>>
>>
>>
>> Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich
>>
>> erhalten haben, informieren Sie bitte sofort den Absender und vernichten
>>
>> Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
>>
>> dieser E-Mail ist nicht gestattet.
>>
>> _______________________________________________
>> pgpool-general mailing list
>> pgpool-general at pgpool.net
>> http://www.pgpool.net/mailman/listinfo/pgpool-general
--
+++++++++++++++++++++++++++++++++++++++++++++++++
+++ Jetzt neu Wissensmanagement für Netzwerke +++
+++ www.knodge.de +++
+++++++++++++++++++++++++++++++++++++++++++++++++
--
www.wisit.com
www.knodge.de
wisit media GmbH
Ehrenbergstr. 11
D-98693 Ilmenau
---------------------------------------------------------------------------
-
wisit media GmbH, Ehrenbergstr. 11, D-98693 Ilmenau
Registergericht Jena HRB 512472
Geschaeftsfuehrung: Dipl. Ing. Markus Duelli
Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtuemlich
erhalten haben, informieren Sie bitte sofort den Absender und vernichten
Sie diese E-Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe
dieser E-Mail ist nicht gestattet.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.sraoss.jp/pipermail/pgpool-general/attachments/20180405/23f89280/attachment-0001.html>
More information about the pgpool-general
mailing list