[pgpool-general: 2950] Re: pgpool 3.3 and watchdog

[Alexandru Cardaniuc]

Hi Joar,

I started another thread on this issue yesterday. All the information is
there with config settings, logs and failover scripts.

The problem is that the failover script according to the logs was NOT
even executed. So, the backend was properly detached, but the failover
script not executed... Very strange...

Joar Jegleim <joar.jegleim at gmail.com> writes:

> Hi,
>
> I haven't seen this, I'm also running 9.3 and 3.3.3 . I've setup a lot
> of logging though in my failover, basebackup and pgpool remote start
> scripts. I've come across several errors that turn out to be
> misconfiguration on my part, it could be anything from host
> authentication (ssh), SElinux blocking 'something' to configuration of
> pgpool itself .
>
> My 2 cent would be to add as much logging as possible to the scripts
> that handle the failover, then at least you may see where things stop
> . Also check your postgres logs, I've noticed output from basebackup
> and pgpoo remote start end up in the postgres logs, unless you've
> redirected the elsewhere .
>
>
> jj
>
>
> On 11 June 2014 08:55, Alexandru Cardaniuc <cardaniuc at gmail.com>
> wrote:
>> Thing is I already have a cluster configured with 2 nodes and
>> watchdog. I use PostgreSQL 9.3 and Pgpool 3.3.3 and for the most
>> part it seems to be working well. But, I had one case recently where
>> I did a failover where I failed the master node, and pgpool properly
>> identified the master node as failed and detached it from the
>> cluster (status 3 in pcp_node_info), but didn't run the script to do
>> the trigger on the slave node for some reason. So, even though the
>> traffic was properly redirected to the new master (former slave),
>> this new master (former slave) was still in READ_ONLY mode. Can't
>> find what would be an issue. Is that a known issue? What should I be
>> looking at?
>> Joar Jegleim <joar.jegleim at gmail.com> writes:
>>
>>> Hi Alexandru,
>>> I looked into pacemaker ++ and decided I didn't need take that
>>> route for my (only) 2 node setup . I've been hardening my 2 node
>>> pgpool, watchdog, floating ip master / slave setup . I plan on
>>> publishing my whole setup as a howto when done ( if my boss give me
>>> go) , I'll keep you posted.
>>> Be patient, this takes time, don't expect any howto on this until
>>> august / september, mabye even october ( got a big release due in
>>> sept. which I'm working hard to make sure our pgpool failover
>>> cluster may serve) .
>>> And please cc my mail directly when mailing me, since I filter all
>>> my mailinglist mail, and came over your post here by accident :)
>>> regards Joar Jegleim
>>>
>>> On 9 June 2014 21:45, Alexandru Cardaniuc <cardaniuc at gmail.com>
>>> wrote:
>>>> Hi Joar, Can you keep us in the loop on your research? That kind
>>>> of setup would be something interesting to consider if it
>>>> increases the robustness of the clustering solution. Joar Jegleim
>>>> <joar.jegleim at gmail.com> writes:
>>>>
>>>>> Please forgive me, false alarm, it works now. When I was
>>>>> reproducing this to send you the logs it suddenly worked (?) . I
>>>>> suspect it's because I've disabled iptables since I was working
>>>>> with this last time. I've previously opened up heartbeat port
>>>>> 9694 and watchdog port 9000 between the 2 nodes (+ pgpool and
>>>>> postgres ports of course), I thought mabye that was enough .
>>>>> Mabye I will have to open up for the delegate ip as well, or
>>>>> mabye there are some more ports I will have to open up. Anyway, I
>>>>> will continue testing with no iptables, and figure this thing out
>>>>> when pgpool and postgres configuration / testing / tuning etc..
>>>>> is done .
>>>>> As an other note I was looking into using pacemaker and all the
>>>>> cluster utilities that comes with CentOS yesterday, I wonder if
>>>>> anyone have any experience in using pacemaker + pgpool VS. simply
>>>>> using pgpool + watchdog and delegate ip . My first impression of
>>>>> pacemaker is that it seem pretty robust + mabye I should consider
>>>>> proper fencing of failed nodes . On the other side going down
>>>>> that route my setup becomes a whole lot more complex so I'm not
>>>>> sure yet which is better for my setup which most probably will
>>>>> consist of 2 nodes ( I'd have to disable cuorum and stuff ) . I
>>>>> will probably look into both solutions and try figure out what
>>>>> works best for us . Any input on this is much appreciated .
>>>> -- "Never let school interfere with your education." - Mark Twain
>>>> _______________________________________________ pgpool-general
>>>> mailing list pgpool-general at pgpool.net
>>>> http://www.pgpool.net/mailman/listinfo/pgpool-general
>> -- "It's very well to be thrifty, but don't amass a hoard of
>> regrets." - Charles D'Orleans
>> _______________________________________________ pgpool-general
>> mailing list pgpool-general at pgpool.net
>> http://www.pgpool.net/mailman/listinfo/pgpool-general

-- 
"If we really understand the problem, the answer will come out of it,
because the answer is not separate from the problem."  
- Krishnamurti