[pgpool-general: 431] Re: strange load balancing issue in Solaris

[Tatsuo Ishii]

Thanks.

2012-05-09 14:31:48 LOG:   pid 22459: r: 268356063.000000 total_weight: 32767.000000

This is really weird. Here pgpool caculate this:

	r = (((double)random())/RAND_MAX) * total_weight;

Total weight is same as RAND_MAX.  It seems your random() returns
bigger than RAND_MAX, which does not make sense because man page of
random(3) on my Linux says:

	   The random() function uses a non-linear additive feedback random number
       generator  employing a default table of size 31 long integers to return
       successive pseudo-random numbers in the range from 0 to RAND_MAX.   The
       period  of  this  random  number generator is very large, approximately
       16 * ((2^31) - 1).

What does your man page for random() say on your system?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> Sorry . I missed it.
> 
> Here is the log file.
> 
> --Aravinth
> 
> 
> On Wed, May 9, 2012 at 2:07 PM, Tatsuo Ishii <ishii at postgresql.org> wrote:
> 
>> > The code you have sent is same in child.c.
>>
>> No.
>>
>>        pool_log("r: %f total_weight: %f", r, total_weight);
>>
>> You need to add the line above to get usefull information.
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese: http://www.sraoss.co.jp
>>
>>
>> > I have attached the log file. Please check
>> >
>> >
>> > --Aravinth
>> >
>> >
>> > On Tue, May 8, 2012 at 6:20 AM, Tatsuo Ishii <ishii at postgresql.org>
>> wrote:
>> >
>> >> I suspect there's some portablity issue with load balance code. The
>> >> actual source code is in select_load_balancing_nodechild.c).
>> >> Please modify source code and connect to pgpool by using psql.
>> >> Please send the log output.
>> >> --
>> >> Tatsuo Ishii
>> >> SRA OSS, Inc. Japan
>> >> English: http://www.sraoss.co.jp/index_en.php
>> >> Japanese: http://www.sraoss.co.jp
>> >>
>> >> int select_load_balancing_node(void)
>> >> {
>> >>        int selected_slot;
>> >>        double total_weight,r;
>> >>        int i;
>> >>
>> >>        /* choose a backend in random manner with weight */
>> >>        selected_slot = MASTER_NODE_ID;
>> >>        total_weight = 0.0;
>> >>
>> >>        for (i=0;i<NUM_BACKENDS;i++)
>> >>        {
>> >>                if (VALID_BACKEND(i))
>> >>                {
>> >>                        total_weight += BACKEND_INFO(i).backend_weight;
>> >>                }
>> >>        }
>> >>        r = (((double)random())/RAND_MAX) * total_weight;
>> >>        pool_log("r: %f total_weight: %f", r, total_weight);
>>  <--
>> >> add this
>> >>
>> >>        total_weight = 0.0;
>> >>        for (i=0;i<NUM_BACKENDS;i++)
>> >>        {
>> >>                if (VALID_BACKEND(i) && BACKEND_INFO(i).backend_weight >
>> >> 0.0)
>> >>                {
>> >>                        if(r >= total_weight)
>> >>                                selected_slot = i;
>> >>                        else
>> >>                                break;
>> >>                        total_weight += BACKEND_INFO(i).backend_weight;
>> >>                 }
>> >>        }
>> >>
>> >>        pool_debug("select_load_balancing_node: selected backend id is
>> %d",
>> >> selected_slot);
>> >>         return selected_slot;
>> >> }
>> >>
>> >>
>> >> > Hi Tatsuo, Thanks for the reply.
>> >> >
>> >> > The normalized weights are 0.5 for both nodes and the selected node is
>> >> always the same node. I hope then it's srandom().
>> >> >
>> >> >
>> >> > Any idea to solve this srandom issue
>> >> >
>> >> >
>> >> > Thanks and Regards,
>> >> > Aravinth
>> >> >
>> >> >
>> >> > ________________________________
>> >> >  From: Tatsuo Ishii <ishii at postgresql.org>
>> >> > To: aravinth at mafiree.com
>> >> > Cc: pgpool-general at pgpool.net
>> >> > Sent: Tuesday, May 1, 2012 4:41 AM
>> >> > Subject: Re: [pgpool-general: 396] strange load balancing issue in
>> >> Solaris
>> >> >
>> >> > First of all please check "normalized" weights are as you expected.
>> >> > Run "show pool_status;" and see "backend_weight0", "backend_weight1"
>> >> > section. You see a floating point numbers, which are the normalized
>> >> > weight between 0.0 and 1.0. If you see both are 0.5, primary and
>> >> > standby are given same weight.
>> >> >
>> >> > If they are ok, I suspect srandom() function behavior is different
>> >> > from other platforms. Pgpool-II chooses the load balance node by using
>> >> > srandom(). select_load_balancing_node() is the function which is
>> >> > responsible for selecting the load balance node. If you run pgpool-II
>> >> > with -d (debug) option, you will see following in the log:
>> >> >
>> >> >     pool_debug("select_load_balancing_node: selected backend id is
>> %d",
>> >> selected_slot);
>> >> >
>> >> > If backend_weight in show pool_status are fine but the line above
>> >> > always shows same number, it is the sign that we have problem with
>> >> > srandom().
>> >> > --
>> >> > Tatsuo Ishii
>> >> > SRA OSS, Inc. Japan
>> >> > English: http://www.sraoss.co.jp/index_en.php
>> >> > Japanese: http://www.sraoss.co.jp
>> >> >
>> >> >> Hi All,
>> >> >>
>> >> >> I am facing a strange issue in load balancing with replication mode
>> set
>> >> to
>> >> >> true in Solaris. Load balancing algorithm always select the same node
>> >> >> whatever may be the backend weight
>> >> >>
>> >> >> Here is the scenario.
>> >> >>
>> >> >> I have a pgpool installed installed in 1 server
>> >> >> 2 postgres nodes in other 2 servers
>> >> >> replication mode set to true and load balancing set to true
>> >> >> backend weight of the 2 nodes is 1.
>> >> >>
>> >> >> When I fire the queries manuall using different connections or using
>> >> >> pgbench all the queries hit the same node. Load balancing algorithm
>> >> always
>> >> >> select the same node.
>> >> >> No effect in changing the backend weight. Only when I set backend
>> >> weight to
>> >> >> 0 hits go to the other server.
>> >> >>
>> >> >>
>> >> >> I face this issue only in solaris. The same setup in other servers (
>> >> centos
>> >> >> ,RHEL, ubunt etc) does the load balancing perfectly.
>> >> >>
>> >> >> Also tries various postgres versions and pgpool version with same
>> >> result.
>> >> >> But every version runs fine in other servers.
>> >> >>
>> >> >> Has anyone faced this issue?
>> >> >>
>> >> >> Any information would highly helpful.
>> >> >>
>> >> >> Regards,
>> >> >> Aravinth
>> >> _______________________________________________
>> >> pgpool-general mailing list
>> >> pgpool-general at pgpool.net
>> >> http://www.pgpool.net/mailman/listinfo/pgpool-general
>> >>
>>