[Pgpool-hackers] patch "Caching query results in pgpool-II"

Fri Jun 17 01:16:21 UTC 2011

You need to mention that users need to build pgpool with
--with-memcached=... if users want to use memcached based query
cache.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> Hello
> 
> I wrote documents about query cache (shmem or memcached).
> I'm sending the following patch file.
> Your comment would be appreciated.
> 
> - pgpool-ja.html
> - pgpool-en.html
> 
> Regards
> Masanori YAMAZAKI
> 
> 
> 
> 
> 2011/6/16 Masanori Yamazaki <m.yamazaki23 at gmail.com>
> 
>> I fixed errors and these program can be compiled.
>>
>> Some functions is declared in header file. Take memqcache.h for example.
>> Other c files include memqcache.h in order to use functions in memqcache.c.
>> (Refer to pool_process_context.c, pool_process_context.h)
>>
>> [memqcache.h]
>> extern POOL_STATUS memqcache_lookup(POOL_
>> CONNECTION *frontend, POOL_CONNECTION_POOL *backend, Node *node);
>> extern int set_cache_on_memcached(POOL_CONNECTION *frontend, const char
>> *md5_query, const char *data);
>> extern int get_cache_on_memcached(POOL_CONNECTION *frontend, const char
>> *md5_query, char **buf, size_t *len);
>> extern int delete_cache_on_memcached(const char *key);
>> ...and so on.
>>
>> Pgpool.conf.sample is included.
>>
>> In pool_memqcache.c, I removed header file which is not being used.
>>
>> Static functions is declared before the function body is defined.
>>
>>
>> > memcached.h and related stuffs should not be included
>> > unconditionaly. I prefer to enable memcached in
>> > configure(i.e. configure --with-memcached=... style). Pgpool
>> > hackers, I would like to hear opinions for this though.
>>
>> Now memcached.h is included in pool.h.
>>
>> "#include <libmemcached/memcached.h>"
>>
>> I implemented it without minding problem.
>> If need to take a method of doing "configure", I do so.
>>
>> Now I am writing documents(doc/pgpool-en.html and doc/pgpool-ja.html).
>> It will take more time to complete it.
>>
>>
>> Thanks
>> Masanori YAMAZAKI
>>
>>
>>
>>
>> 2011/6/15 Masanori Yamazaki <m.yamazaki23 at gmail.com>
>>
>>> 2011/6/15 Tatsuo Ishii <ishii at sraoss.co.jp>
>>>
>>>> Some comments to this patch:
>>>>
>>>> - Static functions should be declared before the function body is
>>>> defined.
>>>>
>>> I correct it.
>>>
>>>
>>>> - What memcache library are you using? There are some and you do not
>>>>  explain which one you are using. Maybe this?
>>>>
>>>>  http://tangent.org/552/libmemcached.html
>>>>
>>>>
>>> Yes, memcached library I am using is "libmemcached".Sorry I forgot to
>>> explain it.
>>> http://libmemcached.org/libMemcached.html
>>>
>>>
>>>
>>>> - memcached.h and related stuffs should not be included
>>>>  unconditionaly. I prefer to enable memcached in
>>>>  configure(i.e. configure --with-memcached=... style). Pgpool
>>>>  hackers, I would like to hear opinions for this though.
>>>>
>>>>
>>> - No pgpool.conf.sample* included
>>>>
>>> I correct it
>>>
>>>
>>>> - No documents(doc/pgpool-en.html and doc/pgpool-ja.html)
>>>>  included. These should be provided in the first place.
>>>>
>>>
>>> I add what is implemented to these documents.
>>>
>>>
>>>
>>>> - Did not compile:
>>>>
>>>> gcc -DHAVE_CONFIG_H -DDEFAULT_CONFIGDIR=\"/usr/local/etc\" -I.
>>>>  -D_GNU_SOURCE -I /usr/local/pgsql/include   -g -O2 -Wall
>>>> -Wmissing-prototypes -Wmissing-declarations -MT child.o -MD -MP -MF
>>>> .deps/child.Tpo -c -o child.o child.c
>>>> child.c: In function 'do_child':
>>>> child.c:153: warning: implicit declaration of function
>>>> 'init_memcached_connection'
>>>> child.c: At top level:
>>>> child.c:1966: warning: conflicting types for 'init_memcached_connection'
>>>> child.c:1966: error: static declaration of 'init_memcached_connection'
>>>> follows non-static declaration
>>>> child.c:153: error: previous implicit declaration of
>>>> 'init_memcached_connection' was here
>>>> child.c: In function 'init_memcached_connection':
>>>> child.c:1969: warning: implicit declaration of function
>>>> 'memcached_connect'
>>>> child.c:1970: error: 'memcached_con' undeclared (first use in this
>>>> function)
>>>> child.c:1970: error: (Each undeclared identifier is reported only once
>>>> child.c:1970: error: for each function it appears in.)
>>>> make[2]: *** [child.o] Error 1
>>>> make[2]: Leaving directory `/home/t-ishii/work/pgfoundry/GSoC/pgpool-II'
>>>> make[1]: *** [all-recursive] Error 1
>>>> make[1]: Leaving directory `/home/t-ishii/work/pgfoundry/GSoC/pgpool-II'
>>>> make: *** [all] Error 2
>>>>
>>>
>>> I correct an error of program and send new patch file as soon as possible.
>>>
>>> comment thanks
>>>
>>>
>>>> --
>>>> Tatsuo Ishii
>>>> SRA OSS, Inc. Japan
>>>> English: http://www.sraoss.co.jp/index_en.php
>>>> Japanese: http://www.sraoss.co.jp
>>>>
>>>> > 2011/6/14 Tatsuo Ishii <ishii at sraoss.co.jp>
>>>> >
>>>> >> Please explain what is implemented and what is not implemented in this
>>>> >> patch.
>>>> >>
>>>> >
>>>> > What is implemented in this patch are as below.
>>>> > - connect to memcached.
>>>> > - set RowDescription or DataRow on memcached.
>>>> > - get RowDescription or DataRow on memcached by using key of md5(SELECT
>>>> > convert).
>>>> > - delete cache on memcached by using key of md5(SELECT convert). only
>>>> > module.
>>>> > - fetch and parse the fellowing data in pgpool.conf.
>>>> >
>>>> > memory_cache_enabled
>>>> > memqcache_method
>>>> > memqcache_memcached_host
>>>> > memqcache_memcached_port
>>>> > memqcache_total_size
>>>> > memqcache_expire
>>>> > memqcache_maxcache
>>>> > memqcache_cache_block_size
>>>> >
>>>> > - add data to buffer before set data on memcached.
>>>> > - get buffer in order to set it on memcached
>>>> > - initialize buffer
>>>> >
>>>> > I am implemented only in Simple Query case.
>>>> >
>>>> >
>>>> > What is not implemented in this patch are as below.
>>>> > - shmem cache.
>>>> > structure of  managing table oids.
>>>> > the space management map.
>>>> > on memory query cache structure.
>>>> >
>>>> > - implementation in Extended Query case.
>>>> >
>>>> > - cache validation.
>>>> > The result of SELECT using none IMMUTABLE functions is not cached.
>>>> >
>>>> > - remove cache
>>>> > If cache is full, the least recently used cache is removed.
>>>> > If a table is dropped or modified, related cache data is deleted.
>>>> > If a table definition is modified(ALTER TABLE), related cache data is
>>>> > deleted.
>>>> > If a schema or database is dropped, related cache data is deleted.
>>>> >
>>>> >
>>>> > This week I am mainly implementing the shmem cache.
>>>> >
>>>> > Regards
>>>> > Masanori YAMAZAKI
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >> --
>>>> >> Tatsuo Ishii
>>>> >> SRA OSS, Inc. Japan
>>>> >> English: http://www.sraoss.co.jp/index_en.php
>>>> >> Japanese: http://www.sraoss.co.jp
>>>> >>
>>>> >> > Hello, hackers
>>>> >> >
>>>> >> > I have implemented a function caching query on memcached.
>>>> >> > I am sending the patch file and it's so nice to give me some
>>>> opinions.
>>>> >> >
>>>> >> > Main program which I implemented is as below.
>>>> >> > - Search for cache data on memcached by using md5 which converted
>>>> SELECT
>>>> >> > into.
>>>> >> > - Set data on memcached. The key is md5 which converted  SELECT
>>>> into.
>>>> >> > The value is RowDescription, DataRow from Backend.
>>>> >> >
>>>> >> > [Modify files]
>>>> >> > - pool.h
>>>> >> > - child.c
>>>> >> > - pool_config.l
>>>> >> > - pool_config.h
>>>> >> > - pool_config.c
>>>> >> > - pool_system.c
>>>> >> > - pool_proto_modules.c
>>>> >> > - pool_process_query.c
>>>> >> >
>>>> >> > [New File]
>>>> >> > - pool_memqcache.c
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > Regards
>>>> >> > Masanori YAMAZAKI
>>>> >>
>>>>
>>>
>>>
>>