[Pgpool-hackers] patch "Caching query results in pgpool-II"

[Tatsuo Ishii]

So where is new document patch?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp

> 2011/6/17 Tatsuo Ishii <ishii at sraoss.co.jp>
> 
>> You need to mention that users need to build pgpool with
>> --with-memcached=... if users want to use memcached based query
>> cache.
>>
> 
> I correct documents(pgpool-ja.html/pgpool-en.html) for memcached users.
> 
> comment thanks.
> Masanori YAMAZAKI
> 
> 
>> --
>> Tatsuo Ishii
>> SRA OSS, Inc. Japan
>> English: http://www.sraoss.co.jp/index_en.php
>> Japanese: http://www.sraoss.co.jp
>>
>> > Hello
>> >
>> > I wrote documents about query cache (shmem or memcached).
>> > I'm sending the following patch file.
>> > Your comment would be appreciated.
>> >
>> > - pgpool-ja.html
>> > - pgpool-en.html
>> >
>> > Regards
>> > Masanori YAMAZAKI
>> >
>> >
>> >
>> >
>> > 2011/6/16 Masanori Yamazaki <m.yamazaki23 at gmail.com>
>> >
>> >> I fixed errors and these program can be compiled.
>> >>
>> >> Some functions is declared in header file. Take memqcache.h for example.
>> >> Other c files include memqcache.h in order to use functions in
>> memqcache.c.
>> >> (Refer to pool_process_context.c, pool_process_context.h)
>> >>
>> >> [memqcache.h]
>> >> extern POOL_STATUS memqcache_lookup(POOL_
>> >> CONNECTION *frontend, POOL_CONNECTION_POOL *backend, Node *node);
>> >> extern int set_cache_on_memcached(POOL_CONNECTION *frontend, const char
>> >> *md5_query, const char *data);
>> >> extern int get_cache_on_memcached(POOL_CONNECTION *frontend, const char
>> >> *md5_query, char **buf, size_t *len);
>> >> extern int delete_cache_on_memcached(const char *key);
>> >> ...and so on.
>> >>
>> >> Pgpool.conf.sample is included.
>> >>
>> >> In pool_memqcache.c, I removed header file which is not being used.
>> >>
>> >> Static functions is declared before the function body is defined.
>> >>
>> >>
>> >> > memcached.h and related stuffs should not be included
>> >> > unconditionaly. I prefer to enable memcached in
>> >> > configure(i.e. configure --with-memcached=... style). Pgpool
>> >> > hackers, I would like to hear opinions for this though.
>> >>
>> >> Now memcached.h is included in pool.h.
>> >>
>> >> "#include <libmemcached/memcached.h>"
>> >>
>> >> I implemented it without minding problem.
>> >> If need to take a method of doing "configure", I do so.
>> >>
>> >> Now I am writing documents(doc/pgpool-en.html and doc/pgpool-ja.html).
>> >> It will take more time to complete it.
>> >>
>> >>
>> >> Thanks
>> >> Masanori YAMAZAKI
>> >>
>> >>
>> >>
>> >>
>> >> 2011/6/15 Masanori Yamazaki <m.yamazaki23 at gmail.com>
>> >>
>> >>> 2011/6/15 Tatsuo Ishii <ishii at sraoss.co.jp>
>> >>>
>> >>>> Some comments to this patch:
>> >>>>
>> >>>> - Static functions should be declared before the function body is
>> >>>> defined.
>> >>>>
>> >>> I correct it.
>> >>>
>> >>>
>> >>>> - What memcache library are you using? There are some and you do not
>> >>>>  explain which one you are using. Maybe this?
>> >>>>
>> >>>>  http://tangent.org/552/libmemcached.html
>> >>>>
>> >>>>
>> >>> Yes, memcached library I am using is "libmemcached".Sorry I forgot to
>> >>> explain it.
>> >>> http://libmemcached.org/libMemcached.html
>> >>>
>> >>>
>> >>>
>> >>>> - memcached.h and related stuffs should not be included
>> >>>>  unconditionaly. I prefer to enable memcached in
>> >>>>  configure(i.e. configure --with-memcached=... style). Pgpool
>> >>>>  hackers, I would like to hear opinions for this though.
>> >>>>
>> >>>>
>> >>> - No pgpool.conf.sample* included
>> >>>>
>> >>> I correct it
>> >>>
>> >>>
>> >>>> - No documents(doc/pgpool-en.html and doc/pgpool-ja.html)
>> >>>>  included. These should be provided in the first place.
>> >>>>
>> >>>
>> >>> I add what is implemented to these documents.
>> >>>
>> >>>
>> >>>
>> >>>> - Did not compile:
>> >>>>
>> >>>> gcc -DHAVE_CONFIG_H -DDEFAULT_CONFIGDIR=\"/usr/local/etc\" -I.
>> >>>>  -D_GNU_SOURCE -I /usr/local/pgsql/include   -g -O2 -Wall
>> >>>> -Wmissing-prototypes -Wmissing-declarations -MT child.o -MD -MP -MF
>> >>>> .deps/child.Tpo -c -o child.o child.c
>> >>>> child.c: In function 'do_child':
>> >>>> child.c:153: warning: implicit declaration of function
>> >>>> 'init_memcached_connection'
>> >>>> child.c: At top level:
>> >>>> child.c:1966: warning: conflicting types for
>> 'init_memcached_connection'
>> >>>> child.c:1966: error: static declaration of 'init_memcached_connection'
>> >>>> follows non-static declaration
>> >>>> child.c:153: error: previous implicit declaration of
>> >>>> 'init_memcached_connection' was here
>> >>>> child.c: In function 'init_memcached_connection':
>> >>>> child.c:1969: warning: implicit declaration of function
>> >>>> 'memcached_connect'
>> >>>> child.c:1970: error: 'memcached_con' undeclared (first use in this
>> >>>> function)
>> >>>> child.c:1970: error: (Each undeclared identifier is reported only once
>> >>>> child.c:1970: error: for each function it appears in.)
>> >>>> make[2]: *** [child.o] Error 1
>> >>>> make[2]: Leaving directory
>> `/home/t-ishii/work/pgfoundry/GSoC/pgpool-II'
>> >>>> make[1]: *** [all-recursive] Error 1
>> >>>> make[1]: Leaving directory
>> `/home/t-ishii/work/pgfoundry/GSoC/pgpool-II'
>> >>>> make: *** [all] Error 2
>> >>>>
>> >>>
>> >>> I correct an error of program and send new patch file as soon as
>> possible.
>> >>>
>> >>> comment thanks
>> >>>
>> >>>
>> >>>> --
>> >>>> Tatsuo Ishii
>> >>>> SRA OSS, Inc. Japan
>> >>>> English: http://www.sraoss.co.jp/index_en.php
>> >>>> Japanese: http://www.sraoss.co.jp
>> >>>>
>> >>>> > 2011/6/14 Tatsuo Ishii <ishii at sraoss.co.jp>
>> >>>> >
>> >>>> >> Please explain what is implemented and what is not implemented in
>> this
>> >>>> >> patch.
>> >>>> >>
>> >>>> >
>> >>>> > What is implemented in this patch are as below.
>> >>>> > - connect to memcached.
>> >>>> > - set RowDescription or DataRow on memcached.
>> >>>> > - get RowDescription or DataRow on memcached by using key of
>> md5(SELECT
>> >>>> > convert).
>> >>>> > - delete cache on memcached by using key of md5(SELECT convert).
>> only
>> >>>> > module.
>> >>>> > - fetch and parse the fellowing data in pgpool.conf.
>> >>>> >
>> >>>> > memory_cache_enabled
>> >>>> > memqcache_method
>> >>>> > memqcache_memcached_host
>> >>>> > memqcache_memcached_port
>> >>>> > memqcache_total_size
>> >>>> > memqcache_expire
>> >>>> > memqcache_maxcache
>> >>>> > memqcache_cache_block_size
>> >>>> >
>> >>>> > - add data to buffer before set data on memcached.
>> >>>> > - get buffer in order to set it on memcached
>> >>>> > - initialize buffer
>> >>>> >
>> >>>> > I am implemented only in Simple Query case.
>> >>>> >
>> >>>> >
>> >>>> > What is not implemented in this patch are as below.
>> >>>> > - shmem cache.
>> >>>> > structure of  managing table oids.
>> >>>> > the space management map.
>> >>>> > on memory query cache structure.
>> >>>> >
>> >>>> > - implementation in Extended Query case.
>> >>>> >
>> >>>> > - cache validation.
>> >>>> > The result of SELECT using none IMMUTABLE functions is not cached.
>> >>>> >
>> >>>> > - remove cache
>> >>>> > If cache is full, the least recently used cache is removed.
>> >>>> > If a table is dropped or modified, related cache data is deleted.
>> >>>> > If a table definition is modified(ALTER TABLE), related cache data
>> is
>> >>>> > deleted.
>> >>>> > If a schema or database is dropped, related cache data is deleted.
>> >>>> >
>> >>>> >
>> >>>> > This week I am mainly implementing the shmem cache.
>> >>>> >
>> >>>> > Regards
>> >>>> > Masanori YAMAZAKI
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >> --
>> >>>> >> Tatsuo Ishii
>> >>>> >> SRA OSS, Inc. Japan
>> >>>> >> English: http://www.sraoss.co.jp/index_en.php
>> >>>> >> Japanese: http://www.sraoss.co.jp
>> >>>> >>
>> >>>> >> > Hello, hackers
>> >>>> >> >
>> >>>> >> > I have implemented a function caching query on memcached.
>> >>>> >> > I am sending the patch file and it's so nice to give me some
>> >>>> opinions.
>> >>>> >> >
>> >>>> >> > Main program which I implemented is as below.
>> >>>> >> > - Search for cache data on memcached by using md5 which converted
>> >>>> SELECT
>> >>>> >> > into.
>> >>>> >> > - Set data on memcached. The key is md5 which converted  SELECT
>> >>>> into.
>> >>>> >> > The value is RowDescription, DataRow from Backend.
>> >>>> >> >
>> >>>> >> > [Modify files]
>> >>>> >> > - pool.h
>> >>>> >> > - child.c
>> >>>> >> > - pool_config.l
>> >>>> >> > - pool_config.h
>> >>>> >> > - pool_config.c
>> >>>> >> > - pool_system.c
>> >>>> >> > - pool_proto_modules.c
>> >>>> >> > - pool_process_query.c
>> >>>> >> >
>> >>>> >> > [New File]
>> >>>> >> > - pool_memqcache.c
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > --
>> >>>> >> > Regards
>> >>>> >> > Masanori YAMAZAKI
>> >>>> >>
>> >>>>
>> >>>
>> >>>
>> >>
>>