View Issue Details

IDProjectCategoryView StatusLast Update
0000015Pgpool-IIBugpublic2013-09-12 13:21
ReportersporkAssigned Tot-ishii 
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionopen 
PlatformOSFreeBSDOS Version8.2
Product Version 
Target VersionFixed in Version 
Summary0000015: pgpool-II 3.1.3 on FreeBSD regex patterns for blacklist/whitelist report errors
DescriptionUsing the default config which is just a simple list of comma separated keywords causes pgpool to throw errors and likely ignore the configuration settings.

config statement in pgpool.conf:

black_function_list = 'currval,lastval,nextval,setval'
                                   # Comma separated list of function names
                                   # that write to database
                                   # Regexp are accepted

log:

Jun 7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern: ^currval$
Jun 7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern: ^lastval$
Jun 7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern: ^nextval$
Jun 7 01:20:22 devrep pgpool[52608]: add_to_patterns: invalid regex pattern: ^setval$

This is on a FreeBSD 8.2 box.

I see a similar report from about a year ago:

http://answerpot.com/showthread.php?3105812-Pgpool+3.1+errors
Steps To ReproduceSetup a config with sample config file, including the "black_function_list" statement.

Start pgpool, watch log output.
TagsNo tags attached.

Activities

t-ishii

2012-06-07 16:58

developer   ~0000037

Works for me (Linux). Maybe FreeBSD specific issue? I don't have FreeBSD at this point, and I cannot confirm now. Sorry.

BTW, the link you are referring is not related to the problem you are facing,
I think. Probably the problem stated by the link has been fixed by:

3.1.1 (hatsuiboshi) 2011/12/06

      * Version 3.1.1

      This is a bugfix release against pgpool-II 3.1.

      * Bug fixes

      - Fix add_regex_pattern(). It does not allocate enough memory
           for each black/white_function_list items. The function adds
           "^" and "$" to each function items which do not contain those
           characters. Unfortunately the function forgot to add extra 2
           bytes for those characters. This may lead to memory corruption
           errors when pgpool starting up.

spork

2012-09-25 13:48

reporter   ~0000091

Still present in 3.1.2.

t-ishii

2012-10-14 12:22

developer   ~0000104

I have tested pgpool-II 3.2.1 and 3.1.5 on FreeBSD 9.0 box. I don't see no problem with black function list so far.

spork

2012-10-18 14:55

reporter   ~0000109

Still see it on 3.2.1 (although there are some build issues on FreeBSD in the watchdog code). Maybe this debug info helps. This is FreeBSD 8.3.

Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: key: white_function_list
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: value: '' kind: 4
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: key: black_function_list
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: value: 'currval,lastval,nextval,setval' kind: 4
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token: currval
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token: lastval
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token: nextval
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: extract_string_tokens: token: setval
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern: ^currval$
Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex pattern: ^currval$
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern: ^lastval$
Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex pattern: ^lastval$
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern: ^nextval$
Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex pattern: ^nextval$
Oct 18 01:49:18 k16 pgpool: DEBUG: pid 6545: add_to_patterns: regex pattern: ^setval$
Oct 18 01:49:18 k16 pgpool: ERROR: pid 6545: add_to_patterns: invalid regex pattern: ^setval$

At least it shows the config parser is getting the values right. :)

Just thought I'd add this. 3.2.1 doesn't quite work on FreeBSD 8.3 (can't connect to any db hosts for the health checks), but I wanted to at least document the regex issue.

t-ishii

2012-10-18 15:15

developer   ~0000110

I have fixed some FreeBSD build issues. Next time please grab 3.2-STABLE head.

spork

2012-10-20 10:48

reporter   ~0000114

Just downloaded, built and installed 3.2 head.

Build issues are totally resolved, but the regex errors are still there:

DEBUG: pid 18118: add_to_patterns: regex pattern: ^currval$
ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^currval$
DEBUG: pid 18118: add_to_patterns: regex pattern: ^lastval$
ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^lastval$
DEBUG: pid 18118: add_to_patterns: regex pattern: ^nextval$
ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^nextval$
DEBUG: pid 18118: add_to_patterns: regex pattern: ^setval$
ERROR: pid 18118: add_to_patterns: invalid regex pattern: ^setval$

t-ishii

2012-10-20 17:39

developer   ~0000118

I confirmed that with FreeBSD 8.3 the problem reproduced. Strange thing is, nm command against pgpool shows following result:
[t-ishii@freebsd /usr/local/src/pgpool/pgpool-II-3.2.1]$ nm ~/bin/pgpool |grep regcomp
0000000000476df0 T regcomp

"T" means regcomp is a global symbol in pgpool. This should be like this:
[t-ishii@localhost tmp]$ nm a.out|grep regcomp
                 U regcomp@@GLIBC_2.2.5

a.out is produced from test program:
#include <regex.h>
#include <stdio.h>
main()
{
        regex_t regexv;
        int rtn;
        int regex_flags = REG_NOSUB;
        regex_flags |= REG_ICASE;
        regex_flags |= REG_EXTENDED;
        rtn = regcomp(&regexv, "^nextval$", regex_flags);
        if (rtn != 0)
        {
                printf("invalid regex pattern %d\n", rtn);
        }
}

This program does exactly same thing in pgpool and works fine of courese. My wild guess is, pgpool uses wrong regcomp module in FreeBSD 8.3. I don't know why though.

spork

2013-08-28 15:54

reporter   ~0000313

Just noting this still exists. We've had to stop using pgpool for now, as we have some queries that use "nextval()" and they get sent to the slave, which causes problems for us:

Aug 19 10:18:24 k29.i postgres[83999]: [17-1] db=xxx,user=xxx ERROR: cannot execute nextval() in a read-only transaction
Aug 19 10:18:24 k29.i postgres[83999]: [17-2] db=xxx,user=xxx STATEMENT: select nextval('signup_id_seq');

I've tested again, now on FreeBSD 8.4 and pgpool 3.3.0. Same error message when pgpool exits:

DEBUG: pid 96388: key: black_function_list
DEBUG: pid 96388: value: 'currval,lastval,nextval,setval' kind: 4
DEBUG: pid 96388: extract_string_tokens: token: currval
DEBUG: pid 96388: extract_string_tokens: token: lastval
DEBUG: pid 96388: extract_string_tokens: token: nextval
DEBUG: pid 96388: extract_string_tokens: token: setval
DEBUG: pid 96388: add_to_patterns: regex pattern: ^currval$
ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^currval$
DEBUG: pid 96388: add_to_patterns: regex pattern: ^lastval$
ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^lastval$
DEBUG: pid 96388: add_to_patterns: regex pattern: ^nextval$
ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^nextval$
DEBUG: pid 96388: add_to_patterns: regex pattern: ^setval$
ERROR: pid 96388: add_to_patterns: invalid regex pattern: ^setval$

I also added a call to the error routine to see what the regex flags are. I don't understand bitwise stuff at all (and barely understand C), so I don't know if "7" represents the correct flags or not:

DEBUG: pid 3484: add_to_patterns: regex pattern: ^currval$
ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^currval$
ERROR: pid 3484: add_to_patterns: regex flags: 7
DEBUG: pid 3484: add_to_patterns: regex pattern: ^lastval$
ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^lastval$
ERROR: pid 3484: add_to_patterns: regex flags: 7
DEBUG: pid 3484: add_to_patterns: regex pattern: ^nextval$
ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^nextval$
ERROR: pid 3484: add_to_patterns: regex flags: 7
DEBUG: pid 3484: add_to_patterns: regex pattern: ^setval$
ERROR: pid 3484: add_to_patterns: invalid regex pattern: ^setval$
ERROR: pid 3484: add_to_patterns: regex flags: 7

What else can I give you to check this out? Is "7" the correct flag to bitwise combine 0001 0002 and 0004?

Putting another debug in there shows me "errno 2" which maps to "invalid regex".

I'm stumped.

Your test program above works as expected.

t-ishii

2013-08-28 18:54

developer   ~0000314

I have no idea how to debug further. As I said before, I think the problem must be in the compiler tool chain, including gcc and linker on FreeBSD, not the pgpool source itself. To verify this, you can do:

- cd to pgpool source code
- compile
- see "nm pool_config.o|grep regcomp". If it's not something like:
                 U regcomp
You are in trouble.

spork

2013-08-29 00:38

reporter   ~0000315

[root@client1 /usr/local/src/pgpool-II-3.3.0]# nm pool_config.o |grep regcomp
         U regcomp
[root@client1 /usr/local/src/pgpool-II-3.3.0]#

t-ishii

2013-08-29 10:47

developer   ~0000316

It turned out if we remove the linking of compat library, everything goes fine.
I did it by hand:
gcc -g -O2 -Wall -Wmissing-prototypes -Wmissing-declarations -o .libs/pgpool main.o child.o pool_auth.o pool_config.o pool_error.o pool_process_query.o pool_stream.o pool_connection_pool.o pool_params.o pool_signal.o pcp_child.o md5.o pool_shmem.o pool_sema.o pool_system.o pool_rewrite_query.o pool_rewrite_outfuncs.o pool_query_cache.o pool_hba.o pool_path.o pool_ip.o ps_status.o strlcpy.o recovery.o pool_relcache.o pool_process_reporting.o pool_ssl.o pool_timestamp.o pool_proto2.o pool_proto_modules.o pool_lobj.o pool_process_context.o pool_memqcache.o pool_session_context.o pool_query_context.o pool_worker_child.o pool_passwd.o pool_globals.o pool_select_walker.o getopt_long.o parser/nodes.o -L/usr/local/pgsql/lib -lpq parser/libsql-parser.a pcp/.libs/libpcp.so watchdog/lib-watchdog.a -lpthread -lcrypt -lm -Wl,-rpath -Wl,/usr/local/lib -Wl,-rpath -Wl,/usr/local/pgsql/lib

However, I don't know how to tell autoconf do that.

spork

2013-08-29 12:12

reporter   ~0000317

Just confirmed here that if I pull "-lcompat" from all Makefiles I get a clean build and the error goes away.

I have no idea how the whole autotools suite works though. I'm also not really sure what the compat lib is for, trying to read up on it a bit. Thanks!

t-ishii

2013-08-29 14:14

developer  

configure.in.patch (366 bytes)
diff --git a/configure.in b/configure.in
index 4d6f6e4..2ac1ed5 100644
--- a/configure.in
+++ b/configure.in
@@ -58,7 +58,6 @@ AC_CHECK_LIB(socket,   main)
 AC_CHECK_LIB(ipc,      main)
 AC_CHECK_LIB(IPC,      main)
 AC_CHECK_LIB(lc,       main)
-AC_CHECK_LIB(compat,   main)
 AC_CHECK_LIB(BSD,      main)
 AC_CHECK_LIB(gen,      main)
 AC_CHECK_LIB(PW,       main)
configure.in.patch (366 bytes)

t-ishii

2013-08-29 14:16

developer   ~0000318

It seems "-lcompat" is in configure.in. Attached patch is just removing it. Unfortunately I couldn't get neccessary tool chains such as autoconf to test it put on my FreeBSD box. Can you please try it out?

spork

2013-09-12 10:32

reporter   ~0000327

I missed your last reply... I can confirm that removing the compat check from configure.in results in a working build.

I also think I sort of understand why the compat libs break things. Looking at the source, I'm guessing this is for really, really old backward compatibility and it appears to include it's own regex functions, which I imagine is how we end up with the errors - the compat regex functions probably take arguments in a different order or something. Does that sound about right?

[root@client1 /usr/local/src/pgpool-II-3.3.0]# ls -R /usr/src/lib/libcompat/
4.1 4.3 4.4 Makefile regexp

/usr/src/lib/libcompat/4.1:
ascftime.c cftime.c ftime.c getpw.c
cftime.3 ftime.3 getpw.3

/usr/src/lib/libcompat/4.3:
cfree.3 re_comp.3 rexec.3
cfree.c regex.c rexec.c

/usr/src/lib/libcompat/4.4:
cuserid.3 cuserid.c

/usr/src/lib/libcompat/regexp:
COPYRIGHT regerror.c regexp.c regsub.c
README regexp.3 regmagic.h

t-ishii

2013-09-12 13:21

developer   ~0000329

Ok, I'm going to remove -lcompat from configure.in.

Issue History

Date Modified Username Field Change
2012-06-07 15:41 spork New Issue
2012-06-07 16:58 t-ishii Note Added: 0000037
2012-09-25 13:48 spork Note Added: 0000091
2012-10-14 12:21 t-ishii Assigned To => t-ishii
2012-10-14 12:21 t-ishii Status new => assigned
2012-10-14 12:22 t-ishii Note Added: 0000104
2012-10-18 14:55 spork Note Added: 0000109
2012-10-18 15:15 t-ishii Note Added: 0000110
2012-10-20 10:48 spork Note Added: 0000114
2012-10-20 17:39 t-ishii Note Added: 0000118
2013-08-28 15:54 spork Note Added: 0000313
2013-08-28 18:54 t-ishii Note Added: 0000314
2013-08-29 00:38 spork Note Added: 0000315
2013-08-29 10:47 t-ishii Note Added: 0000316
2013-08-29 12:12 spork Note Added: 0000317
2013-08-29 14:14 t-ishii File Added: configure.in.patch
2013-08-29 14:16 t-ishii Note Added: 0000318
2013-09-12 10:32 spork Note Added: 0000327
2013-09-12 13:21 t-ishii Note Added: 0000329
2013-09-12 13:21 t-ishii Status assigned => resolved