View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000357 | Pgpool-II | Bug | public | 2017-10-19 12:12 | 2017-10-25 12:30 |
| Reporter | Devin | Assigned To | t-ishii | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | closed | Resolution | open | ||
| Platform | Linux | OS | CentOS | OS Version | 7.3.1611 |
| Product Version | 3.6.6 | ||||
| Summary | 0000357: Memcached status for different sessions is inconsistent | ||||
| Description | One session detected memcached error and then discarded it, but didn't notify other sessions or other pgpool instances. So other sessions may continue to use memcached, then issue happened. | ||||
| Steps To Reproduce | ------------------------------------------------------------------------------------------------------------------------------------ Client A: test=# select count(*) from a; count ------- 1 (1 row) test=# select count(*) from a; count ------- 1 (1 row) test=# insert into a select 1; INSERT 0 1 test=# select count(*) from a; count ------- 2 (1 row) test=# select count(*) from a; count ------- 2 (1 row) Log: 2017-10-14 16:28:26: pid 77675: LOG: statement: select count(*) from a; 2017-10-14 16:28:26: pid 77675: LOG: DB node id: 2 backend pid: 55417 statement: select count(*) from a; 2017-10-14 16:28:32: pid 77675: LOG: statement: select count(*) from a; 2017-10-14 16:28:32: pid 77675: LOG: fetch from memory cache 2017-10-14 16:28:32: pid 77675: DETAIL: query result fetched from cache. statement: select count(*) from a; 2017-10-14 16:29:00: pid 77675: LOG: statement: insert into a select 1; 2017-10-14 16:29:00: pid 77675: LOG: DB node id: 0 backend pid: 77700 statement: insert into a select 1; 2017-10-14 16:29:04: pid 77675: LOG: statement: select count(*) from a; 2017-10-14 16:29:04: pid 77675: LOG: DB node id: 2 backend pid: 55417 statement: select count(*) from a; 2017-10-14 16:29:05: pid 77675: LOG: statement: select count(*) from a; 2017-10-14 16:29:05: pid 77675: LOG: fetch from memory cache 2017-10-14 16:29:05: pid 77675: DETAIL: query result fetched from cache. statement: select count(*) from a; ------------------------------------------------------------------------------------------------------------------------------------ Suppose that memcached is restarted at this time. $ pkill memcached $ memcached -m32 -d -v ------------------------------------------------------------------------------------------------------------------------------------ Client A: test=# insert into a select 1; INSERT 0 1 test=# select count(*) from a; count ------- 3 (1 row) test=# select count(*) from a; count ------- 3 (1 row) Log: 2017-10-14 16:30:02: pid 77675: LOG: statement: insert into a select 1; 2017-10-14 16:30:02: pid 77675: LOG: fetching from cache storage, memcached_get failed with error: "CONNECTION FAILURE" 2017-10-14 16:30:02: pid 77675: LOG: DB node id: 0 backend pid: 77700 statement: insert into a select 1; 2017-10-14 16:30:06: pid 77675: LOG: statement: select count(*) from a; 2017-10-14 16:30:06: pid 77675: LOG: DB node id: 2 backend pid: 55417 statement: select count(*) from a; 2017-10-14 16:30:07: pid 77675: LOG: statement: select count(*) from a; 2017-10-14 16:30:07: pid 77675: LOG: DB node id: 2 backend pid: 55417 statement: select count(*) from a; ------------------------------------------------------------------------------------------------------------------------------------ Client B: $ psql -h127.0.0.1 -p9999 -Udevin test Password for user devin: psql (10beta4) Type "help" for help. test=# select count(*) from a; count ------- 3 (1 row) test=# select count(*) from a; count ------- 3 (1 row) Log: 2017-10-14 16:30:24: pid 77674: LOG: new connection received 2017-10-14 16:30:24: pid 77674: DETAIL: connecting host=localhost port=36272 2017-10-14 16:30:25: pid 77674: LOG: new connection received 2017-10-14 16:30:25: pid 77674: DETAIL: connecting host=localhost port=36280 2017-10-14 16:30:35: pid 77674: LOG: statement: select count(*) from a; 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) FROM pg_catalog.pg_proc AS p WHERE p.proname = 'count' AND p.provolatile = 'i' 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) FROM pg_catalog.pg_class AS c, pg_attribute AS a WHERE c.relname = 'pg_class' AND a.attrelid = c.oid AND a.attname = 'relistemp' 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) FROM pg_class AS c, pg_namespace AS n WHERE c.relname = 'a' AND c.relnamespace = n.oid AND n.nspname ~ '^pg_temp_' 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) from (SELECT has_function_privilege('devin', 'to_regclass(cstring)', 'execute') WHERE EXISTS(SELECT * FROM pg_catalog.pg_proc AS p WHERE p.proname = 'to_regclass')) AS s 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) FROM pg_catalog.pg_class AS c WHERE c.relname = 'pg_namespace' 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) FROM pg_class AS c, pg_namespace AS n WHERE c.oid = to_regclass('"a"') AND c.relnamespace = n.oid AND n.nspname = 'pg_catalog' 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) FROM pg_catalog.pg_class AS c WHERE c.oid = to_regclass('"a"') AND (c.relkind = 'v' OR c.relkind = 'm') 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) FROM pg_catalog.pg_class AS c, pg_catalog.pg_attribute AS a WHERE c.relname = 'pg_class' AND a.attrelid = c.oid AND a.attname = 'relpersistence' 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT count(*) FROM pg_catalog.pg_class AS c WHERE c.oid = to_regclass('"a"') AND c.relpersistence = 'u' 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: select count(*) from a; 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT COALESCE(to_regclass('"a"')::oid, 0) 2017-10-14 16:30:35: pid 77674: LOG: DB node id: 0 backend pid: 77928 statement: SELECT oid FROM pg_database WHERE datname = 'test' 2017-10-14 16:30:36: pid 77674: LOG: statement: select count(*) from a; 2017-10-14 16:30:36: pid 77674: LOG: fetch from memory cache 2017-10-14 16:30:36: pid 77674: DETAIL: query result fetched from cache. statement: select count(*) from a; ------------------------------------------------------------------------------------------------------------------------------------ Client A: test=# insert into a select 1; INSERT 0 1 test=# select count(*) from a; count ------- 4 (1 row) test=# select count(*) from a; count ------- 4 (1 row) Log: 2017-10-14 16:30:51: pid 77675: LOG: statement: insert into a select 1; 2017-10-14 16:30:51: pid 77675: LOG: DB node id: 0 backend pid: 77700 statement: insert into a select 1; 2017-10-14 16:30:53: pid 77675: LOG: statement: select count(*) from a; 2017-10-14 16:30:53: pid 77675: LOG: DB node id: 2 backend pid: 55417 statement: select count(*) from a; 2017-10-14 16:30:54: pid 77675: LOG: statement: select count(*) from a; 2017-10-14 16:30:54: pid 77675: LOG: DB node id: 2 backend pid: 55417 statement: select count(*) from a; ------------------------------------------------------------------------------------------------------------------------------------ Client B: test=# select count(*) from a; count ------- 3 (1 row) Log: 2017-10-14 16:31:03: pid 77674: LOG: statement: select count(*) from a; 2017-10-14 16:31:03: pid 77674: LOG: fetch from memory cache 2017-10-14 16:31:03: pid 77674: DETAIL: query result fetched from cache. statement: select count(*) from a; ------------------------------------------------------------------------------------------------------------------------------------ | ||||
| Tags | memcached | ||||
|
|
I think that's an expected behavior. Client A does not know memcached is restartred after detecting the failure of memcached. So it never invalidates the query cache after inserting data. On the other hand Client B startred after the restart of memcached, and feels free to create a new cache entry. You need to restart Pgpool-II after restarting memcached. |
|
|
Hi, I see, thank you for your reply. Regards, Devin |
|
|
Hi Devin, Thank you for the report and your understanding. I think we would be able to do better in the future, for example, checking the connection to memcached after detecting failures in connection every time DML is issued. If it succeeded in re-connecting, invalidate the query cache. Of course this may give us additional overhead, so we have to think more how to deal with it though. Thanks, Tatsuo Ishii |
|
|
I am going to close this issue. If you have further questions, please post to our mailing list. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2017-10-19 12:12 | Devin | New Issue | |
| 2017-10-19 12:12 | Devin | Tag Attached: memcached | |
| 2017-10-20 08:37 | t-ishii | Assigned To | => t-ishii |
| 2017-10-20 08:37 | t-ishii | Status | new => assigned |
| 2017-10-20 08:44 | t-ishii | Note Added: 0001768 | |
| 2017-10-20 08:44 | t-ishii | Status | assigned => feedback |
| 2017-10-20 10:18 | Devin | Note Added: 0001769 | |
| 2017-10-20 10:18 | Devin | Status | feedback => assigned |
| 2017-10-20 13:08 | t-ishii | Note Added: 0001770 | |
| 2017-10-25 12:30 | t-ishii | Note Added: 0001784 | |
| 2017-10-25 12:30 | t-ishii | Status | assigned => closed |
