[pgpool-general: 4971] Segfault when using Master/Slave and Load Balancing

[Steven Livingstone]

Hi. I am using a web framework which creates a pool of connections and
uses transactions for each request. I am uses PGBouncer and PGPool to
proxy connections to my master/slave backend.

I find that with replication_mode=true it all works fine but load
balancing doesn't work as it's in a transaction, so i will get inserts
set to read only slaves and so on. To fix this i am looking at
Master/Slave mode with streaming replication which i believe does not
have this issue - and some tests using psql seem to show this is the
case.

However, when i use Master/Slave i get segfaults for every request
made and despite a week of debugging i don't know why this is ...
taken me a while just to get to here.

I have logs below for using psql from the same server as the web
framework, testing using a transaction and not from pSQL and finally
using the web framework. The queries are similar but there are some
variances that may be causing the problem. Does anyone have any
insight int where i should be looking? I don't know whether the
problem is in the web framework connection pooling side or PGPool load
balancing in this manner.

thanks
/steven

>From PSQL select in transaction (works) - [ begin;SELECT u0."id",
u0."name", u0."email", u0."bio", u0."number_of_pets",
u0."inserted_at", u0."updated_at" FROM "users" AS u0;commit; ] :

Sep  8 08:52:59 ha-node-0 pgpool[57646]: [537-1] 2016-09-08 08:52:59:
pid 57646: LOG:  statement: begin;
Sep  8 08:52:59 ha-node-0 pgpool[57646]: [538-1] 2016-09-08 08:52:59:
pid 57646: LOG:  DB node id: 0 backend pid: 53060 statement: begin;
Sep  8 08:52:59 ha-node-0 pgpool[57646]: [539-1] 2016-09-08 08:52:59:
pid 57646: LOG:  DB node id: 1 backend pid: 45037 statement: begin;
Sep  8 08:52:59 ha-node-0 pgpool[57646]: [540-1] 2016-09-08 08:52:59:
pid 57646: LOG:  statement: SELECT u0."id", u0."name", u0."email",
u0."bio", u0."number_of_pets", u0."inserted_at", u0."updated_at" FROM
"users" AS u0;
Sep  8 08:52:59 ha-node-0 pgpool[57646]: [541-1] 2016-09-08 08:52:59:
pid 57646: LOG:  DB node id: 1 backend pid: 45037 statement: SELECT
u0."id", u0."name", u0."email", u0."bio", u0."number_of_pets",
u0."inserted_at", u0."updated_at" FROM "users" AS u0;
Sep  8 08:52:59 ha-node-0 pgpool[57646]: [542-1] 2016-09-08 08:52:59:
pid 57646: LOG:  statement: commit;
Sep  8 08:52:59 ha-node-0 pgpool[57646]: [543-1] 2016-09-08 08:52:59:
pid 57646: LOG:  DB node id: 1 backend pid: 45037 statement: commit;
Sep  8 08:52:59 ha-node-0 pgpool[57646]: [544-1] 2016-09-08 08:52:59:
pid 57646: LOG:  DB node id: 0 backend pid: 53060 statement: commit;
Sep  8 08:52:59 ha-node-0 pgpool[57646]: [545-1] 2016-09-08 08:52:59:
pid 57646: LOG:  statement: DISCARD ALL


>From PSQL select NOT in a transaction (works) - [ SELECT u0."id",
u0."name", u0."email", u0."bio", u0."number_of_pets",
u0."inserted_at", u0."updated_at" FROM "users" AS u0; ] :

Sep  8 08:54:08 ha-node-0 pgpool[57646]: [551-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 0 backend pid: 53060 statement: BEGIN
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [552-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 1 backend pid: 45037 statement: BEGIN
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [553-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 0 backend pid: 53060 statement: SET
application_name='psql';
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [554-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 1 backend pid: 45037 statement: SET
application_name='psql';
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [555-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 1 backend pid: 45037 statement: COMMIT
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [556-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 0 backend pid: 53060 statement: COMMIT
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [557-1] 2016-09-08 08:54:08:
pid 57646: LOG:  statement: SELECT u0."id", u0."name", u0."email",
u0."bio", u0."number_of_pets", u0."inserted_at", u0."updated_at" FROM
"users" AS u0;
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [558-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 1 backend pid: 45037 statement: SELECT
u0."id", u0."name", u0."email", u0."bio", u0."number_of_pets",
u0."inserted_at", u0."updated_at" FROM "users" AS u0;
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [559-1] 2016-09-08 08:54:08:
pid 57646: LOG:  statement: DISCARD ALL
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [560-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 0 backend pid: 53060 statement: DISCARD
ALL
Sep  8 08:54:08 ha-node-0 pgpool[57646]: [561-1] 2016-09-08 08:54:08:
pid 57646: LOG:  DB node id: 1 backend pid: 45037 statement: DISCARD
ALL


>From Phoenix (Fails):

Sep  8 08:49:20 ha-node-0 pgpool[55577]: [470-1] 2016-09-08 08:49:20:
pid 55577: LOG:  statement: DISCARD ALL
Sep  8 08:49:20 ha-node-0 pgpool[55577]: [471-1] 2016-09-08 08:49:20:
pid 55577: LOG:  DB node id: 0 backend pid: 52931 statement: DISCARD
ALL
Sep  8 08:49:20 ha-node-0 pgpool[55577]: [472-1] 2016-09-08 08:49:20:
pid 55577: LOG:  DB node id: 1 backend pid: 44960 statement: DISCARD
ALL
Sep  8 08:49:21 ha-node-0 pgpool[55577]: [473-1] 2016-09-08 08:49:21:
pid 55577: LOG:  DB node id: 1 backend pid: 44960 statement: Parse:
SELECT u0."id", u0."name", u0."email", u0."bio", u0."number_of_pets",
u0."inserted_at", u0."updated_at" FROM "users" AS u0
Sep  8 08:49:21 ha-node-0 pgpool[55577]: [474-1] 2016-09-08 08:49:21:
pid 55577: LOG:  DB node id: 1 backend pid: 44960 statement: D message
Sep  8 08:49:21 ha-node-0 kernel: [51411.763896] pgpool[56590]:
segfault at ac ip 00000000004354a5 sp 00007ffd9f345be0 error 4 in
pgpool[400000+132000]
Sep  8 08:49:21 ha-node-0 kernel: [51411.814139] pgpool[55575]:
segfault at ac ip 00000000004354a5 sp 00007ffd9f345c40 error 4 in
pgpool[400000+132000]