[pgpool-general: 4368] pgpool-II 3.4.0 and pgpoolAdmin 3.4.0 officially released
Yugo Nagata
nagata at sraoss.co.jp
Fri Jan 29 18:56:57 JST 2016
Pgpool Global Development Group is pleased to announce the availability of
pgpool-II 3.5.0 and pgpoolAdmin 3.5.0.
You can download the source codes and RPMs from:
http://pgpool.net/mediawiki/index.php/Downloads
V3.5 new features are:
- Improved performance in extended query protocol
- Overcoming the thundering herd problem
- Watchdog feature enhancements to be more robust and adaptable
- PCP command enhancements
- Import PostgreSQL 9.5 parser
- etc.
Please see the below for details:
http://pgpool.net/mediawiki/index.php?title=pgpool-II_3.5_features&redirect=no
Release notes are here:
=========================================================================
3.5.0 (ekieboshi) 2016/01/29
* Version 3.5.0
This is the first version of pgpool-II 3.5 series.
That is, a "major version up" from 3.4 series.
__________________________________________________________________
* Incompatible changes
- Remove parallel query mode codes (Yugo Nagata)
There are very few users and the maintenance efforts are not worth it.
Codes for system db are also removed, since this was used in only parallel
query mode and old query cache on disk.
- Rename parameter name from ifconfig_path to if_cmd_path (Yugo Nagata)
ifconfig_path isn't appropriate for the parameter name because
ifconfig command is obsolete and now ip command is used as default.
- Change the syntax of pcp commands arguments (Muhammad Usama)
- Change the output and the argument semantics of pcp_watchdog_info command
(Muhammad Usama)
The command is enhanced to show watchdog information about all pgpool-II
nodes in the cluster. According to this, the argument to specify node index
is changed so that 0 means local pgpool-II node instead of the first remote
node.
__________________________________________________________________
* New features
- Add new parameter "serialize_accept" (Tatsuo Ishii)
This parameter defines whether to serialize accept() call for incoming
client connections. Default is off, which means no serializing (same
behavior as pgpool-II 3.4 or before). If this is off, the kernel
wakes up all of pgpool-II child process to execute accept() and one of
them actually accepts the incoming connection. Problem here is,
because so my child process wake up at a same time, heavy context
switching occurred and the performance is affected. This phenomena is
a classic problem called "the thundering herd problem". By enabling
serialize_accept, only one of pgpool-II child process is woken up and
executes accept() and the problem can be avoided.
- Import parser of PostgreSQL 9.5 (Yugo Nagata)
pgpool-II can understand new syntax introduced in PostgreSQL 9.5 now.
Especially GROUPING SET, CUBE, ROLLUP and TABLESAMPLE can be be load
balanced and can be used in query cache (except for TABLESAMPLE).
Also INSERT...ON CONFLICT and UPDATE tab SET (col1, col2, ...) =
(SELECT ...) ... can now be properly handled in query rewriting in
native replication mode.
- Allow to specify database for health check and streaming replication
delay check. (Tatsuo Ishii)
For this purpose new directive "health_check_database" and
"sr_check_database" are added.
__________________________________________________________________
* Enhancements
- Performance improvement in extened query protocol (Tatsuo Ishii)
Unnecessary "flush" messages which used to be sent in each step of
extended protocol messages (parse, bind, describe and execute) and
brought significant communication overhead are removed. For now it
only affects to streaming replication mode, that is, the performance
of other modes remains same.
- watchdog feature enhancements (Muhammad Usama, Yugo Nagata)
The goal of this enhancement is to address the shortcomings and problems
in the pgpool watchdog and make the watchdog system more robust and
adaptable. Patch created by Usama, and reviewed, tested, and debugged
by Yugo.
-- The watchdog should consider the quorum and only elect the master/leader
node if the quorum exist in the watchdog cluster.
-- All the participating pgpool-II nodes in the watchdog cluster should have
similar pgpool-II configurations.
-- Watchdog nodes should have configurable watchdog node priority, to give
users more control on which node should become a leader node.
-- More options for the node health-checking, especially watchdog should allow
external/3rd party node health checking system to integrate with it.
-- The watchdog should keep looking for problems like split-brain syndrome and
should automatically recover from it.
-- Allow user to provide scripts to be executed at time of escalation and
de-escalation to master/leader nodes.
Add these new parameters:
-- wd_ipc_socket_dir:
This parameter is used to specify the directory where the UNIX domain
socket accepting pgpool-II watchdog IPC connections will be created.
-- wd_priority:
This new parameter can be used to elevate the current watchdog node
priority in leader elections. The node with the higher wd_priority
value will get selected as master/coordinator watchdog node when
cluster will be electing its master node at cluster startup or in the
event of old master watchdog node failure.
-- wd_de_escalation_command:
This parameter holds the command that watchdog will execute on the
master pgpool-II watchdog node when that node resigns from the master
node responsibilities.
-- wd_monitoring_interfaces_list:
This parameter holds a comma separated list of network device names to
be monitored by the watchdog process for the network link state.
- Overhauling pcp commands (Muhammad Usama)
-- Improved handling of command argument:
Long command line options can be used now.
-- PCP password is safe:
Don't pass password via command line, which causes security risks.
-- Mutiple concurrent execution of pcp commands:
Allow to execute a pcp command while running pcp_recovery_node
which takes sometime to finish.
- SELECT count statics in "show pool_nodes" command results (Tatsuo Ishii)
show pool_nodes results which briefly describes the status of each backend
now shows how many SELECTs are issued to them. So you can quickly
recognize the effect of load balancing for example;
test=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt
---------+----------+-------+--------+-----------+---------+------------
0 | /tmp | 11002 | 2 | 0.500000 | primary | 338230
1 | /tmp | 11003 | 2 | 0.500000 | standby | 163939
(2 rows)
- Parser enhancements for query rewriting (Yugo Nagata)
INSERT/UPDATE/DELETE with WITH clause, writable CTE, RETURNING clause
can now be properly handled in query rewriting in native replication mode.
- Allow to open pool_passwd file in read only mode by pgpool-II main
process (Tatsuo Ishii)
- Allow to run pgpool in debug mode in regression tests (Tatsuo Ishii)
- doc:Add pgpool_adm documents (Tatsuo Ishii)
- doc:Update Chinese documents (Bambo Huang)
__________________________________________________________________
* Bug fixes (since 3.4.3)
- doc: Fix misinformation regarding load balancing in docs (Tatsuo Ishii)
In streaming replication mode, DECLARE, FETCH, CLOSE and SHOW are sent
to primary node only. Pointed out in [pgpool-general-jp: 1378].
- Issue fsync() when writing pgpool_status. (Tatsuo Ishii)
This ensures that pgpool_status is saved to permanent storage and
allow to survive after system crash.
- Fix reset query stuck problem (Muhammad Usama)
It is reported that reset query (DISCARD ALL etc.) occasionally does
not finish and pgpool child remain waiting for reply from the backend
thus client cannot connect to pgpool (for example bug report #107).
The cause of problem is not identified yet but if client suddenly
closes connection to pgpool in the middle of query processing, backend
may not accept the reset queries because they are not ready for query.
The fix is to keep track of the status of frontend socket state and not
caching the PostgreSQL connection if the connection to frontend was not
properly terminated.
- test: Fix false alerm of regression tset 062 (Yugo Nagata)
- test: Fix broken regression test for native replication (Yugo Nagata)
Test for insert-lock didn't work.
- test: Fix regress test to sleep after pgpool_reload (Yugo Nagata)
Some regression tests modify configuration file and reloads. However,
sometimes these were not applied to pgpool just after reloading.
- Fix inconsistency of sequence values in replication mode (Yugo Nagata)
When a schema name was provided, the table name was quoted wrongly as
like "public.mytbl" instead of "public"."mytbl". So, pool_regclass
and to_regclass couldn't find right talbe oid and insert lock was
never executed. This caused inconsistency between DB nodes.
- test: Fix complication error and segfault of timestamp rewrite test (Yugo Nagata)
- doc: Fix untranslated sentence in Japanese document (Yugo Nagata)
- Fix ancient bug of pool_push() and friends. (Tatsuo Ishii)
It allocates buffer using realloc and uses the pointer
returned. However it does the pointer calculation *before* realloc
gets called. So the calculation uses the old pointer value, which
causes various problems including segfault later.
Also there were other problems with it. The buffer pointer and buffer
size variable is not initialized. The buffer is not freed by
pool_close. Typo in debugging message (3.4 or later only). They are
fixed as well.
- doc: Fix wrong description in documents about log_standby_delay (Yugo Nagata)
- Fix segfalut that occurs when function is used in FROM clause (Yugo Nagata)
PRPARE statements in streaming-reaplication mode and INSERT/UPDATE
with SELECT in native-replication mode were affected. For example:
- prepare p as select * from generate_series(1,1);
- insert into tbl select now(), * from generate_series(1,1);
- doc: Add caution about JDBC driver version regarding app_name_redirect_preference_list
(Tatsuo Ishii)
- Fix hang problmen reported in bug #145. (Tatsuo Ishii)
The problem occurs when all the condition below are met:
1) pgpool-II 3.4 or later
2) streaming replication mode
3) primary node is also load balance node
4) extended protocol is used
5) client_idle_limit reached
- Fix a in-memory query cache bug (Tatsuo Ishii)
If extended query protocol is used and a bind/execute message arrives
which uses a statement created by prior parse message, the temp_cache
is not initialized by a parse messages. Thus messages are added to pre
existing temp cache buffer which causes the trouble, that is, when the
cached result returns, Data Row message and Command Complete message
appeared twice.
Per by bug #152.
- test: Fix regression test 065. (Tatsuo Ishii)
The path to JDBC driver was explicitly defined. This is wrong. This
should be inherited from JDBC_DRIVER environment variable.
- test: Fix possible hang-up of regression test 054.postgres_fdw (Yugo Nagata)
- test: Add option to regress.sh and pgpool_setup for unix domain socket
directory (Yugo Nagata)
- doc: Add missing descriptions about default values to documents
(Yugo Nagata)
- test: Fix regression test 055 for rhel7 rpm (Yugo Nagata)
- Fix reset query stuck problem (Muhammad Usama)
The issue is already fixed in older branches and this fix adopts the
same solution used by 3.3 series, i.e. closing the backend connection
when client idle limit is reached.
- Fix bug with "SET TRANSACTION READ ONLY" (Tatsuo Ishii)
Pgpool-II remembers that non read only queries (including SET) were
executed in an explicit transaction and adds a "writing transaction"
mark to the transaction. The mark affects the query routing behavior
of pgpool-II while running in streaming replication mode. Pgpool-II
starts sending queries to the primary after the mark is set. Because
the effect of writing queries may appear on standbys after some delay
in streaming replication mode, it is safer to route read queries to
the primary after the mark is set.
However there's oversight here. "SET TRANSACTION READ ONLY" does no
data modification and should be treated as an exception.
Per bug #157.
- test: Fix to use timeout command to handle time out of regress test 062
(Yugo Nagata)
- Fix to show wrong error (Tatsuo Ishii)
connect_with_timeout() does not show proper error info when
getsockopt(SO_ERROR) reports an error. Pointed out in bug #159.
- test: Add regress.sh missing \n in help messages (Yugo Nagata)
- Fix the logic issue in get_backends_status() function (Muhammad Usama)
get_backends_status () function counts the number of current valid and down
backend nodes. The function assumed that the node was also invalid when its
connection status was down. However, that is not always right.
- Fix white/black_memqcache_table_list not to require quotaion (Yugo Nagata)
The tables and schemas name specified in *_memqcache_table_list
were needed to be quoted by double quotation.
Patch contributed by Dang Minh Huong.
Per [pgpool-hackers: 1323]
- Fix FATAL error with reloading (Tatsuo Ishii)
While reloading pgpool.conf, the number of DB nodes is tentatively set
to 0, then counted up until reaching to the actual number of backends
by the pgpool main process. Unfortunately the variable was on the
shared memory and it confused pgpool child process when they were using
the variable and this caused FATAL error.
Per bug #156 report by harukat.
- test: Add some tests for white/black_memqcache_table_list to regression
test 006.memqcache (Yugo Nagata)
- Fix reset query stuck problem (Muhammad Usama, Tatsuo Ishii)
When pool_read fails to read from frontend or pool_flush fails to
write to the frontend, report FRONTEND_ERROR, rather than ERROR to
disconnect and terminate pgpool child process, to prevent the query
stuck problem.
- test: Fix some regression tests that failed in debug mode (Yugo Nagata)
- Fix performance degradation while using IPv6 (Muhammad Usama)
Per bug #165.
--
Yugo Nagata <nagata at sraoss.co.jp>
More information about the pgpool-general
mailing list