<div><div>Ok, thanks for the update.</div></div><div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 27 Jun 2022 at 3:12 Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">I am investigating this issue using pgproto with the data you<br>
provided. It seems there are lots of "Flush" messages in the data,<br>
and I think the Flush message are causing troubles in pgpool. I guess<br>
it will take some time before I find a solution.<br>
<br>
> Any thoughts?<br>
> On Wed, 22 Jun 2022 at 8:45 Avi Raboah <<a href="mailto:avi.raboah@gmail.com" target="_blank">avi.raboah@gmail.com</a>> wrote:<br>
> <br>
>> Yea you right<br>
>><br>
>> On Wed, 22 Jun 2022 at 8:45 Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
>><br>
>>> The default for backend_clustering_mode varies depending on the pgpool<br>
>>> versions. Can I assume you are using 4.3?<br>
>>><br>
>>> > All the configurations I didn’t provided are using the default.<br>
>>> ><br>
>>> > One more thing, the issue is occurring only if query cache enabled<br>
>>> ><br>
>>> > Thanks,<br>
>>> ><br>
>>> > Avi<br>
>>> ><br>
>>> > On Wed, 22 Jun 2022 at 4:10 Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>> wrote:<br>
>>> ><br>
>>> >> What is backend_clustering_mode?<br>
>>> >><br>
>>> >> > Pgpool.conf-<br>
>>> >> ><br>
>>> >> > listen_addresses = '*'<br>
>>> >> ><br>
>>> >> > port = '9999'<br>
>>> >> ><br>
>>> >> > socket_dir = '/var/run/pgpool'<br>
>>> >> ><br>
>>> >> > pcp_socket_dir = '/var/run/pgpool'<br>
>>> >> ><br>
>>> >> > wd_ipc_socket_dir = '/var/run/pgpool'<br>
>>> >> ><br>
>>> >> > pid_file_name = '/var/run/pgpool/pgpool.pid'<br>
>>> >> ><br>
>>> >> > backend_hostname0 = db_instance<br>
>>> >> ><br>
>>> >> > backend_port0 = '5432'<br>
>>> >> ><br>
>>> >> > backend_flag0 = 'ALLOW_TO_FAILOVER'<br>
>>> >> ><br>
>>> >> > backend_weight0 = '1'<br>
>>> >> ><br>
>>> >> > connection_cache = on<br>
>>> >> ><br>
>>> >> > num_init_children = 200<br>
>>> >> ><br>
>>> >> > child_life_time = 0<br>
>>> >> ><br>
>>> >> > max_pool = 4<br>
>>> >> ><br>
>>> >> > log_destination = 'stderr'<br>
>>> >> ><br>
>>> >> > log_line_prefix = '%m: %a pid %p: '<br>
>>> >> ><br>
>>> >> > log_connections = off<br>
>>> >> ><br>
>>> >> > log_disconnections = off<br>
>>> >> ><br>
>>> >> > log_statement = off<br>
>>> >> ><br>
>>> >> > log_client_messages = on<br>
>>> >> ><br>
>>> >> > client_min_messages = info<br>
>>> >> ><br>
>>> >> > log_min_messages = debug<br>
>>> >> ><br>
>>> >> > logging_collector = on<br>
>>> >> ><br>
>>> >> > log_directory = '/tmp/pgpool_logs'<br>
>>> >> ><br>
>>> >> > log_filename = 'pgpool-%Y-%m-%d_%H%M%S.log'<br>
>>> >> ><br>
>>> >> > log_file_mode = 0600<br>
>>> >> ><br>
>>> >> > log_truncate_on_rotation = off<br>
>>> >> ><br>
>>> >> > log_rotation_age = 1d<br>
>>> >> ><br>
>>> >> > log_rotation_size = 200MB<br>
>>> >> ><br>
>>> >> > sr_check_user = 'user'<br>
>>> >> ><br>
>>> >> > sr_check_password = 'password'<br>
>>> >> ><br>
>>> >> > sr_check_period = '10'<br>
>>> >> ><br>
>>> >> > memory_cache_enabled = 'on'<br>
>>> >> ><br>
>>> >> > memqcache_total_size = 64MB<br>
>>> >> ><br>
>>> >> > memqcache_max_num_cache = 1000000<br>
>>> >> ><br>
>>> >> > memqcache_maxcache = 400kB<br>
>>> >> ><br>
>>> >> > memqcache_cache_block_size = 1MB<br>
>>> >> ><br>
>>> >> > memqcache_oiddir = '/var/log/pgpool/oiddir'<br>
>>> >> ><br>
>>> >> > cache_safe_memqcache_table_list = ''<br>
>>> >> ><br>
>>> >> > cache_unsafe_memqcache_table_list = ''<br>
>>> >> ><br>
>>> >> > check_unlogged_table = on<br>
>>> >> ><br>
>>> >> > health_check_period = '10'<br>
>>> >> ><br>
>>> >> > health_check_user = 'user'<br>
>>> >> ><br>
>>> >> > failover_on_backend_error = 'off'<br>
>>> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> >> > On Tue, 21 Jun 2022 at 10:51 Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> wrote:<br>
>>> >> ><br>
>>> >> >> Please share the log with log_client_messages = on so that I can see<br>
>>> >> >> what your client actually is sending. Also I need pgpool.conf.<br>
>>> >> >><br>
>>> >> >> > We are using asyncPg module in python in order to send our<br>
>>> queries the<br>
>>> >> >> > difference between that module to psycopg is that asyncpg using<br>
>>> >> extended<br>
>>> >> >> > queries.<br>
>>> >> >> > And I can see that the issue is reproduced in case transaction<br>
>>> aborted<br>
>>> >> >> and<br>
>>> >> >> > the catalog queries are launched.<br>
>>> >> >> ><br>
>>> >> >> > I checked in the code and found that<br>
>>> check_transaction_state_and_abort<br>
>>> >> >> > happens only in simpleQuery process.<br>
>>> >> >> ><br>
>>> >> >> > Thanks,<br>
>>> >> >> ><br>
>>> >> >> > Avi<br>
>>> >> >> ><br>
>>> >> >> > On Tue, 21 Jun 2022 at 6:43 Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> wrote:<br>
>>> >> >> ><br>
>>> >> >> >> Can you elaborate more?<br>
>>> >> >> >><br>
>>> >> >> >> I tried following case and pgpool ignores all queries from client<br>
>>> >> >> >> after a parse error occurs as expected.<br>
>>> >> >> >><br>
>>> >> >> >> pgproto -d test -p 11000 -f ../pgproto.data<br>
>>> >> >> >> FE=> Parse(stmt="", query="BEGIN")<br>
>>> >> >> >> FE=> Bind(stmt="", portal="")<br>
>>> >> >> >> FE=> Execute(portal="")<br>
>>> >> >> >> FE=> Parse(stmt="", query="INSERT INTO non_existing_table<br>
>>> values(1)")<br>
>>> >> >> >> FE=> Bind(stmt="", portal="")<br>
>>> >> >> >> FE=> Execute(portal="")<br>
>>> >> >> >> FE=> Parse(stmt="", query="END")<br>
>>> >> >> >> FE=> Bind(stmt="", portal="")<br>
>>> >> >> >> FE=> Execute(portal="")<br>
>>> >> >> >> FE=> Sync<br>
>>> >> >> >> <= BE ParseComplete<br>
>>> >> >> >> <= BE BindComplete<br>
>>> >> >> >> <= BE CommandComplete(BEGIN)<br>
>>> >> >> >> <= BE ErrorResponse(S ERROR V ERROR C 42P01 M relation<br>
>>> >> >> >> "non_existing_table" does not exist P 13 F parse_relation.c L<br>
>>> 1381 R<br>
>>> >> >> >> parserOpenTable )<br>
>>> >> >> >> <= BE ReadyForQuery(E)<br>
>>> >> >> >> FE=> Terminate<br>
>>> >> >> >><br>
>>> >> >> >> > Hi,<br>
>>> >> >> >> ><br>
>>> >> >> >> > After investigating the code I found that the patch you<br>
>>> provided is<br>
>>> >> >> >> fixing<br>
>>> >> >> >> > the problem only for simple query. In extended query the issue<br>
>>> >> still<br>
>>> >> >> >> exist.<br>
>>> >> >> >> ><br>
>>> >> >> >> > Do you know where is the right place to add the transaction<br>
>>> failure<br>
>>> >> >> >> > verification in extended query process?<br>
>>> >> >> >> > On Mon, 9 May 2022 at 16:17 Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> >> wrote:<br>
>>> >> >> >> ><br>
>>> >> >> >> >> The patch (plus more fix) has been committed to master and 4.3<br>
>>> >> stable<br>
>>> >> >> >> >> branches.<br>
>>> >> >> >> >><br>
>>> >> >> >> >><br>
>>> >> >><br>
>>> <a href="https://www.pgpool.net/pipermail/pgpool-committers/2022-May/008592.html" rel="noreferrer" target="_blank">https://www.pgpool.net/pipermail/pgpool-committers/2022-May/008592.html</a><br>
>>> >> >> >> >><br>
>>> >> >><br>
>>> <a href="https://www.pgpool.net/pipermail/pgpool-committers/2022-May/008591.html" rel="noreferrer" target="_blank">https://www.pgpool.net/pipermail/pgpool-committers/2022-May/008591.html</a><br>
>>> >> >> >> >><br>
>>> >> >> >> >> The patches will appear in the next release (May 19, 2022).<br>
>>> >> >> >> >><br>
>>> >> >> >> >> I have not pushed to other stable branches because the fix is<br>
>>> not<br>
>>> >> >> >> >> small and I wanted to minimize risks.<br>
>>> >> >> >> >><br>
>>> >> >> >> >> > Thanks a lot for the patch have a great weekend<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> > On Sat, 7 May 2022 at 13:35 Tatsuo Ishii <<br>
>>> <a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a>><br>
>>> >> >> wrote:<br>
>>> >> >> >> >> ><br>
>>> >> >> >> >> >> > Hi,<br>
>>> >> >> >> >> >> ><br>
>>> >> >> >> >> >> > Still thinking.<br>
>>> >> >> >> >> >> ><br>
>>> >> >> >> >> >> > Best reagards,<br>
>>> >> >> >> >> >> > --<br>
>>> >> >> >> >> >> > Tatsuo Ishii<br>
>>> >> >> >> >> >> > SRA OSS, Inc. Japan<br>
>>> >> >> >> >> >> > English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>>> >> >> >> >> >> > Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>>> >> >> >> >> >> ><br>
>>> >> >> >> >> >> ><br>
>>> >> >> >> >> >> >> Hi,<br>
>>> >> >> >> >> >> >><br>
>>> >> >> >> >> >> >> Any news about this thread?<br>
>>> >> >> >> >> >> >><br>
>>> >> >> >> >> >> >> Thanks,<br>
>>> >> >> >> >> >> >> Avi.<br>
>>> >> >> >> >> >> >><br>
>>> >> >> >> >> >> >> On Mon, 25 Apr 2022 at 11:13 Avi Raboah <<br>
>>> >> <a href="mailto:avi.raboah@gmail.com" target="_blank">avi.raboah@gmail.com</a>><br>
>>> >> >> >> >> wrote:<br>
>>> >> >> >> >> >> >><br>
>>> >> >> >> >> >> >>> It will be perfect thanks a lot!<br>
>>> >> >> >> >> >> >>><br>
>>> >> >> >> >> >> >>> On Mon, 25 Apr 2022 at 10:47 Tatsuo Ishii <<br>
>>> >> <a href="mailto:ishii@sraoss.co.jp" target="_blank">ishii@sraoss.co.jp</a><br>
>>> >> >> ><br>
>>> >> >> >> >> wrote:<br>
>>> >> >> >> >> >> >>><br>
>>> >> >> >> >> >> >>>> > No, it doesn't.<br>
>>> >> >> >> >> >> >>>> > In that case when you make the select statement,<br>
>>> piggy<br>
>>> >> >> >> internal<br>
>>> >> >> >> >> >> queries<br>
>>> >> >> >> >> >> >>>> > failed and the session rebooted<br>
>>> >> >> >> >> >> >>>><br>
>>> >> >> >> >> >> >>>> I see it now.<br>
>>> >> >> >> >> >> >>>><br>
>>> >> >> >> >> >> >>>> test=# begin;<br>
>>> >> >> >> >> >> >>>> BEGIN<br>
>>> >> >> >> >> >> >>>> test=*# insert into ttt values(1);<br>
>>> >> >> >> >> >> >>>> ERROR: relation "ttt" does not exist<br>
>>> >> >> >> >> >> >>>> LINE 1: insert into ttt values(1);<br>
>>> >> >> >> >> >> >>>> ^<br>
>>> >> >> >> >> >> >>>> test=!# select * from t1;<br>
>>> >> >> >> >> >> >>>> FATAL: Backend throw an error message<br>
>>> >> >> >> >> >> >>>> DETAIL: Exiting current session because of an error<br>
>>> from<br>
>>> >> >> >> backend<br>
>>> >> >> >> >> >> >>>> HINT: BACKEND Error: "current transaction is aborted,<br>
>>> >> >> commands<br>
>>> >> >> >> >> >> ignored<br>
>>> >> >> >> >> >> >>>> until end of transaction block"<br>
>>> >> >> >> >> >> >>>> server closed the connection unexpectedly<br>
>>> >> >> >> >> >> >>>> This probably means the server terminated<br>
>>> >> abnormally<br>
>>> >> >> >> >> >> >>>> before or while processing the request.<br>
>>> >> >> >> >> >> >>>> The connection to the server was lost. Attempting<br>
>>> reset:<br>
>>> >> >> >> Succeeded.<br>
>>> >> >> >> >> >> >>>><br>
>>> >> >> >> >> >> >>>> I think pgpool could remember that current<br>
>>> transaction is<br>
>>> >> in<br>
>>> >> >> >> abort<br>
>>> >> >> >> >> >> >>>> status and does not issue the piggyback query when the<br>
>>> >> selec<br>
>>> >> >> is<br>
>>> >> >> >> >> >> >>>> issued. Let me see what we can do for this.<br>
>>> >> >> >> >> >><br>
>>> >> >> >> >> >> I have come up with a patch. In this patch pgpool checks if<br>
>>> >> >> current<br>
>>> >> >> >> >> >> transaction is in abort status. If so, it does not accept<br>
>>> new<br>
>>> >> >> query<br>
>>> >> >> >> at<br>
>>> >> >> >> >> >> all and just reply back frontend with the message:<br>
>>> >> >> >> >> >><br>
>>> >> >> >> >> >> "current transaction is aborted, commands ignored until<br>
>>> end of<br>
>>> >> >> >> >> transaction<br>
>>> >> >> >> >> >> block"<br>
>>> >> >> >> >> >><br>
>>> >> >> >> >> >> until commit or abort command is sent from frontend. After<br>
>>> >> >> applying<br>
>>> >> >> >> >> >> the patch the example session above looks like this:<br>
>>> >> >> >> >> >><br>
>>> >> >> >> >> >> test=# begin;<br>
>>> >> >> >> >> >> BEGIN<br>
>>> >> >> >> >> >> test=*# insert into ttt values(1);<br>
>>> >> >> >> >> >> ERROR: relation "ttt" does not exist<br>
>>> >> >> >> >> >> LINE 1: insert into ttt values(1);<br>
>>> >> >> >> >> >> ^<br>
>>> >> >> >> >> >> test=!# select * from t1;<br>
>>> >> >> >> >> >> ERROR: current transaction is aborted, commands ignored<br>
>>> until<br>
>>> >> >> end of<br>
>>> >> >> >> >> >> transaction block<br>
>>> >> >> >> >> >><br>
>>> >> >> >> >> >> The patch was tested in master branch, but I believe it<br>
>>> can be<br>
>>> >> >> >> >> >> applied to 4.3 stable branch as well.<br>
>>> >> >> >> >> >><br>
>>> >> >> >> >> >> Best reagards,<br>
>>> >> >> >> >> >> --<br>
>>> >> >> >> >> >> Tatsuo Ishii<br>
>>> >> >> >> >> >> SRA OSS, Inc. Japan<br>
>>> >> >> >> >> >> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>>> >> >> >> >> >> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>>> >> >> >> >> >><br>
>>> >> >> >> >><br>
>>> >> >> >><br>
>>> >> >><br>
>>> >><br>
>>><br>
>><br>
</blockquote></div></div>
</div>