View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000376Pgpool-IIBugpublic2018-01-12 17:482018-01-24 08:36
Reporteryremmet Assigned Tot-ishii  
PrioritynormalSeverityminorReproducibilityhave not tried
Status resolvedResolutionopen 
PlatformOpenstack/KVMOSUbuntuOS Version14.04.1
Product Version3.7.1 
Target Version3.7.2 
Summary0000376: child process was terminated by segmentation fault
DescriptionI'm getting the following error every time I execute a command.

2018-01-11 15:41:36 [No Connection] MAIN[23999] [No Connection]@[No Connection]: DEBUG: reaper handler
2018-01-11 15:41:36 [No Connection] MAIN[23999] [No Connection]@[No Connection]: WARNING: child process with pid: 24019 was terminated by segmentation fault
2018-01-11 15:41:36 [No Connection] MAIN[23999] [No Connection]@[No Connection]: LOG: fork a new child process with pid: 24204
2018-01-11 15:41:36 [No Connection] MAIN[23999] [No Connection]@[No Connection]: DEBUG: reaper handler: exiting normally
2018-01-11 15:41:36 [No Connection] CHILD[24204] [No Connection]@[No Connection]: DEBUG: initializing backend status

I'm using PostgreSQL-9.5.1 with streaming replication.
TagsNo tags attached.

Activities

yremmet

2018-01-12 17:48

reporter  

pgpool.conf (1,697 bytes)

yremmet

2018-01-15 16:43

reporter   ~0001892

I added the full log file.

yremmet

2018-01-15 16:51

reporter   ~0001893

The file was to big.

t-ishii

2018-01-16 15:59

developer   ~0001895

Please provide a stack trace.

BTW, your pgpool.conf lacks definitions of weight parameters. i.e. backend_weight0 etc.

yremmet

2018-01-16 21:27

reporter   ~0001896

Hi,
could you tell me how I get a stack trace of the failing process? I don't know gdb so well.

I found out that the error only occurs when pgpool is started with the --debug flag.

t-ishii

2018-01-17 10:18

developer   ~0001899

That depends on how you install/start Pgpool-II. Please provide the info.

yremmet

2018-01-18 01:11

reporter   ~0001900

I compile pgpool from source and its run by the attached control script.
ctl (1,201 bytes)    


 ctl (1,201 bytes)   
			




	




		
		
		

		

		

			 t-ishii		

		

						2018-01-18 07:46					

		

			developer			 
									
				~0001901			
					

				

		

		

				

		

			
	
	Probably just add following line to "ctl" is enough.

ulimit -c unlimited



Pgpool-II will produce a core file as soon as it segfaults. Then you can attach gdb to the core file like:



gdb path-to-pgpool-command core-file

bt



This will produce a stack trace.			




	




		
		
		

		

		

			 jhiemer		

		

						2018-01-23 16:20					

		

			reporter			 
									
				~0001907			
					

				

		

		

				

		

			
	
	Hi t-ishii,

on behalf of yremmet I have collected some logs and tried to get some data regarding the segmentation fault. What I was able to produce are the following files:



- plain-connect-no-gdb-attached.txt opening a connection to the database, which directly results in the segmentation fault as you can see from the logs.

- plain-connection-gdb-attached.txt opening a connection to the database, which does not result in a segmentation fault.



For both cases it is not possible to connect to the instances. The error is always:



"server closed the connection unexpectedly

This probably means the server terminated abnormally

before or while processing the request."



Attached is also the gdb log.



Directly connection to both postgres instances works flawlessly.


 gdb-logs.txt (2,069 bytes)   



 gdb-logs.txt (2,069 bytes)   


 plain-connect-no-gdb-attached.txt (28,832 bytes)   



 plain-connect-no-gdb-attached.txt (28,832 bytes)   


 plain-connection-gdb-attached.txt (28,210 bytes)   



 plain-connection-gdb-attached.txt (28,210 bytes)   
			




	




		
		
		

		

		

			 t-ishii		

		

						2018-01-23 16:35					

		

			developer			 
									
				~0001908			
					

				

		

		

				

		

			
	
	Thanks but none of these are useful to me. Since it seems it is hard for you to take a stack trace on a core file, I will show another method to get stack trace when a segfault occurs.



1. Connect to pgpool using psql.

2. Look into the log file to find out the child pgpool process pid. In the log file "2018-01-23 07:12:34 Postico 1.3.1 CHILD[5140]...", 5140 is the process id you want.

3. Attach gdb. Run gdb from another terminal.

gdb path-to-pgpool 5140 (of course you need to replace "5140" with actual process id)

4. run "cont" command in gdb prompt.

5.  Issue the query in question from psql.

6. gdb should react with the segafault and show a prompt. Type "bt" from the prompt and share with us.			




	




		
		
		

		

		

			 jhiemer		

		

						2018-01-23 16:58					

		

			reporter			 
									
				~0001909			
					

				

		

		

				

		

			
	
	Hi t-ishii,

thanks a lot for your quick reply. The command issued was a simple: "\dt"



postgres/8c52e1ba-127c-4282-bb2d-033b8cfbc628:/var/vcap/packages/postgres/bin# ./psql -h 10.244.0.3 -U admin

Password for user admin:

psql (9.5.10)

Type "help" for help.



admin=# \dt



I attached the segmentation fault. 



Thanks for your support!
 segmentation-fault.txt (485,055 bytes)
			




	




		
		
		

		

		

			 t-ishii		

		

						2018-01-23 18:17					

		

			developer			 
									
				~0001910			
					

				

		

		

				

		

			
	
	Thanks. I will look into this.	




	




		
		
		

		

		

			 t-ishii		

		

						2018-01-23 22:56					

		

			developer			 
									
				~0001911			
					

				

		

		

				

		

			
	
	It turned out if log_line_prefix includes %a and debug mode is on, an infinite recursion occurs, which eventually causes segfault.

Attached patch should fix the problem.


 elog.diff (1,143 bytes)   



 elog.diff (1,143 bytes)   
			




	




		
		
		

		

		

			 jhiemer		

		

						2018-01-23 23:06					

		

			reporter			 
									
				~0001912			
					

				

		

		

				

		

			
	
	Hi t-ishii,

thanks we are going to try it out.			




	




		
		
		

		

		

			 jhiemer		

		

						2018-01-23 23:13					

		

			reporter			 
									
				~0001913			
					

				

		

		

				

		

			
	
	Fixes the error. Are you going to put it into 3.6.9 and 3.7.2?	




	




		
		
		

		

		

			 t-ishii		

		

						2018-01-24 06:12					

		

			developer			 
									
				~0001914			
					

				

		

		

				

		

			
	
	Thanks for testing it out. Yes, we are going to put it into them (plus 3.5 and 3.4 series).	




	














	

		
		

	
		

	

		

						Issue History		

		

			
							
		

	

	

	

	

	

		
			

				
					Date Modified				
				
					Username				
				
					Field				
				
					Change				
			

		
	
		
				

				
					2018-01-12 17:48				
				
					yremmet				
				
					New Issue				
				
									
			

				

				
					2018-01-12 17:48				
				
					yremmet				
				
					File Added: pgpool.conf				
				
									
			

				

				
					2018-01-15 16:43				
				
					yremmet				
				
					Note Added: 0001892				
				
									
			

				

				
					2018-01-15 16:51				
				
					yremmet				
				
					Note Added: 0001893				
				
									
			

				

				
					2018-01-16 15:59				
				
					t-ishii				
				
					Note Added: 0001895				
				
									
			

				

				
					2018-01-16 21:27				
				
					yremmet				
				
					Note Added: 0001896				
				
									
			

				

				
					2018-01-17 10:18				
				
					t-ishii				
				
					Note Added: 0001899				
				
									
			

				

				
					2018-01-18 01:11				
				
					yremmet				
				
					File Added: ctl				
				
									
			

				

				
					2018-01-18 01:11				
				
					yremmet				
				
					Note Added: 0001900				
				
									
			

				

				
					2018-01-18 07:46				
				
					t-ishii				
				
					Note Added: 0001901				
				
									
			

				

				
					2018-01-18 07:47				
				
					t-ishii				
				
					Status				
				
					new => feedback				
			

				

				
					2018-01-19 09:58				
				
					t-ishii				
				
					Assigned To				
				
					 => t-ishii				
			

				

				
					2018-01-19 09:58				
				
					t-ishii				
				
					Status				
				
					feedback => assigned				
			

				

				
					2018-01-23 16:20				
				
					jhiemer				
				
					File Added: gdb-logs.txt				
				
									
			

				

				
					2018-01-23 16:20				
				
					jhiemer				
				
					File Added: plain-connect-no-gdb-attached.txt				
				
									
			

				

				
					2018-01-23 16:20				
				
					jhiemer				
				
					File Added: plain-connection-gdb-attached.txt				
				
									
			

				

				
					2018-01-23 16:20				
				
					jhiemer				
				
					Note Added: 0001907				
				
									
			

				

				
					2018-01-23 16:35				
				
					t-ishii				
				
					Note Added: 0001908				
				
									
			

				

				
					2018-01-23 16:36				
				
					t-ishii				
				
					Status				
				
					assigned => feedback				
			

				

				
					2018-01-23 16:58				
				
					jhiemer				
				
					File Added: segmentation-fault.txt				
				
									
			

				

				
					2018-01-23 16:58				
				
					jhiemer				
				
					Note Added: 0001909				
				
									
			

				

				
					2018-01-23 18:17				
				
					t-ishii				
				
					Note Added: 0001910				
				
									
			

				

				
					2018-01-23 18:17				
				
					t-ishii				
				
					Status				
				
					feedback => assigned				
			

				

				
					2018-01-23 22:56				
				
					t-ishii				
				
					File Added: elog.diff				
				
									
			

				

				
					2018-01-23 22:56				
				
					t-ishii				
				
					Note Added: 0001911				
				
									
			

				

				
					2018-01-23 22:57				
				
					t-ishii				
				
					Status				
				
					assigned => feedback				
			

				

				
					2018-01-23 22:57				
				
					t-ishii				
				
					Target Version				
				
					 => 3.7.2				
			

				

				
					2018-01-23 23:06				
				
					jhiemer				
				
					Note Added: 0001912				
				
									
			

				

				
					2018-01-23 23:13				
				
					jhiemer				
				
					Note Added: 0001913				
				
									
			

				

				
					2018-01-24 06:12				
				
					t-ishii				
				
					Note Added: 0001914				
				
									
			

				

				
					2018-01-24 08:36				
				
					t-ishii				
				
					Status				
				
					feedback => resolved