'Supervisor stops/terminates before timeout

I know this might be a hard question as it concerns supervisord queueing but I hope someone would be able to try and think with me to solve an application-breaking problem.

The scenario I'm in at the moment is that I have an application that runs huge imports, each on its own supervisor thread/proc and even though the timeout of the queue is set to either 0 or 3600, the supervisor queue keeps terminating before the 3600 mark, generally after about 30-40 minutes.

I've tracked the RAM- & CPU-usage for an import but this could not be the problem as I am using max. 2GB / 8GB ram and the CPU is using only 1 thread with around 20% usage.

My supervisor queue looks as follows:

[program:laravel_queue]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/application/artisan queue:work --sleep=3 --tries=1 --queue=application --timeout=0
autostart=true
autorestart=true
user=administrator
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/application/storage/logs/queue/laravel_queue.out.log
stderr_logfile=/var/www/application/storage/logs/queue/laravel_queue.err.log

I've tried the same variant with --timeout=3600, but the queue still terminates at about 40 minutes.

I'm running the same setup in a Homestead/Laravel VM, and it runs perfectly in the Homestead VM.

Am I missing some redis configuration setting that terminates connections after 40 minutes? I'm running a cluster-system where there is 1 seperate - remote - redis-server which the application(s) talk to.

I want to provide more information but I'm not sure what more I could provide that would be helpful, so if you have any thoughts, I'm probably willing to provide most information.

Thanks in advance.

--- EDIT ---

I got a part of the supervisor log where a termination happens right here:

2017-11-28 12:54:34,151 INFO waiting for application_laravel_queue_02 to stop
2017-11-28 12:54:34,151 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:34,151 INFO waiting for application_laravel_queue_00 to stop
2017-11-28 12:54:34,151 INFO waiting for application_laravel_queue_01 to stop
2017-11-28 12:54:34,155 INFO stopped: application_laravel_queue_02 (terminated by SIGKILL)
2017-11-28 12:54:34,155 INFO stopped: application_laravel_queue_00 (terminated by SIGKILL)
2017-11-28 12:54:34,156 INFO stopped: application_laravel_queue_01 (terminated by SIGKILL)
2017-11-28 12:54:36,158 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:38,161 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:40,164 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:42,170 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:44,173 WARN killing 'application_laravel_queue_03' (1004) with SIGKILL
2017-11-28 12:54:44,174 INFO waiting for application_laravel_queue_03 to stop
2017-11-28 12:54:44,182 INFO stopped: application_laravel_queue_03 (terminated by SIGKILL)
2017-11-28 12:54:45,193 INFO spawned: 'application_laravel_queue_02' with pid 4235
2017-11-28 12:54:45,196 INFO spawned: 'application_laravel_queue_00' with pid 4236
2017-11-28 12:54:45,199 INFO spawned: 'application_laravel_queue_01' with pid 4237
2017-11-28 12:54:46,378 INFO success: application_laravel_queue_02 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-11-28 12:54:46,379 INFO success: application_laravel_queue_00 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-11-28 12:54:46,379 INFO success: application_laravel_queue_01 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

it looks like it it getting a termination signal (SIGKILL), but I don't know how or from where. The php-fpm logs is cleaned for this timeframe.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source