Laravel: problem running a queue of chained jobs with supervisor (need chains to be worked sequentially, not in parallel)

Question

i really can't grasp queues; not enough to get a specific workflow to work as I intend.

I have a series of chained jobs (driver : database); and would like worker/s to finish the current chain before moving to the next chain.

Jobs are chained like this, handled by a command (php artisan FakeQueue 2):

    class FakeQueue extends Command{
    
        public function handle() {
          $server_id = $this->argument('server_id');
          $queue_name = 'backups_server_'.$server_id;
        
          $reference_uuid = (string) Str::orderedUuid(); // keep track of this as a whole
    
          $sites = [1, 2, 3, 4];
    
          foreach($sites as $site) {
            $chain_uuid = (string) Str::orderedUuid();
            // each fake job just logs:  get_class($this) . ' site_id ' . $site;
            $jobChain = [
              new FakeA($site, $chain_uuid, $reference_uuid),
              new FakeB($site, $chain_uuid, $reference_uuid)
            ];
            Bus::chain($jobChain)->onQueue($queue_name)->onConnection('database')->dispatch();
          }
        
          $command = 'supervisorctl start server-'. $server_id . '-backups-worker:*';
          shell_exec($command);
        }
    }

Now, i would like the worker to finish a chain of jobs, before passing to the next, something similar to:

with one proc:

- App\Jobs\FakeA site_id 1  
- App\Jobs\FakeB site_id 1 // finish site 1, then proceed!

- App\Jobs\FakeA site_id 2   
- App\Jobs\FakeB site_id 2 // finish site 2 ...

- App\Jobs\FakeA site_id 3 
- App\Jobs\FakeB site_id 3

- App\Jobs\FakeA site_id 4  
- App\Jobs\FakeB site_id 4

with two procs:

- App\Jobs\FakeA site_id 1 // worker1 starts first job for site 1 
- App\Jobs\FakeA site_id 2 // worker2 starts first job for site 2 (operation lasts a lot so second worker is busy, from now on)
- App\Jobs\FakeB site_id 1 // worker1 finishes jobs for site 1 --> proceed to next site
- App\Jobs\FakeA site_id 3 // worker1 picks next site and starts site 3

Unfortunately, i just get them executed in parallel:

local.NOTICE:  App\Jobs\FakeA site_id 1  
local.NOTICE:  App\Jobs\FakeA site_id 2  
local.NOTICE:  App\Jobs\FakeA site_id 3  
local.NOTICE:  App\Jobs\FakeA site_id 4  
local.NOTICE:  App\Jobs\FakeB site_id 1  
local.NOTICE:  App\Jobs\FakeB site_id 2  
local.NOTICE:  App\Jobs\FakeB site_id 3  
local.NOTICE:  App\Jobs\FakeB site_id 4

Any hints?

Answer 1

Chained jobs will execute in order relative to other jobs in the same chain but don't care about other chained jobs. This explains why FakeA is always before FakeB but runs in parallel with other FakeA.

If you want all the jobs to run in sequence then you might want to put everything in the same chain:

class FakeQueue extends Command{
    
        public function handle() {
          $server_id = $this->argument('server_id');
          $queue_name = 'backups_server_'.$server_id;
        
          $reference_uuid = (string) Str::orderedUuid(); // keep track of this as a whole
    
          $sites = [1, 2, 3, 4];
          $jobChain = [];
    
          foreach($sites as $site) {
            $chain_uuid = (string) Str::orderedUuid();
            // each fake job just logs:  get_class($this) . ' site_id ' . $site;
            $jobChain[] = new FakeA($site, $chain_uuid, $reference_uuid);
            $jobChain[] = new FakeB($site, $chain_uuid, $reference_uuid)
            
          }
          Bus::chain($jobChain)->onQueue($queue_name)->onConnection('database')->dispatch();
        
          $command = 'supervisorctl start server-'. $server_id . '-backups-worker:*';
          shell_exec($command);
        }
    }