'GNU parallel - running command through multiple ssh jumphosts

I need to run commands on multiple hosts accessible via multiple jumphosts.

I have a slf file

ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@worker1
ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@worker2
ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@worker3
...

Running parallel I get

$ parallel --slf workers-ssh "ls -la"
parallel: Warning: Input is read from the terminal. You either know what you
parallel: Warning: are doing (in which case: YOU ARE AWESOME!) or you forgot
parallel: Warning: ::: or :::: or to pipe data into parallel. If so
parallel: Warning: consider going through the tutorial: man parallel_tutorial
parallel: Warning: Press CTRL-D to exit.
parallel: Warning: Could not figure out number of cpus on worker1 (). Using 1.

Jumphost and worker IPs are dynamically generated in previous scripts so I would prefer no ~/.ssh/config edits.

What is the correct way to solve this problem with gnu parallel?

linuxbashparallel-processinggnu


Solution 1:[1]

I think there's no big problem in your slf file but in the way you specify the command.

When you run parallel, you should specify the "variable" part, or parallel will ask you to enter that part in stdin just like your situation.

In your case, you have to fill the command in this way:

parallel --slf workers-ssh ::: "ls -la"

which is equivalent to

parallel --slf workers-ssh {} ::: "ls -la"

So far you should be able to issue commands on remote machines, but since only one argumant ("ls -la") is given, only one machine will be connected. If you want to issue ls -la on all machines, you may apply the following edit:

commands.txt

ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@worker1 "ls -la"
ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@worker2 "ls -la"
ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@worker3 "ls -la"

then run

parallel :::: commands.txt

Or another way

parallel ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@worker{} ls -a ::: {1..3}

Substitute by real IPs.

parallel ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2 user@{} ls -a ::: ip1 ip2 ip3

Solution 2:[2]

You might be able to use --ssh:

parallel --ssh 'ssh -i /root/.ssh/id_rsa -nJ jumphost1,user@jumphost2' -S user@worker1 ::: "ls -la"

Solution 3:[3]

There are two available options in GNU parallel designed for this use-case.

without arguments:

parallel --slf sshloginfile.txt --nonall "hostname"

and with arguments:

parallel --slf sshloginfile.txt --onall ::: hostname

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2 Ole Tange
Solution 3 Robert Smith

Related Questions