Running Pythonscript over SSH from a Jenkinsagent doesn't start the script

Question

Update:

The problem seems to be quite different from what I initially thought. Information about this is at the bottom of the post.

I've been having quite the struggle with starting my Python-script on a remote Raspberry Pi. I can't quite get my script working, either it's running, but blocking the terminal in Jenkins - blocking the deployment from ever finishing, or it's not starting at all. The weird thing in my opinion, is that when I'm running the exact same commands from my Windows machine, or even directly from the Jenkins node (Ubuntu Server), the script starts absolutely fine and without blocking the terminal. It just behaves like the good background-process I want it to be. But no! Not through the Jenkins pipeline itself.

I'll summarize the setup:

• Jenkins controller: Docker container in my Unraid Server with SWAG reverse proxy for GitHub hook.
• Jenkins node: VM in Unraid Server.
• Target machine: Raspberry Pi, currently available on my local network (will set up VPN for this later).
• Target script: A python script to update a small screen through GPIO with information gotten through a HTTP GET request, has a while True loop as the main loop and some time.sleep().

The following is my pipeline:

pipeline {
    agent any
        stage('Deploy') {
            steps {
            sh 'ls'
                sshagent(credentials: ['jenkinsvm-to-pi'])
                {
                    // Clear any existing instances
                    sh """
                    ssh ${target} pkill -f ${filename} & 
                    """
                    // Copy downloaded files to target machine.
                    sh """
                    scp lcd1602.py ${target}:home/pi/Documents/${filename} 
                    """
                    // Run script on target machine.
                    sh """
                    ssh ${target} "nohup python3 home/pi/Documents/${filename}" & 
                    """
                }
            }
        }
}

This kills the process if I had started it through other means than from the pipeline (the pkill -f command). But it does not start it after this. I have tried with lots of variations, with & and without.

The entire pipeline seems quite simple to me, and I can't for the life of me figure out what's causing this issue.

Output from Jenkins

I would greatly appreciate some assistance with this. Thanks!

EDIT1:

Also tried this setup:

      steps {
                sshagent(credentials: ['conn-to-mmrasp'])
                {
                    sh "ssh ${target} pkill -f ${filename} || echo 'No process was running'"
                    sh "scp lcd1602.py ${target}:~/Documents"
                    sh "ssh ${target} nohup python3 Documents/${filename} &"
                }
            }

A pretty weird thing is that it seemed to work the first time I updated the pipeline. At least so it seemed, but subsequent attempts didn't work. The console log isn't very helpful either, but I'll include the output.

[ssh-agent] Using credentials pi
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-SSrqVoXbnr0v/agent.40947
SSH_AGENT_PID=40949
Running ssh-add (command line suppressed)
Identity added: /home/jenkins/workspace/AlbinTracker_main@tmp/private_key_4492026892118281470.key (/home/jenkins/workspace/AlbinTracker_main@tmp/private_key_4492026892118281470.key)
[ssh-agent] Started.
[Pipeline] {
[Pipeline] sh
+ ssh [email protected] pkill -f lcd1602.py
+ echo No process was running
No process was running
[Pipeline] sh
+ scp lcd1602.py [email protected]:~/Documents
[Pipeline] sh
+ ssh [email protected] nohup python3 Documents/lcd1602.py
[Pipeline] }
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 40949 killed;
[ssh-agent] Stopped.
[Pipeline] // sshagent
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] } 
[Pipeline] // node
[Pipeline] End of Pipeline

I've done some more testing. It seems like something is wrong even running the script from a terminal-SSH session. When I'm running the command: nohup python3 lcd1602.py & while I'm in a SSH session, there's no output to the nohup.out file, while when I'm not running it in nohup (python3 lcd1602.py &), I get the desired output directly in the terminal. If I run the command without the "&", the output is directed to nohup.out like it should, but this blocks the terminal, which clogs the deploy stage. I'm starting to think this is a problem related to nohup (my use of it) and not jenkins?

EDIT2:

I seem to have fixed a part of the problem, but another part remains.

1. I added the UNBUFFERED flag -u to the python command, to make sure that the logging went through instead of being stored in the buffer.

2. I added a bash script to the repository, that did the executing instead of running the python script directly.

Now I have some logging to make sure that I know when the script has been running etc. But I do think that the added bash script was actually what made the script start running. This works on my Raspberry Pi 3 B. When I try to run the same pipeline targeting my Raspberry Pi 1 B, it doesn't work... I'm faced with the same issue as before, the script isn't started at all (the RPi 1B is my original target machine, just did some testing with the RPi 3B).

The current Jenkinsfile:

String target = '[email protected]'
pipeline {
    agent any
    stages {
        stage('Deploy') {
            steps {
                sshagent(credentials: ['conn-to-mmrasp'])
                {
                    sh "scp -r ./* ${target}:~"
                    sh "ssh ${target} bash startScript.sh &"
                }
            }
        }
    }
}

The bash script:

filename=lcd1602.py

pkill -f $filename || echo No process running
nohup python3 -u $filename >> nohup.out

Update:

I just want to clarify the situation of the problem, so that you don't have to read the wall of text just to get to this. The same pipeline, with only the target IP changed, works on my RPi3B, starting the script and running the GPIO-screen etc. So it seems to be an issue with RPi1B, something is not entirely right with it, even though I've reflashed the OS. Any ideas on what might be the difference? The SSH configuration should not be a problem, as that would've been clear from the logs in Jenkins.