Deploy app via pipeline using ssh and PM2

Question

I am using Gitlab CE for CI/CD, and I have a pipeline to deploy to an ubuntu server, which runs the apps with pm2. The actions are:

• Removes /node_modules, /dist and /tmp folders
• Pulls updated code
• Builds the code into /dist
• Removes the app from pm2
• Start the app with pm2 again

Here is the pipeline:

image: node:16

before_script:
  - apt-get update -qq
  - apt-get install -qq git
  - 'which ssh-agent || ( apt-get install -qq openssh-client )'
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$K8S_SECRET_SSH_PRIVATE_KEY" | base64 -d)
  - ssh-add <(echo "$K8S_SECRET_SSH_PRIVATE_KEY_AWS" | base64 -d)
  - mkdir -p ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

stages:
  - deploy_production

deploy_production:
  stage: deploy_production
  only:
    - master
  script:
    - ssh [email protected] "cd /var/www/html/example2/example2.com && rm -rf node_modules dist/* tmp"
    - ssh [email protected] "cd /var/www/html/example2/example2.com && git pull origin master"
    - ssh [email protected] "source ~/.bash_profile && export NPM_TOKEN=${NPM_TOKEN} && export SECRET=${SECRET} && cd /var/www/html/example2/example2.com && nvm use --delete-prefix && git checkout master && npm ci --verbose && npm run prod"
    - ssh [email protected] "cd /var/www/html/example2/example2.com && pm2 delete example2.com"
    - ssh [email protected] "cd /var/www/html/example2/example2.com && export SECRET=${SECRET} && NODE_ENV=production pm2 start ./dist/server.js --name example2.com --update-env"

Now, when the job succeeds, if I enter the production server and list pm2 instances I see:

┌─────┬───────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name          │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼───────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 80  │ example2.com  │ default     │ 0.0.1   │ fork    │ 3258598  │ 0s     │ 0    │ online    │ 0%       │ 13.7mb   │ foo      │ disabled │
└─────┴───────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

But when I try to access the app, it is down. Now, if I access via ssh directly and delete and start it manually, it will work:

ssh [email protected]
$ pm2 delete example2.com
$ export SECRET=my_secret && NODE_ENV=production pm2 start ./dist/server.js --name example2.com --update-env

If someone has any idea why doing it directly it is working, but doing it via pipline is not, please let me know.

Answer 1

one thing you could double-check would be your user permissions that you are using inside the gitlab runner to ssh - are these creds different to the ones you use when doing it manually?

I would also suggest maybe changing your approach to how you are deploying to your production server. Keeping a copy of a git repo on it doesn't sound very safe to me. Instead, I would look to ship only the compiled binaries or dist folder in your case and not remotely run git based commands on another server.

You could also install a self-hosted gitlab runner directly onto your production server with limited user permissions for the gitlab-runner user, then creating a build job to generate and zip your dist folder from your project. Take this zipped folder as an artifact and then you can have another job on the production server to "unzip" and deploy with the correct user permissions.

what I mean by this is:

1. Install a self hosted gitlab (bash based would work fine) runner on your production server.
2. Edit your ci script to build your dist folder, zip it and store it as an artifact in gitlab.
3. Have a production job stage that will download the artifact (happens automatically within a pipeline) and unzip in the correct location
4. amend the user permissions with chmod to make sure the unzipped dist version of your app is owned by the correct user - for example if you are running nginx then the nginx user needs to own the files under the directory that you want to serve.

Hope the user permissions thing helps :)