'Gitlab CI: How do I use the environment variable from one stage as needs:project ref in another

I have two jobs in the same project: job A and job B.

job A creates an environment variable EXTERNAL_PROJ_REF=some_tag and exports it through a .env file.

job B needs to download artifacts from an external_project and package them with other artifacts from the current project. I want to be able to dynamically choose the commit reference from which these external artifacts get downloaded. I am trying to use the environment variable EXTERNAL_PROJ_REF as the ref for external_project needed by job B.

job A:
  stage: build
  script:
    - echo "EXTERNAL_PROJ_REF=`./generate_variable.sh`" > build.env  # evaluates to EXTERNAL_PROJ_REF=some_tag
  artifacts:
    reports:
      dotenv: build.env

job B:
  stage: package
  script:
    - ./do_packaging_job.sh
  needs:
    - job: job A
      artifacts: true
    - project: external_project
      ref: $EXTERNAL_PROJ_REF
      job: external_job
      artifacts: true

When I run this pipeline though, job B instantly fails with the following error:

This job depends on other jobs with expired/erased artifacts:

If I hardcode ref to some_tag, the job does not fail, and I can confirm the EXTERNAL_PROJ_REF is successfully passed to job B.

job B:
  stage: package
  script:
    - echo "Ref = $EXTERNAL_PROJ_REF"  # Correctly prints "Ref = some_tag"
    - ./do_packaging_job.sh
  needs:
    - job: job A
      artifacts: true
    - project: external_project
      ref: some_tag   # hardcoded so the job doesn't fail
      job: external_job
      artifacts: true

However, when I have ref:$EXTERNAL_PROJ_REF, the pipeline fails. Can somebody tell me if I'm missing something?

gitlab gitlab-ci

Solution 1:^[1]

I finally realized Gitlab does not support what I want to do, at least not this way. According to this link, a variable passed from a different job can only be used in before_script, script or after_script sections of a job; it cannot be used to configure jobs. I cannot use it the needs section of job B.

Luckily, I found a simple workaround using the Gitlab API. I have API access to external_project, so I just use wget to download the artifact I need from the dynamically selected commit reference. Afterwards, I directly pass the artifact to job B.

job A:
  stage: build
  script:
    # Dynamically create commit reference
    - EXTERNAL_PROJ_REF=`./generate_commit_ref.sh`
    # Download artifact with Gitlab API
    - wget --header "PRIVATE-TOKEN:${EXTERNAL_PROJ_TOKEN}" --output-document outputFileName.file "${CI_API_V4_URL}/projects/${EXTERNAL_PROJ_ID}/jobs/artifacts/${EXTERNAL_PROJ_REF}/raw/${EXTERNAL_ARTIFACT_PATH}?job=${EXTERNAL_JOB_NAME}"
    # btw CI_API_V4_URL is a gitlab defined variable
  artifacts:
    paths:
      - outputFileName.file

job B:
  stage: package
  script:
    - ./do_packaging_job.sh
  needs:
    # Now this packaging job only needs job A. It doesn't need the external job anymore
    - job: job A
      artifacts: true

Solution 2:^[2]

A simple solution I use for sharing variables between two GitLab CI jobs:

stages:
 - build
 - package
 
job A:
  stage: build
  script:
    - echo "EXTERNAL_PROJ_REF=`./generate_variable.sh`" > build.env
  artifacts:
    paths:
      - build.env

job B:
  stage: package
  before_script:
    - export $(cat build.env | xargs)
  script:
    - ./do_packaging_job.sh

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Ngoma
Solution 2	Davide Madrisan

'Gitlab CI: How do I use the environment variable from one stage as needs:project ref in another

Solution 1:[1]

Solution 2:[2]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]