Write log on "TIME LIMIT"

Question

I run a bunch of commands during my job. For example:

#!/bin/bash -l
#SBATCH --time 1-00:00:00
#SBATCH --nodes 1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --job-name foo
#SBATCH --out foo_%j.out

stdbuf -o0 -e0 mycommand input1.yaml
stdbuf -o0 -e0 mycommand input2.yaml
stdbuf -o0 -e0 mycommand input3.yaml
stdbuf -o0 -e0 mycommand input4.yaml
stdbuf -o0 -e0 mycommand input5.yaml
stdbuf -o0 -e0 mycommand input6.yaml
stdbuf -o0 -e0 mycommand input7.yaml

If my job get cut when it reaches the time limit, I would like to know where it was cut in order to be able to easily continue and/or remove corrupted data.

Answer 1

I don't know whether this is what you need, but anyways you could just print it out using echo.

#!/bin/bash -l
#SBATCH --time 1-00:00:00
#SBATCH --nodes 1
#SBATCH --ntasks 1
#SBATCH --cpus-per-task 1
#SBATCH --job-name foo
#SBATCH --out foo_%j.out

stdbuf -o0 -e0 mycommand input1.yaml
echo "Finished Job step 1"
stdbuf -o0 -e0 mycommand input2.yaml
echo "Finished Job step 2"
stdbuf -o0 -e0 mycommand input3.yaml
echo "Finished Job step 3"

You could also write this into a separate file like. For example, echo "message" >> job_record.out