getting "Jenkins is going to shutdown" message during long jobs

Question

Sometimes, when one of our longer builds is running (around 2 hours), Jenkins will start displaying the "Jenkins is going to shutdown ..." message. And no, this has not been done by an admin (me).

When I last saw this, I checked the console output of the running job, and it was still churning through it's tests and was running normally. It was not hung.

Then later, I checked again, and the console had the "BUILD SUCCESSFUL" message, followed by "Pausing (Preparing for shutdown)" - and it just sat there.

So I clicked on the kill job button, and killed it. and got the "Aborted by ..." message.

Then 15 seconds later it displayed "Click here to forcibly terminate running steps". I did that. It then displayed "Terminating withAnt".

Then 15 seconds later, it displayed "Click here to forcibly kill entire build". Which I did - and Jenkins return to normal operation and cleared the "going to shutdown" message.

WHAT IS GOING ON!

One related note: Due to getting too much "state" bleedthrough on our JUNits, we recently added the forkmode="perTest" setting to the Ant JUnit task. This has resulted in random tests failing with a "vm exited unexpectedly" message. It happens randomly for different tests. (which is a PITA since we can no longer count on Test Failed status in jenkins meaning anything.) And no, I'm not sure if that has always happened when the Jenkins job has the termination problem.

Answer 1

Well, I think I figured this out. The system was running low on disk space. So SOME jobs that used more were triggering this problem - and others would run without a problem.

When I finally received a low-disk space error in one of the logs, I did some cleaning (found a bunch of files that were supposed to have been deleted.). Since then, this error has stopped occuring.