Coco-annotator server crashes unexpectedly

Question

I'm running coco-annotator in a dedicated e2-micro Instance on GCP, and it first it was running smoothly, but recently it has been hanging very frequently, and not only the annotation url gets inaccessible, but the whole VM! I need to manually stop and start the instance on GCP.

I checked the docker-compose logs and these are the last logs before the machine hanging:

annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0> ** Generic server aten_detector terminating
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0> ** Last message in was poll
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0> ** When Server state == {state,#Ref<0.2293947595.4227596289.176964>,5000,0.99,
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0>                                #{},#{}}
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0> ** Reason for termination ==
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0> ** {{timeout,{gen_server,call,[aten_sink,get_failure_probabilities]}},
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0>     [{gen_server,call,2,[{file,"gen_server.erl"},{line,239}]},
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0>      {aten_detector,handle_info,2,[{file,"src/aten_detector.erl"},{line,109}]},
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0>      {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0>      {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0>      {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
annotator_message_q | 2022-04-19 19:09:16.735303+00:00 [erro] <0.183.0> 
annotator_message_q | 2022-04-19 19:09:18.480362+00:00 [erro] <0.183.0>   crasher:

It seems that the problem is linked to rabbitmq, but I have little to no knowledge about it. Can anyone point me some hints to where the problem may be?

Thanks in advance.

Answer 1

So, since this problem arose I changed to a bigger machine. It was 1GB RAM before, now it's running in a 4GB machine, and I haven't had this problem since then. This issue on github also seems to suggest that this was the problem.