Command timed out errors while RabbitMQ is running on Azure container instance

Question

I'm using rabbitmq:3-management docker image on Azure container instance with the following path mounted on Azure storage account /var/lib/rabbitmq/mnesia

Sometimes the following error is received:

[38;5;160m2022-04-27 10:39:14.916059+00:00 [error] <0.366.0> Command timed out: '/usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@localhost'[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0> ** Generic server rabbit_disk_monitor terminating[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0> ** Last message in was update[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0> ** When Server state == {state,"/var/lib/rabbitmq/mnesia/rabbit@localhost",[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>                                50000000,5497558073344,100,10000,[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>                                #Ref<0.942492558.3792175105.38974>,false,true,[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>                                10,120000}[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0> ** Reason for termination ==[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0> ** {function_clause,[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>        [{lists,reverse,[{error,timeout}],[{file,"lists.erl"},{line,147}]},[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>         {string,tokens,2,[{file,"string.erl"},{line,1934}]},[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>         {rabbit_disk_monitor,parse_free_unix,1,[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>             [{file,"rabbit_disk_monitor.erl"},{line,262}]},[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>         {rabbit_disk_monitor,internal_update,1,[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>             [{file,"rabbit_disk_monitor.erl"},{line,216}]},[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>         {rabbit_disk_monitor,handle_info,2,[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>             [{file,"rabbit_disk_monitor.erl"},{line,166}]},[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>         {gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>         {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0>         {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}[0m
[38;5;160m2022-04-27 10:39:14.931111+00:00 [error] <0.366.0> [0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>   crasher:[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     initial call: rabbit_disk_monitor:init/1[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     pid: <0.366.0>[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     registered_name: rabbit_disk_monitor[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     exception error: no function clause matching[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>                      lists:reverse({error,timeout}) (lists.erl, line 147)[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>       in function  string:tokens/2 (string.erl, line 1934)[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>       in call from rabbit_disk_monitor:parse_free_unix/1 (rabbit_disk_monitor.erl, line 262)[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>       in call from rabbit_disk_monitor:internal_update/1 (rabbit_disk_monitor.erl, line 216)[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>       in call from rabbit_disk_monitor:handle_info/2 (rabbit_disk_monitor.erl, line 166)[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>       in call from gen_server:try_dispatch/4 (gen_server.erl, line 695)[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>       in call from gen_server:handle_msg/6 (gen_server.erl, line 771)[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.228.0>][0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     message_queue_len: 0[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     messages: [][0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     links: [<0.365.0>][0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     dictionary: [][0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     trap_exit: false[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     status: running[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     heap_size: 10958[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     stack_size: 29[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>     reductions: 93242225[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0>   neighbours:[0m
[38;5;160m2022-04-27 10:39:14.970878+00:00 [error] <0.366.0> [0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>     supervisor: {local,rabbit_disk_monitor_sup}[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>     errorContext: child_terminated[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>     reason: {function_clause,[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                 [{lists,reverse,[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                      [{error,timeout}],[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                      [{file,"lists.erl"},{line,147}]},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                  {string,tokens,2,[{file,"string.erl"},{line,1934}]},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                  {rabbit_disk_monitor,parse_free_unix,1,[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                      [{file,"rabbit_disk_monitor.erl"},{line,262}]},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                  {rabbit_disk_monitor,internal_update,1,[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                      [{file,"rabbit_disk_monitor.erl"},{line,216}]},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                  {rabbit_disk_monitor,handle_info,2,[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                      [{file,"rabbit_disk_monitor.erl"},{line,166}]},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                  {gen_server,try_dispatch,4,[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                      [{file,"gen_server.erl"},{line,695}]},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                  {gen_server,handle_msg,6,[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                      [{file,"gen_server.erl"},{line,771}]},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                  {proc_lib,init_p_do_apply,3,[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                      [{file,"proc_lib.erl"},{line,226}]}]}[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>     offender: [{pid,<0.366.0>},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                {id,rabbit_disk_monitor},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                {mfargs,{rabbit_disk_monitor,start_link,[50000000]}},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                {restart_type,{transient,1}},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                {shutdown,300000},[0m
[38;5;160m2022-04-27 10:39:14.989350+00:00 [error] <0.365.0>                {child_type,worker}][0m

If I'm trying to connect to the container and running the command /usr/bin/df -kP /var/lib/rabbitmq/mnesia/rabbit@localhost by myself I'm getting the following result immediately:

Any help understanding this error and resolving it will be appreciated.

Answer 1

Seems like the Mnesia database was corrupted. Had to delete it to get sudo service rabbitmq-server start going again !

$ sudo rm -rf /var/lib/rabbitmq/mnesia/

Also make sure that any stray rabbit processes are killed before clearing out

$ ps auxww | grep rabbit | awk '{print $2}' | sudo xargs kill -9

And then

$ sudo service rabbitmq-server start

There are more troubleshooting provided in this thread you can refer the same