EC2 error: cannot create temp file for here-document: Read-only file system

Question

Looks like my Ubuntu 14.04 EC2 made the fs read-only.

cd /var/  (pressing tab for autocomplete)

cannot create temp file for here-document: Read-only file system

But I have plenty of free space and memory is not full either:

Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-48-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Wed Feb  3 14:40:58 UTC 2016

  System load:  0.0                Processes:           126
  Usage of /:   14.9% of 11.67GB   Users logged in:     0
  Memory usage: 19%                IP address for eth0: 172.31.15.38
  Swap usage:   0%

df -hi:

/dev/xvda1       768K   85K  684K   12% /
none             251K     2  251K    1% /sys/fs/cgroup
udev             249K   387  249K    1% /dev
tmpfs            251K   309  250K    1% /run
none             251K     1  251K    1% /run/lock
none             251K     1  251K    1% /run/shm
none             251K     2  251K    1% /run/user

free:

             total       used       free     shared    buffers     cached
Mem:       2048484    1199420     849064       6248     180300     635596
-/+ buffers/cache:     383524    1664960

du -sch /tmp*

9.9M    /tmp
9.9M    total

What's the solution here? How can I fix the fs without losing my data?

Should I run:

mount -o remount,rw /

or should I reboot?

Thanks in advance!

Answer 1

Do you have btrfs filesystems? If so, when there´s not enough space for more snapshots, the OS changes their properties to read-only (including /tmp). For me, the solution was to delete snapshots and disable the snapper.

Use the following commands:

• snapper list #shows a numbered list of snapshots
• snapper delete nmbr #deletes snapshot number nmbr (retry after reboot if doesn't work at first)

Also, disable automatic snapshots by deleting corresponding files under /etc/cron.hourly, /etc/cron.daily, and so on.

Answer 2

I got the same issue and fixed the same as below: There was a bad arguments given by someone for /etc/fstab

Wrong Entry in "/etc/fstab":

[ec2-user@XXXXXXXXXXX ~]$ cat /etc/fstab
#
UUID=XXXXXX     /           xfs    defaults,noation  1 1

and Corrected entry is as below:

[ec2-user@XXXXXXXXXXX ~]$ cat /etc/fstab
#
UUID=XXXXXX     /           xfs    defaults,noatime  1 1

Answer 3

Maybe you don't have write permission to the /tmp/ directory.

Check permission and it should be look like

ls -ld /tmp  
drwxrwxrwt 10 root root 4096 Jun  5 11:32 /tmp

You can fix the permissions followed by

chmod a+rwxt /tmp