You are hereBlogs / Technology's blog / Recovering a root file system error in Amazon EC2 Ubuntu instance

Recovering a root file system error in Amazon EC2 Ubuntu instance

By Technology - Posted on 01 October 2014

First, do not play around with your home directory permissions

Recently, I was playing around with my Ubuntu free tier t2.micro instance and used a sudo command to change the access rights of the /home/ubuntu directory to 777. I promptly logged out of my putty session without realizing that EC2 does not take this easy. Result: My next putty login failed ( with an error like "server returned public key") and ftp stopped working as well. I had lots of unsaved work on that instance. I thought I was doomed. My first impulse was to let go, reinitialize the instance and restart from the ground up.

Second, All is not lost when you do. Don't panic

However, with some patience and Googling around, I found that I could solve the problem easily and recover my work. The solution was to

  • create another Ubuntu t2.micro instance
  • detach the root volume from the broken instance after stopping it
  • attach it to the newly created instance (after stopping it) as a second volume
  • start the newly created instance
  • putty into the second instance. Your private key-pair will work for this instance
  • mount the second volume in this instance to a newly created directory
  • restore the home directory permissions to pristine conditions expected by EC2
  • unmount the second volume
  • detach the volume from this instance after stopping it
  • reattach it as root volume of the broken instance
  • restart the broken instance

Voila! putty session started working again for me.

Detailed Procedure

The detailed procedure is as follows:

Stop (do not terminate) the broken instance A, wait for it to come to a complete stop, detach the root EBS volume from the instance, wait for it to be detached, then attach the volume to instance B as /dev/sdf.

All the operations of starting/stopping instances and attaching/detaching volumes can be performed through your browser based EC2 Management Console.Normally, I found all root volumes are attached as /dev/sda1

ssh (using your private key-pair) to instance B and mount the volume so that you can access its file system.

ssh [instance b]

sudo mkdir -m 000 /vol-a
sudo mount /dev/sdf1 /vol-a

Fix It

At this point your entire root file system from instance A is available for viewing and editing under /vol-a on instance B. For example, you may want to:

  • Change directory to /vol-a by doing cd /vol-a

  • Change permissions of directory back to original state using sudo chmod 755 home/ubuntu

  • Check that the permissions do not allow write access to group and others

  • If required copy all important files out of /vol-a/

Note: Please cross check and make sure that the owner of the home/ubuntu directory is still ubuntu. This normally should not be a problem as I found that both instances were identically configured with respect to user and group ids.

Finish up

After you are done and you are happy with the files under /vol-a, unmount the file system (still on instance-B):

sudo umount /vol-a
sudo rmdir /vol-a

Go back to the EC2 Management Console and reattach the broken instance's root volume (that you fixed just now) back as /dev/sda. Start this instance now. Your putty login should work like a breeze.

Facebook Comments Box

Tags for Recovering a root file system error in Amazon EC2 Ubuntu instance