TestVM root FS can't be grown after boot

[Andrey Sledzinskiy]

version - 9.0-213
Steps:
1. Deploy next cluster - neutron vxlan, 3 controller, 2 compute, 1 cinder

2. Ssh to controller after deployment and create key pair
nova keypair-add instancekey
chmod 400 root/.ssh/instancekey_rsa

3. Create instance with keypair
nova boot myins --image 8ae0a8a0-f6c5-40b1-8131-9466fc3955e6 --flavor 1 --nic net-id=7d7da90b-0d44-439e-88e5-9cede5ccbd49 --key-name instancekey

4. Try to ssh to instance and check connectivity from controller using generated key
ip netns exec qdhcp-7d7da90b-0d44-439e-88e5-9cede5ccbd49 ssh -i /root/.ssh/instancekey_rsa -o 'StrictHostKeyChecking no' cirros@10.109.19.4 "ping -c 1 8.8.8.8"

Expected - password isn't required
Actual - password was asked

nova console-log - http://paste.openstack.org/show/494419/ :
failed to get http://169.254.169.254/2009-04-04/user-data
warning: no ec2 metadata for user-data
mkdir: can't create directory '.ssh': No space left on device

so ssh key wasn't added on instance

it also might be a problem with cirros image that has been changed a lot recently

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Roman Podoliaka (rpodolyaka)]

I tried the recent TestVM image from http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/pool/main/c/cirros-testvm/ on my 3-4 weeks old 9.0 environment, specifically package cirros-testvm_0.3.4-2~u14.04+mos2_amd64.deb

Both images provided by this package (AFAUI, there is an additional image with a newer kernel) are booted with the very same problem:

mkdir: can't create directory '.ssh': No space left on device
failed to make .ssh
Starting dropbear sshd: OK
GROWROOT: NOCHANGE: partition 1 is size 49152. it cannot be grown
/run/cirros/datasource/data/user-data was not '#!' or executable
/dev/root resized successfully [took 0.02s]

MOS Linux, could you please take a look? My current understanding is that the latest update of the image content was not followed by tweaking of the FS settings.

[Timur Nurlygayanov (tnurlygayanov)]

This bug affects Tempest tests results, marked as blocker for QA team.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Albert Syriy <email address hidden>
Review: https://review.fuel-infra.org/19820

[Albert Syriy (asyriy)]

The package is available from the link
http://perestroika-repo-tst.infra.mirantis.net/review/CR-19820/mos-repos//ubuntu/master

[Timur Nurlygayanov (tnurlygayanov)]

this issue is blocker for Tempest, priority raised to Critical.

We should fix it in 9.0 release.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19820
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: 0ecd1ca6dabb09156b30c5f4b63f46a449a57333
Author: Albert Syriy <email address hidden>
Date: Tue Apr 19 14:28:08 2016

Bug fix: TestVM root FS can't be grown after boot

# The full size of image was increased to 48 M.
# Initially full size of image was 24M (9M occupied by system).
# After adding new packages the system occupies 32 M (all image size).
# To keep the free space approximately the same new size was
# put up to 48 M.

Change-Id: I75c17dd840caabd42c95939f047370a2e0fab520
Closes-Bug: #1571704

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19934
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 58392062fd682d8a34c5fe890052b5e9c2017dfd
Author: Albert Syriy <email address hidden>
Date: Thu Apr 21 13:50:03 2016

Cherry-picked two commits from master e57920b and 0ecd1ca to update the kernel 3.13.0-85 and add tcpdump package

This is a cherry-pick of two commits. The first one adds tcpdump and
other packages into cirros image and updates the kernel to 3.13.0-85.
The second commit adds some space to the has grown image.

Closes-Bug: #1571704
Closes-Bug: #1260268

Change-Id: I89dbf2656b95edd18c9dedec96b70dc8a450c2d4

[Alexander Gubanov (ogubanov)]

Verified on MOS9.0 (build 232) -
I've created 2 instances from this new image:

with root_gb == 0 - image(disk) size was grown, but we still have the same size for partition and fs, so growpart/resize2fs didn't apply
Details: http://pastebin.com/WJuGcUY9

with root_gb == 1 - image(disk)/partition/fs were increased
Details: http://pastebin.com/A6BBZW1Z

So, my current understanding, that if we add more packages to this image in future and boot instances with flavor where disk == "0" we will get the same situation with disk space.
I guess, in this case (root_gb == 0) we should have "growpart/resize2fs" in this image in initrd.

[Alexander Gubanov (ogubanov)]

Please, correct me if I am wrong.

[Roman Podoliaka (rpodolyaka)]

Alexander, that's expected: root_db=0 tells Nova *not* to resize a disk image. It's up to the image author to ensure it works correctly without resize.