Cloud-config configuration does not work with Ubuntu 16.04 LTS

[Sergey Arkhipov]

Detailed bug description:
cloud-config configuration on Ubuntu Xenial Xerus (16.04 LTS) do not work on MOS 9.0. In the same time, the same provision with Ubuntu Trusty Tahr (14.04 LTS) works as expected.

Steps to reproduce:
1. Upload Xenial Xerus image. I've used
https://cloud-images.ubuntu.com/xenial/20160516.1/xenial-server-cloudimg-amd64-disk1.img
2. Proceed to Horizon on 'Project -> Instances' page and push 'Launch instance'
3. Fill the form with following settings:
  * Details:
    - Instance name: <arbitrary name>
    - Count: 1
  * Source:
    - Select Xenial image
    - Create new volume: no
  * Flavor:
    - m1.small
  * Network
    - Select any network with access to external network
  * No changes on 'Network ports', 'Security groups', 'Key Pair', 'Metadata'
  * Configuration:
    - Put following 'Customization script':

#cloud-config
chpasswd:
  list: |
    ubuntu:ubuntu
  expire: False
packages:
  - stress

4. Launch instance
5. Try to access instance with login/password 'ubuntu/ubuntu'

Expected results:
1. User with login 'ubuntu' and password 'ubuntu' exists and accessible.
2. Package 'stress' is installed.

Actual result:
1. Cannot login with 'ubuntu' and password 'ubuntu'
2. Package is not installed.

At the same time, the same usecase with Ubuntu 14.04 LTS works

Reproducibility:
100%

Workaround:
1. Do not use Xenial yet.

Impact:
1. New LTS of Ubuntu does not work with MOS 9.0 completely.

Description of the environment:
- MOS 9.0 build 310
- OpenStack Release: Mitaka on Ubuntu 14.04
- Compute: KVM
- Network: Neutron with VLAN segmentation
- Storage Backends
    Ceph RBD for volumes (Cinder)
    Ceph RadosGW for objects (Swift API)
    Ceph RBD for ephemeral volumes (Nova)
    Ceph RBD for images (Glance)

Additional information:
Console log of Xenial Xerus: http://paste.openstack.org/show/497476/
Console log of Trusty Tahr: http://paste.openstack.org/show/497477/

[Roman Podoliaka (rpodolyaka)]

Sergey, I see the following error in the Xenial log:

[ 52.733180] cloud-init[960]: 2016-05-18 09:10:07,616 - util.py[WARNING]: Failed fetching userdata from url http://169.254.169.254/2009-04-04/user-data

i.e. the customization script you've provided hasn't been fetched by the instance.

Could you please attach the diagnostic snapshot? (we need logs from the compute node, as well as nova-api/neutron-metadata-proxy logs from all controllers) Cloud init logs from the instance would also be nice to have.

Another point is that if it's really a problem with the Ubuntu cloud image (as you claim Trusty works on the same environment 100% of times), then, perhaps, it's better just to report it directly to upstream, bypassing the MOS Linux team, as they don't have any control over ubuntu cloud images.

[Sergey Arkhipov (sarkhipov)]

Roman,

Please check output of Trusty, it has similar line but still - works;

Trusty:
2016-05-18 08:51:47,247 - util.py[WARNING]: Failed fetching metadata from url http://169.254.169.254/latest/meta-data/

Xenial:
[ 52.733180] cloud-init[960]: 2016-05-18 09:10:07,616 - util.py[WARNING]: Failed fetching userdata from url http://169.254.169.254/2009-04-04/user-data

Diagnostic snapshot: http://mos-scale-share.mirantis.com/env11/fuel-snapshot-2016-05-18_13-16-11.tar.xz

Also, I've reproduced the problem again with following instances:
UUID of Trusty instance: 5de68a3e-7012-4638-85b1-a3e00a27faaa (run on node-16)
UUID of Xenial instance: 8bc748dc-fa8c-4b5c-b10e-59290ec0fdb4 (run on node-49)

Logs from computes are here:
http://mos-scale-share.mirantis.com/env11/bug1583098-node16.tar.xz
http://mos-scale-share.mirantis.com/env11/bug1583098-node49.tar.xz

[Sergey Arkhipov (sarkhipov)]

Unfortunately, I do not have a possiblity to extract instance logs today, will do tomorrow.

[Dmitry Teselkin (teselkin-d)]

We're not supporting cloud-init from upstream Ubuntu images. If it works with Ubuntu Trusty then it means that cloud-init is incompatible with metadata service, so it's either mos-neutron or canonical.

[Ilya Shakhat (shakhat)]

Did the following (ISO 374, Neutron VXLAN, non-DVR):
1) Create glance image from https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img
2) Boot instance in admin project from an image created in 1) with the key injected
3) Open SSH port in security group
4) Assign floating IP to the instance

It's observed:
1) The instance is accessible via SSH by floating IP (was able to login)
2) According to instance console log Cloud-init is configured properly (Cloud-init v. 0.7.7 finished at Fri, 20 May 2016 12:46:47 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 83.14 seconds)

Thus the issue is not in Ubuntu 16.04

[Ilya Shakhat (shakhat)]

Addition to #5 - customization script works too:

#cloud-config
chpasswd:
  list: |
    ubuntu:ubuntu
  expire: False
packages:
  - stress

in the instance console:
[ 110.986323] cloud-init[1054]: Reading package lists...
[ 111.489426] cloud-init[1054]: Reading package lists...
[ 111.524817] cloud-init[1054]: Building dependency tree...
[ 111.560223] cloud-init[1054]: Reading state information...
[ 112.598604] cloud-init[1054]: The following NEW packages will be installed:
[ 113.142896] cloud-init[1054]: stress
[ 113.853319] cloud-init[1054]: 0 upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
[ 113.875288] cloud-init[1054]: Need to get 16.7 kB of archives.
[ 113.892353] cloud-init[1054]: After this operation, 75.8 kB of additional disk space will be used.
[ 113.914900] cloud-init[1054]: Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 stress amd64 1.0.4-1 [16.7 kB]
[ 114.778712] cloud-init[1054]: Fetched 16.7 kB in 0s (20.8 kB/s)
[ 114.844837] cloud-init[1054]: Selecting previously unselected package stress.
[ 115.274045] cloud-init[1054]: (Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 53577 files and directories currently installed.)
[ 115.412621] cloud-init[1054]: Preparing to unpack .../stress_1.0.4-1_amd64.deb ...
[ 115.445318] cloud-init[1054]: Unpacking stress (1.0.4-1) ...
[ 115.476833] cloud-init[1054]: Processing triggers for man-db (2.7.5-1) ...
[ 116.771810] cloud-init[1054]: Processing triggers for install-info (6.1.0.dfsg.1-5) ...
[ 117.359604] cloud-init[1054]: Setting up stress (1.0.4-1) ...

[Ilya Shakhat (shakhat)]

Taking #5 and #6 into account moving bug into Invalid state

[Dina Belova (dbelova)]

Moving bug to invalid state after more than a month in the incomplete. If the issue is still in place, please mark it as confirmed and comment if Ilya's comments did help somehow.