[regression] Deploying ISO triggers to an Error: DVR, NFV, Ceilometer, Sahara, Ironic

[Sergii Turivnyi]

The deployment of the environment with NFV, Neutron DVR, Sahara, Ceilometer and Ironic failed in 100% of cases. It looks like the root of the issue can be in NFV experimental feature or updated puppet manifests, QA team is still debugging the issue.

Detailed bug description:
Deploying ISO
=== ===

Triggers to an Error
========================
Error
Deployment has failed. All nodes are finished. Failed tasks: Task[ceilometer-compute/5], Task[ceilometer-compute/4] Stopping the deployment process!
========================

Steps to reproduce:
Task: https://mirantis.jira.com/browse/PROD-4355
1. Get ISO: http://srv52-bud.infra.mirantis.net/fuelweb-iso/fuel-9.0-mos-465-2016-06-09_22-51-38.iso.torrent
2. Nodes = 6
    Conroller + Mongo = 3
    Compute + Cinder = 2
    Ironic = 1
    Neutron DVR
    Nova quotas
    Cinder LVM over iSCSI for volumes
    OpenStack debug logging
    Install Sahara
    Install Ceilometer and Aodh
    Install Ironic
3. Add Repo
`nfv deb http://172.18.162.63/feature-nfv-repos/ubuntu/9.0/ mos9.0 main 1200`
4. Deploy

Expected results:
Deployment finishes successfully

Actual result:
Error
Deployment has failed. All nodes are finished. Failed tasks: Task[ceilometer-compute/5], Task[ceilometer-compute/4] Stopping the deployment process!

Reproducibility:
--

Workaround:
--

Impact:
--

Description of the environment:
see attachments

Additional information:
Snapshot: https://drive.google.com/a/mirantis.com/file/d/0B8hkiEm94sEtc1Nic0RtbW9fUkU/view?usp=sharing

See attachments

[Timur Nurlygayanov (tnurlygayanov)]

This bug is blocker for NFV feature, priority is critical because deployment of popular configuration failed.

[Timur Nurlygayanov (tnurlygayanov)]

It looks like a regression issue.

Dev team, could you please provide RCA and ETA for the fix?

Thank you!

[Alex Schultz (alex-schultz)]

2016-06-13 13:40:36 +0000 /Stage[main]/Ceilometer/User[ceilometer]/groups (err): change from adm,nova,libvirtd to adm,libvirt,libvirtd,nova failed: Could not set groups on user[ceilometer]: Execu
tion of '/usr/sbin/usermod -G adm,libvirt,libvirtd,nova ceilometer' returned 6: usermod: group 'libvirt' does not exist

[Alex Schultz (alex-schultz)]

Seems related to https://github.com/openstack/fuel-library/blob/master/deployment/puppet/openstack_tasks/manifests/ceilometer/compute.pp#L95-L102

[Sergii Turivnyi (sturivnyi)]

I've try to deploy without additional Repo
`nfv deb http://172.18.162.63/feature-nfv-repos/ubuntu/9.0/ mos9.0 main 1200`
and got another Error
http://s32.postimg.org/hfws5t2l1/Selection_093.jpg

[Sergii Turivnyi (sturivnyi)]

New Snapshot
https://drive.google.com/a/mirantis.com/file/d/0B8hkiEm94sEtY0dxWDhNZ09xMTA/view?usp=sharing

[Sergey Kolekonov (skolekonov)]

From logs:
2016-06-13T17:55:35.755348+00:00 warning: (Package[glance-api](provider=apt_fuel)) The following packages have unmet dependencies:
2016-06-13T17:55:35.755348+00:00 warning: (Package[glance-api](provider=apt_fuel)) glance-api : Depends: glance-common (= 2:12.0.0-3~u14.04+mos12) but it is not going to be installed
2016-06-13T17:55:35.757358+00:00 warning: (Package[glance-api](provider=apt_fuel)) E: Unable to correct problems, you have held broken packages.
2016-06-13T17:55:51.854041+00:00 warning: (Package[glance-api](provider=apt_fuel)) Attempt '3' of '3' has failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install glance-api' returned 100: Reading package lists...

Are you sure you had correctly disabled experimental repo before trying to re-deploy your cluster?

[Sergey Kolekonov (skolekonov)]

The env without custom repo was broken because it was deployed using Ubuntu image built with custom repo. So libxml2 package was installed to a clean cluster and broke it. Sergii is going to re-deploy a clean cluster with new Ubuntu image

[Sergii Turivnyi (sturivnyi)]

Yes, I'm sure

[Sergey Kolekonov (skolekonov)]

So basic envs should not be affected by this bug.
Speaking about experimental repo, libvirt packages from it uses libvirtd user/group instead of libvirt/libvirt which is currently hard-coded in fuel-library. Probably this problem should be fixed on packaging side

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: feature/nfv
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/22012

[Sergey Kolekonov (skolekonov)]

Moving the bug to Ivan Suzdal as he proposed a fix that will likely fix the problem.

[Timur Nurlygayanov (tnurlygayanov)]

Does it fix the issue even for cases without custom NFV repositories?

[Dina Belova (dbelova)]

Adding non-release tag due to previous Sergey's comments. This bug is reproducible *only* with experimental packages being included. This should be fixed by commit to packages/trusty/libvirt (feature/nfv branch).

[Timur Nurlygayanov (tnurlygayanov)]

Priority changed to High because we can't reproduce the issue on clear MOS 9.0 build without customizations.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/22012
Submitter: Pkgs Jenkins <email address hidden>
Branch: feature/nfv

Commit: d309c1f072021b5874f2c5bcf02555a5c939c74f
Author: Ivan Suzdal <email address hidden>
Date: Tue Jun 14 16:26:41 2016

Change libvirt group from "libvirtd" to "libvirt"

Change-Id: Iefef562e8c3bec3ba4002562a02e44dee96d59a5
Related-Bug: #1592019

[Timur Nurlygayanov (tnurlygayanov)]

The fix was merged for 9.0 as well (to experimental repository)

[Sergii Turivnyi (sturivnyi)]

Can reproduce with different error:
Deployment has failed. All nodes are finished. Failed tasks: Task[ironic_upload_images/1] Stopping the deployment process!

(/Stage[main]/Openstack_tasks::Openstack_cinder::Create_cinder_types/Osnailyfacter::Openstack::Manage_cinder_types[volumes_lvm]/Cinder_type[volumes_lvm]/ensure) change from absent to present failed: Command: 'openstack ["volume type", "create", "--format", "shell", ["--property", "volume_backend_name=LVM-backend", "volumes_lvm"]]' has been running for more then 20 seconds!

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 577
cat /etc/fuel_build_number:
 577
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6349.noarch
 fuelmenu-9.0.0-1.mos274.noarch
 fuel-notify-9.0.0-1.mos8460.noarch
 fuel-ostf-9.0.0-1.mos936.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8743.noarch
 fuel-mirror-9.0.0-1.mos141.noarch
 fuel-openstack-metadata-9.0.0-1.mos8743.noarch
 rubygem-astute-9.0.0-1.mos750.noarch
 fuel-misc-9.0.0-1.mos8460.noarch
 python-fuelclient-9.0.0-1.mos325.noarch
 fuel-9.0.0-1.mos6349.noarch
 fuel-utils-9.0.0-1.mos8460.noarch
 fuel-setup-9.0.0-1.mos6349.noarch
 nailgun-mcagents-9.0.0-1.mos750.noarch
 fuel-library9.0-9.0.0-1.mos8460.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-agent-9.0.0-1.mos285.noarch
 fuel-ui-9.0.0-1.mos2717.noarch
 fuel-migrate-9.0.0-1.mos8460.noarch
 python-packetary-9.0.0-1.mos141.noarch
 fuel-bootstrap-cli-9.0.0-1.mos285.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-nailgun-9.0.0-1.mos8743.noarch

[Sergii Turivnyi (sturivnyi)]

Looks like problem with the server.
Now I'm trying to redeploy =)

[Sergii Turivnyi (sturivnyi)]

Can reproduce. Error same.
Snapshot: http://172.18.198.49:8231/fuel-snapshot-2016-07-08_10-58-46.tar.gz

[Sergii Turivnyi (sturivnyi)]

Posted another bug:
https://bugs.launchpad.net/mos/+bug/1600791

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Dmitry Teselkin <email address hidden> on branch: 9.0
Review: https://review.fuel-infra.org/25427
Reason: Merged in https://review.fuel-infra.org/#/q/topic:group/prod-7907