libvirt-lxc experimental gate unstable

[Thomas Maddox]

This has been an ongoing issue in an effort to get better testing upstream around Nova's Libvirt-LXC driver. The current experimental job is unstable and not ready for non-voting status. This bug is meant to track the progress towards stability for that job related to the testing infrastructure itself.

The specific job definition resides here: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L599-L632

Lots of details, notes, and simulation instructions can be found here: https://etherpad.openstack.org/p/lxc_driver_devstack_gate

Other related bugs/reviews:
Libvirt/LXC instability with reboots, specifically found on Ubuntu 14.04 LTS: https://bugs.launchpad.net/nova/+bug/1552740
Don't zero out logical volums when using LVM in devstack: https://review.openstack.org/#/c/215929
WIP patch used for invoking experimental pipeline upstream for Nova: https://review.openstack.org/#/c/274792

[Thomas Maddox (thomas-maddox)]

Recently switched to Fedora 23 for the upstream job due to local simulations looking for more stable for that distribution: https://review.openstack.org/#/c/293585/. This has apparently backfired for me, as now it seems to be failing even more consistently upstream.

So far all I can tell is the nova compute service stops responding at some point to the scheduler: http://logs.openstack.org/92/274792/17/experimental/gate-tempest-dsvm-lxc-f23/607a97e/logs/screen-n-sch.txt.gz#_2016-03-24_14_37_20_330

This causes the rest of the tests to fail as it never becomes available again, according to the scheduler logs.

I wasn't able to figure out why the compute service stopped sending heartbeats, however.

[Devdatta Kulkarni (devdatta-kulkarni)]

@thomas-maddox: Curious to know if you were seeing this behavior of nova compute service stopping responding to the scheduler in your local simulations as well?

[Devdatta Kulkarni (devdatta-kulkarni)]

More research and analysis is available on:

https://etherpad.openstack.org/p/lxc-driver-testing