OpenStack Compute (nova)

Comment 0 for bug 1621818

Revision history for this message

Joris S'heeren (jsheeren) wrote on 2016-09-09: nova-compute unexpected input/output errors on starting instances

Our setup consists of multiple controllers and multiple hypervisors. Our shared storage for the instances is on a nfs 4.1 export. Using Ubuntu 16.04 LTS and Openstack Mitaka

When we launch an instance, nova updates the mtime for the _base image to let the image cache manager know the image is actively used. I think this was added here: https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=fb6ca3e7c8a38328d384cd41c061ded6623dac90
Because of this, in our setup, we are seeing unexpected input/output errors:

Stderr: u"/bin/touch: setting times of '/var/lib/nova/instances/_base/79e34519bacb47ad6f64c4baca4d33fd5c57d34d': Input/output error

A full trace can be found here: http://paste.openstack.org/show/570161/

This error particularly shows itself when launching multiple instances at once.

Also, because of this error, the instances are rescheduled. The assigned neutron ports, however, are not deleted. This results in multiple ip's assigned to the instances, with only one of them UP. This also results in attached floating ip's not working ..
This is similar to https://bugs.launchpad.net/nova/+bug/1609526, nova should tell neutron, either to delete the unused port, or update it instead of creating a new one.

Some more info on our environment:
----------------------------------
Using libvirt + kvm, neutron with openvswitch L3 HA

Our setup consists of multiple controllers and multiple hypervisors. Our shared storage for the instances is on a nfs 4.1 export.  Using Ubuntu 16.04 LTS and Openstack Mitaka

When we launch an instance, nova updates the mtime for the _base image to let the image cache manager know the image is actively used.  I think this was added here: https://review.openstack.org/gitweb?p=openstack/nova.git;a=commitdiff;h=fb6ca3e7c8a38328d384cd41c061ded6623dac90
Because of this, in our setup, we are seeing unexpected input/output errors:

Stderr: u"/bin/touch: setting times of '/var/lib/nova/instances/_base/79e34519bacb47ad6f64c4baca4d33fd5c57d34d': Input/output error

A full trace can be found here: http://paste.openstack.org/show/570161/

This error particularly shows itself when launching multiple instances at once.

Also, because of this error, the instances are rescheduled.  The assigned neutron ports, however, are not deleted.  This results in multiple ip's assigned to the instances, with only one of them UP.  This also results in attached floating ip's not working ..
This is similar to https://bugs.launchpad.net/nova/+bug/1609526, nova should tell neutron, either to delete the unused port, or update it instead of creating a new one.

Some more info on our environment:
----------------------------------
Using libvirt + kvm, neutron with openvswitch L3 HA

# dpkg -l | grep nova
ii  nova-common                        2:13.0.0-0ubuntu2                   all          OpenStack Compute - common files
ii  nova-compute                       2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node base
ii  nova-compute-kvm                   2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt               2:13.0.0-0ubuntu2                   all          OpenStack Compute - compute node libvirt support
ii  python-nova                        2:13.0.0-0ubuntu2                   all          OpenStack Compute Python libraries
ii  python-novaclient                  2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7