Kernel does not provide mount namespace: No such file or directory

[Amol Kahat]

Tempest test :

test_minimum_basic_instance_hard_reboot_after_vol_snap_deletion[compute,id-a8fd48ec-1d01-4895-b932-02321661ec1e,image,network,volume]

Error:

2022-09-20 12:17:32,587 255226 INFO [tempest.lib.common.rest_client] Request (TestMinimumBasicScenario:_run_cleanups): 204 DELETE http://10.0.0.5:9292/v2/images/bdae6c43-895b-4b0b-91cc-e9ff71e50e16 0.522s
2022-09-20 12:17:32,587 255226 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'content-type': 'text/html; charset=UTF-8', 'x-openstack-request-id': 'req-4c4d0438-ffeb-4b19-8f8d-53aa2bd2b95f', 'date': 'Tue, 20 Sep 2022 12:17:32 GMT', 'connection': 'close', 'status': '204', 'content-location': 'http://10.0.0.5:9292/v2/images/bdae6c43-895b-4b0b-91cc-e9ff71e50e16'}
        Body: b''
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_minimum_basic.py", line 240, in test_minimum_basic_instance_hard_reboot_after_vol_snap_deletion
    self.assertIsNotNone(
  File "/usr/lib/python3.9/site-packages/testtools/testcase.py", line 415, in assertIsNotNone
    self.assertThat(observed, matcher, message)
  File "/usr/lib/python3.9/site-packages/testtools/testcase.py", line 478, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: None matches Is(None): Failed to find floating IP '10.0.0.188' in server addresses: {'tempest-TestMinimumBasicScenario-996788801-network': [{'version': 4, 'addr': '10.100.0.11', 'OS-EXT-IPS:type': 'fixed', 'OS-EXT-IPS-MAC:mac_addr': 'fa:16:3e:a3:3b:b6'}]}

[Amol Kahat (amolkahat)]

Logs:
- https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/c63333a/logs/undercloud/var/log/tempest/stestr_results.html.gz
- https://logserver.rdoproject.org/65/31165/101/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/45261b3/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Amol Kahat (amolkahat)]

For me looks like nova side issue.

Logs: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/c63333a/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz

2022-09-20 12:21:34.031 2 INFO nova.compute.manager [req-5c8e71bf-342f-4f55-9066-8108f8fd360b 1654cd25fd534c72b62bee52fe788f1f b664e6aea9a24928a8adec720d72bbe5 - default default] [instance: e0e488b2-c944-4925-a297-da196cd129d3] Get console output
2022-09-20 12:21:34.035 173 INFO nova.privsep.libvirt [-] Ignored error while reading from instance console pty: can't concat NoneType to bytes

[Amol Kahat (amolkahat)]

Periodic jobs reported Kernel does not provide mount namespace: No such file or directory.

2022-09-20 12:17:06.037+0000: 21959: error : virProcessRunInFork:1361 : internal error: child reported (status=125): Kernel does not provide mount namespace: No such file or directory
2022-09-20 12:17:06.037+0000: 21959: error : virProcessRunInFork:1365 : Kernel does not provide mount namespace: No such file or directory
2022-09-20 12:17:06.040+0000: 21959: error : virProcessRunInFork:1361 : internal error: child reported (status=125): Kernel does not provide mount namespace: No such file or directory
2022-09-20 12:17:06.040+0000: 21959: error : virProcessRunInFork:1365 : Kernel does not provide mount namespace: No such file or directory
2022-09-20 12:17:06.040+0000: 21959: warning : qemuBlockRemoveImageMetadata:2969 : Unable to remove disk metadata on vm instance-00000002 from /var/lib/nova/instances/c68b6c9f-0a8d-4558-bdad-a5b762021e46/disk (disk target vda)
2022-09-20 12:17:06.043+0000: 21959: error : virProcessRunInFork:1361 : internal error: child reported (status=125): Kernel does not provide mount namespace: No such file or directory
2022-09-20 12:17:06.043+0000: 21959: error : virProcessRunInFork:1365 : Kernel does not provide mount namespace: No such file or directory

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/c63333a/logs/overcloud-novacompute-0/var/log/containers/libvirt/virtqemud.log.1.gz

https://logserver.rdoproject.org/65/31165/101/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/45261b3/logs/overcloud-novacompute-1/var/log/containers/libvirt/virtqemud.log.1.gz

In upstream gate it is something different

2022-09-21 11:43:59.833+0000: 123138: error : virCommandWait:2752 : internal error: Child process (ovs-vsctl --timeout=5 set Interface tap97fb6f4f-59 ingress_policing_rate=0 ingress_policing_burst=0) unexpected exit status 1: ovs-vsctl: no row "tap97fb6f4f-59" in table Interface

2022-09-21 11:43:59.833+0000: 123138: debug : virCommandRun:2479 : Result status 0, stdout: '' stderr: 'ovs-vsctl: no row "tap97fb6f4f-59" in table Interface
'
2022-09-21 11:43:59.833+0000: 123138: error : virNetDevOpenvswitchInterfaceClearRxQos:916 : internal error: Unable to reset ingress on port tap97fb6f4f-59

Tap device not found.

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c97/857476/3/gate/tripleo-ci-centos-8-standalone/c97136a/logs/undercloud/var/log/containers/libvirt/virtqemud.log.1

[Amol Kahat (amolkahat)]

Latest run:

- https://logserver.rdoproject.org/69/41469/21/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/cffd32e/logs/overcloud-novacompute-0/var/log/containers/libvirt/virtqemud.log.1.gz

2022-09-21 14:13:30.608+0000: 21665: debug : virCgroupRemoveRecursively:2594 : Removing cgroup /sys/fs/cgroup/machine/qemu-8-instance-0000000d.libvirt-qemu//emulator
2022-09-21 14:13:30.608+0000: 21665: debug : virCgroupRemoveRecursively:2594 : Removing cgroup /sys/fs/cgroup/machine/qemu-8-instance-0000000d.libvirt-qemu/
2022-09-21 14:13:30.611+0000: 21665: error : virProcessRunInFork:1361 : internal error: child reported (status=125): Kernel does not provide mount namespace: No such file or directory
2022-09-21 14:13:30.611+0000: 21665: error : virProcessRunInFork:1365 : Kernel does not provide mount namespace: No such file or directory
2022-09-21 14:13:30.614+0000: 21665: error : virProcessRunInFork:1361 : internal error: child reported (status=125): Kernel does not provide mount namespace: No such file or directory
2022-09-21 14:13:30.614+0000: 21665: error : virProcessRunInFork:1365 : Kernel does not provide mount namespace: No such file or directory
2022-09-21 14:13:30.614+0000: 21665: warning : qemuBlockRemoveImageMetadata:2969 : Unable to remove disk metadata on vm instance-0000000d from /var/lib/nova/instances/292071d0-af22-40ae-8c28-1d2628607a41/disk (disk target vda)
2022-09-21 14:13:30.616+0000: 21665: error : virProcessRunInFork:1361 : internal error: child reported (status=125): Kernel does not provide mount namespace: No such file or directory
2022-09-21 14:13:30.616+0000: 21665: error : virProcessRunInFork:1365 : Kernel does not provide mount namespace: No such file or directory
2022-09-21 14:13:30.618+0000: 21665: error : virProcessRunInFork:1361 : internal error: child reported (status=125): Kernel does not provide mount namespace: No such file or directory
2022-09-21 14:13:30.618+0000: 21665: error : virProcessRunInFork:1365 : Kernel does not provide mount namespace: No such file or directory
2022-09-21 14:13:30.618+0000: 21665: warning : qemuBlockRemoveImageMetadata:2969 : Unable to remove disk metadata on vm instance-0000000d from /var/lib/nova/instances/_base/234ffff2aa5bd43417ae6a2edae6538b34a42adc (disk target vda)
2022-09-21 14:13:30.619+0000: 21665: debug : qemuDomainObjEndJob:1204 : Stopping job: destroy (async=none vm=0x7fb51000d2f0 name=instance-0000000d)

[Ananya Banerjee (frenzyfriday)]

I see the string "Kernel does not provide mount namespace: No such file or directory" in passing logs [1] as well. the root cause of the tempest failures might be different.

[1] https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/b8c4c5d/logs/overcloud-novacompute-0/var/log/containers/libvirt/virtqemud.log.1.gz

[Soniya Murlidhar Vyas (svyas)]

So we have 2 green runs and few retries so doesnt look like too bad. Moving this to fix_released, please move back if you see this issue again.

[1] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby&skip=0