Cleaning up ovnmeta namespace fails, unexpected exception in notify_loop
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
High
|
wes hayutin |
Bug Description
{2} tempest.
wes hayutin (weshayutin) wrote : | #1 |
description: | updated |
wes hayutin (weshayutin) wrote : | #2 |
Noticed... there is an error when removing namespaces
2020-01-07 07:47:12.722 48153 INFO networking_
2020-01-07 07:47:13.602 48153 ERROR neutron.
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event [-] Unexpected exception in notify_loop: ProcessExecutio
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event Traceback (most recent call last):
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event File "/usr/lib/
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event match.run(event, row, updates)
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event File "/usr/lib/
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event self.agent.
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event File "/usr/lib/
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event self.teardown_
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event File "/usr/lib/
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event self._process_
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event File "/usr/lib/
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event pm.disable()
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event File "/usr/lib/
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event utils.execute(cmd, run_as_
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event File "/usr/lib/
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event returncode=
2020-01-07 07:47:13.603 48153 ERROR ovsdbapp.event ProcessExecutio
I don't see the same issue w/ http://
tags: |
added: promotion-blocker removed: alert |
wes hayutin (weshayutin) wrote : | #3 |
See a very similar trace here:
2020-01-06 13:28:09.712 28498 INFO networking_
2020-01-06 13:28:09.713 28498 DEBUG neutron.
2020-01-06 13:28:10.046 28498 DEBUG ovsdbapp.
2020-01-06 13:28:10.076 28498 DEBUG ovsdbapp.
summary: |
- master tempest fs020 timeout tracker + Cleaning up ovnmeta namespace fails, unexpected exception in notify_loop |
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master) | #4 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 4a1526f8d48d5da
Author: Wes Hayutin <email address hidden>
Date: Tue Jan 7 10:10:43 2020 -0700
update master skip list, nova issues and timeout
We'll need to update the bugs on this, but
for now we need to get fs020 running w/o
timing out. clearly there is an issue w/ nova
Related-Bug: #1858662
Change-Id: Ia30a3c07bbffc6
Terry Wilson (otherwiseguy) wrote : | #5 |
I notice in http://
==== (id: (4)====
### IPv4 addresses
Cannot open network namespace "(id:": No such file or directory
### IPv4 routing
Cannot open network namespace "(id:": No such file or directory
### IPTables (IPv4)
Cannot open network namespace "(id:": No such file or directory
==== (id: (6)====
### IPv6 addresses
Cannot open network namespace "(id:": No such file or directory
### IPv6 routing
Cannot open network namespace "(id:": No such file or directory
### IPTables (IPv6)
Cannot open network namespace "(id:": No such file or directory
Cannot open network namespace: No such file or directory
==== 13) (4)====
### IPv4 addresses
Cannot open network namespace "13)": No such file or directory
### IPv4 routing
Cannot open network namespace "13)": No such file or directory
### IPTables (IPv4)
Cannot open network namespace "13)": No such file or directory
==== 13) (6)====
### IPv6 addresses
Cannot open network namespace "13)": No such file or directory
### IPv6 routing
Cannot open network namespace "13)": No such file or directory
### IPTables (IPv6)
Cannot open network namespace "13)": No such file or directory
Cannot open network namespace: No such file or directory
And given the ip netns output:
[root@overcloud
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
ovnmeta-
It looks like something is parsing the output of 'ip netns' incorrectly. I don't remember seeing the (id: nn) previously, is this something that has changed recently?
Changed in tripleo: | |
milestone: | ussuri-1 → ussuri-2 |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-ansible (master) | #6 |
Related fix proposed to branch: master
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (master) | #7 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: master
commit 17d97f2618e56be
Author: Brent Eagles <email address hidden>
Date: Mon Jan 13 13:56:08 2020 -0330
Remove --rm=true from sidecar container sync
Neutron uses kill-scripts which remove the container after stopping it.
If the container is launched with docker and --rm=true, the container
will automatically be cleaned up and the $(CLI) rm <container id> in the
kill script with error out because the container can't be found.
Related-Bug: #1858662
Change-Id: I3d7940cb0816ad
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master) | #8 |
Related fix proposed to branch: master
Review: https:/
wes hayutin (weshayutin) wrote : | #9 |
working from 1861296 now.
Changed in tripleo: | |
status: | Triaged → Fix Released |
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-ansible (stable/train) | #10 |
Related fix proposed to branch: stable/train
Review: https:/
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart-extras (master) | #11 |
Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: master
Review: https:/
Reason: Got removed with this revert https:/
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (stable/train) | #12 |
Reviewed: https:/
Committed: https:/
Submitter: Zuul
Branch: stable/train
commit 60342041a3eb2c0
Author: Alex Schultz <email address hidden>
Date: Thu Nov 7 16:06:54 2019 -0700
[TRAIN] Backport tripleo-
This is a combination of 4 commits.
This is the 1st commit message:
Implement tripleo-
This patch adds a new role that will be used to manage side containers
with systemd instead of docker.socket or nsenter. The main use case here
is Neutron, although this role is designed to work with any service.
This role will create a series of systemd files to monitor a file which
gets mounted into a container. Additionally a wrapper script is
generated which is mounted in the container that will provide the
arguments that should be used to launch new containers.
Blueprint: safe-side-
Change-Id: I4821b7ca0260e4
Co-Authored-By: Dan Prince <email address hidden>
Co-Authored-By: Emilien Macchi <email address hidden>
Co-Authored-By: Alex Schultz <email address hidden>
(cherry picked from commit 699249f1790dd56
This is the commit message #2:
Remove --rm=true from sidecar container sync
Neutron uses kill-scripts which remove the container after stopping it.
If the container is launched with docker and --rm=true, the container
will automatically be cleaned up and the $(CLI) rm <container id> in the
kill script with error out because the container can't be found.
Related-Bug: #1858662
Change-Id: I3d7940cb0816ad
(cherry picked from commit 17d97f2618e56be
This is the commit message #3:
Fix substitution in kill-script
In the kill-script there is a string "Unknown action ${SIG} for
${$CT_NAME} ${CT_ID}" which results in a "bad substitution" error, as
there is no variable named with what the contents of the CT_NAME
environment variable contains. Remove the extraneous '$'.
Change-Id: I4c76071083bf5c
Fixes-Bug: #1860155
(cherry picked from commit b45d4c6d219e8e2
This is the commit message #4:
Add handling of signal 15 in kill script
The reason bug #1860155 was triggered was because the kill script did
not have a stanza for handling the signal that was passed in, which is
signal 15. Since signal 15 is unhandled, keepalived processes will
still stick around. Add handling for signal 15.
Change-Id: I632a3ef5ec137d
Related-bug: #1860155
(cherry picked from commit 06dc258a28784db
tags: | added: in-stable-train |
{2} tempest. api.compute. admin.test_ migrations. MigrationsAdmin Test.test_ list_migrations _in_flavor_ resize_ situation [306.507839s] ... FAILED
{2} tempest. api.compute. admin.test_ migrations. MigrationsAdmin Test.test_ resize_ server_ revert_ deleted_ flavor [306.310031s] ... FAILED
{1} setUpClass (tempest. api.compute. servers. test_create_ server. ServersTestBoot FromVolume) [0.000000s] ... FAILED
tempest. api.compute. admin.test_ migrations. MigrationsAdmin Test.test_ revert_ cold_migration [306.110744s] ... FAILED
{1} tempest. api.compute. servers. test_delete_ server. DeleteServersTe stJSON. test_delete_ server_ while_in_ attached_ volume [605.951803s] ... FAILED
{2} tempest. api.compute. admin.test_ volumes_ negative. VolumesAdminNeg ativeTest. test_update_ attached_ volume_ with_nonexisten t_volume_ in_body [602.707313s] ... FAILED
{1} tempest. api.compute. servers. test_delete_ server. DeleteServersTe stJSON. test_delete_ server_ while_in_ verify_ resize_ state [305.738424s] ... FAILED
{2} tearDownClass (tempest. api.compute. admin.test_ volumes_ negative. VolumesAdminNeg ativeTest) [0.000000s] ... FAILED
{0} tempest. api.compute. volumes. test_attach_ volume. AttachVolumeTes tJSON.test_ attach_ detach_ volume [465.673337s] ... FAILED
{1} tempest. api.compute. servers. test_server_ rescue_ negative. ServerRescueNeg ativeTestJSON. test_rescued_ vm_detach_ volume [509.927364s] ... FAILED api.compute. servers. test_device_ tagging. TaggedBootDevic esTest. test_tagged_ boot_devices [629.952744s] ... FAILED
{2} tempest.
{0} tempest. api.compute. volumes. test_attach_ volume. AttachVolumeTes tJSON.test_ list_get_ volume_ attachments [919.333764s] ... FAILED
{2} tearDownClass (tempest. api.compute. servers. test_device_ tagging. TaggedBootDevic esTest) [0.000000s] ... FAILED
{0} tearDownClass (tempest. api.compute. volumes. test_attach_ volume. AttachVolumeTes tJSON) [0.000000s] ... FAILED
{2} tempest. api.compute. servers. test_device_ tagging. TaggedBootDevic esTest_ v242.test_ tagged_ boot_devices [510.463829s] ... FAILED
{2} tearDownClass (tempest. api.compute. servers. test_device_ tagging. TaggedBootDevic esTest_ v242) [0.000000s] ... FAILED
{2} tempest. api.compute. servers. test_disk_ config. ServerDiskConfi gTestJSON. test_resize_ server_ from_auto_ to_manual [306.839021s] ... FAILED
{2} tempest. api.compute. servers. test_disk_ config. ServerDiskConfi gTestJSON. test_resize_ server_ from_manual_ to_auto [306.628151s] ... FAILED
{0} tempest. scenario. test_minimum_ basic.TestMinim umBasicScenario .test_minimum_ basic_scenario [623.687822s] ... FAILED
{2} tempest. api.compute. servers. test_server_ actions. ServerActionsTe stJSON. test_rebuild_ server_ with_volume_ attached [393.756537s] ... FAILED
{2} tempest. api.compute. servers. test_server_ actions. ServerActionsTe stJSON. test_resize_ server_ confirm [311.078329s] ... FAILED
{2} tempest. api.compute. servers. test_server_ actions. ServerActionsTe stJSON. test_resize_ server_ confirm_ from_stopped [669.624568s] ... FAILED
{2} tempest. api.compute. servers. test_server_ actions. ServerActionsTe stJSON. test_resize_ server_ revert [314.802816s] ... FAILED
{0} tempest. scenario. test_shelve_ instance. TestShelveInsta nce.test_ shelve_ volume_ backed_ instance [408.551256s] ... FAILED
{2} tempest. api.compute. servers. test_server_ actions. ServerAc. ..