test_external_network_visibility intermittent failure

Bug #1553595 reported by Armando Migliaccio
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Kevin Benton
Kilo
Won't Fix
Undecided
Unassigned

Bug Description

Very odd failure:

http://logs.openstack.org/79/288279/3/gate/gate-neutron-dsvm-api/300ee95/testr_results.html.gz

ft33.1: neutron.tests.api.test_networks.NetworksIpV6TestJSON.test_external_network_visibility[id-af774677-42a9-4e4b-bb58-16fe6a5bc1ec,smoke]_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{
2016-03-05 17:07:49,598 11488 INFO [tempest.lib.common.rest_client] Request (NetworksIpV6TestJSON:test_external_network_visibility): 200 GET http://127.0.0.1:9696/v2.0/networks?router%3Aexternal=True 0.232s
2016-03-05 17:07:49,599 11488 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'X-Auth-Token': '<omitted>', 'Accept': 'application/json'}
        Body: None
    Response - Headers: {'status': '200', 'content-location': 'http://127.0.0.1:9696/v2.0/networks?router%3Aexternal=True', 'content-type': 'application/json; charset=UTF-8', 'connection': 'close', 'x-openstack-request-id': 'req-7c15efb9-e07d-47de-8f49-e77dc2059f57', 'content-length': '1199', 'date': 'Sat, 05 Mar 2016 17:07:49 GMT'}
        Body: {"networks": [{"status": "ACTIVE", "router:external": true, "availability_zone_hints": [], "availability_zones": ["nova"], "qos_policy_id": null, "subnets": ["1ee8f3fc-1957-46c6-8a7c-6a5335342871", "068121cc-6ed9-4bdb-8813-35fe689642c2"], "shared": false, "tenant_id": "e118b21bf7a74b36a7e1339918290567", "created_at": "2016-03-05T16:53:27", "tags": [], "ipv6_address_scope": null, "updated_at": "2016-03-05T16:53:27", "is_default": true, "admin_state_up": true, "ipv4_address_scope": null, "port_security_enabled": true, "mtu": 1450, "id": "85a04141-b614-406d-b7d8-912c2a37bc4b", "name": "public"}, {"status": "ACTIVE", "router:external": true, "availability_zone_hints": [], "availability_zones": ["nova"], "qos_policy_id": null, "subnets": ["d3ea9b6d-a20e-48c0-b7ec-50f6239c5199"], "shared": true, "tenant_id": "d6562d45e82f4a85a30dc0cec714e04d", "created_at": "2016-03-05T17:07:31", "tags": [], "ipv6_address_scope": null, "updated_at": "2016-03-05T17:07:31", "is_default": false, "admin_state_up": true, "ipv4_address_scope": "978d5509-cfa9-4753-9ff3-6bb11fdb6f57", "port_security_enabled": true, "mtu": 1450, "id": "a005c6f8-1438-42aa-a86c-68d04796d2e9", "name": "sharednetwork--1158192641"}]}
2016-03-05 17:07:49,947 11488 INFO [tempest.lib.common.rest_client] Request (NetworksIpV6TestJSON:test_external_network_visibility): 200 GET http://127.0.0.1:9696/v2.0/subnets 0.348s
2016-03-05 17:07:49,948 11488 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'X-Auth-Token': '<omitted>', 'Accept': 'application/json'}
        Body: None
    Response - Headers: {'status': '200', 'content-location': 'http://127.0.0.1:9696/v2.0/subnets', 'content-type': 'application/json; charset=UTF-8', 'connection': 'close', 'x-openstack-request-id': 'req-7a2b800a-eeb9-4c4d-92c8-1e2bf556fb89', 'content-length': '1641', 'date': 'Sat, 05 Mar 2016 17:07:49 GMT'}
        Body: {"subnets": [{"name": "", "enable_dhcp": true, "network_id": "a005c6f8-1438-42aa-a86c-68d04796d2e9", "tenant_id": "d6562d45e82f4a85a30dc0cec714e04d", "created_at": "2016-03-05T17:07:35", "dns_nameservers": [], "updated_at": "2016-03-05T17:07:35", "gateway_ip": "8.0.0.1", "ipv6_ra_mode": null, "allocation_pools": [{"start": "8.0.0.2", "end": "8.0.0.14"}], "host_routes": [], "ip_version": 4, "ipv6_address_mode": null, "cidr": "8.0.0.0/28", "id": "d3ea9b6d-a20e-48c0-b7ec-50f6239c5199", "subnetpool_id": "b5058565-3ce7-448c-a581-3411f1aa764b"}, {"name": "tempest-BaseTestCase-467862474-subnet", "enable_dhcp": true, "network_id": "00d56d28-c7c2-4059-b3f5-146e60110b67", "tenant_id": "f3ef1b7cfa324fb29d4ea00646a1bb61", "created_at": "2016-03-05T17:07:37", "dns_nameservers": [], "updated_at": "2016-03-05T17:07:37", "gateway_ip": "10.100.0.1", "ipv6_ra_mode": null, "allocation_pools": [{"start": "10.100.0.2", "end": "10.100.0.14"}], "host_routes": [], "ip_version": 4, "ipv6_address_mode": null, "cidr": "10.100.0.0/28", "id": "08548e7c-5e95-4371-8694-1d4ceba7c2e1", "subnetpool_id": null}, {"name": "", "enable_dhcp": true, "network_id": "d527821a-86b1-4bcc-be1f-7231c8640a60", "tenant_id": "f3ef1b7cfa324fb29d4ea00646a1bb61", "created_at": "2016-03-05T17:07:47", "dns_nameservers": [], "updated_at": "2016-03-05T17:07:47", "gateway_ip": "2003:0:0:ffff::1", "ipv6_ra_mode": null, "allocation_pools": [{"start": "2003:0:0:ffff::2", "end": "2003::ffff:ffff:ffff:ffff:ffff"}], "host_routes": [], "ip_version": 6, "ipv6_address_mode": null, "cidr": "2003:0:0:ffff::/64", "id": "2ae2b2e2-5af1-4044-a962-1cb3eb7781d0", "subnetpool_id": null}]}
}}}

Traceback (most recent call last):
  File "neutron/tests/api/test_networks.py", line 425, in test_external_network_visibility
    self.assertEmpty(subnets, "Public subnets visible")
  File "/tmp/openstack/tempest/tempest/test.py", line 636, in assertEmpty
    self.assertTrue(len(list) == 0, msg)
  File "/opt/stack/new/neutron/.tox/api-constraints/local/lib/python2.7/site-packages/unittest2/case.py", line 702, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : Public subnets visible

tags: added: gate-failure
Changed in neutron:
importance: Undecided → Critical
status: New → Confirmed
milestone: none → mitaka-rc1
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

I get a feeling this might have something to do with BGP interfering with the external network state.

@Carl/Ryan: any chance you can provide an insight as to why this might happen?

Changed in neutron:
assignee: nobody → Carl Baldwin (carl-baldwin)
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/288957

Changed in neutron:
assignee: Carl Baldwin (carl-baldwin) → Kevin Benton (kevinbenton)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/288957
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=391c2327c3b9de0e2b9875bab8d6f6909fa0983a
Submitter: Jenkins
Branch: master

commit 391c2327c3b9de0e2b9875bab8d6f6909fa0983a
Author: Kevin Benton <email address hidden>
Date: Sat Mar 5 17:49:48 2016 -0800

    Fix API test for external subnet visibility

    The test_external_network_visibility test incorrectly
    assumes that subnets will never be visible on external
    networks. However, this is not true in the case that a
    network is both external and shared.

    This updates the test to ignore shared networks.

    Change-Id: I0219e17d2ec70939039344826d35cb533b3ce065
    Closes-Bug: #1553595

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/289166

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kevin Benton (<email address hidden>) on branch: master
Review: https://review.openstack.org/289166
Reason: Armando is mean :)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/289166

Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/neutron 8.0.0.0rc1

This issue was fixed in the openstack/neutron 8.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/293776

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/291787
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cc863e07156e3cfe020c5acfcd9875daff477de6
Submitter: Jenkins
Branch: master

commit cc863e07156e3cfe020c5acfcd9875daff477de6
Author: Armando Migliaccio <email address hidden>
Date: Fri Mar 11 08:00:34 2016 -0800

    Remove test_external_network_visibility

    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.

    [1] https://review.openstack.org/#/c/280427/

    Related-bug: 1553595
    Related-bug: 1552960

    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    Depends-on: I0219e17d2ec70939039344826d35cb533b3ce065

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/mitaka)

Reviewed: https://review.openstack.org/293776
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e9088d09328793495aba260a05d6161f92aae4cd
Submitter: Jenkins
Branch: stable/mitaka

commit e9088d09328793495aba260a05d6161f92aae4cd
Author: Armando Migliaccio <email address hidden>
Date: Fri Mar 11 08:00:34 2016 -0800

    Remove test_external_network_visibility

    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.

    [1] https://review.openstack.org/#/c/280427/

    Related-bug: 1553595
    Related-bug: 1552960

    (cherry-picked from commit cc863e07156e3cfe020c5acfcd9875daff477de6)
    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    Depends-on: I0219e17d2ec70939039344826d35cb533b3ce065

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/296527

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/296528

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/liberty)

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/296986

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/kilo)

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/296987

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/liberty)

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/296527
Reason: Proposed removal backport: https://review.openstack.org/296986

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/kilo)

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/296528
Reason: Proposed removal backport: https://review.openstack.org/296987

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/296987
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=af4492359b50383c95b5eac5041c08d8c7e92275
Submitter: Jenkins
Branch: stable/kilo

commit af4492359b50383c95b5eac5041c08d8c7e92275
Author: Armando Migliaccio <email address hidden>
Date: Fri Mar 11 08:00:34 2016 -0800

    Remove test_external_network_visibility

    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.

    [1] https://review.openstack.org/#/c/280427/

    Related-bug: 1553595
    Related-bug: 1552960

    Conflicts:
     neutron/tests/api/test_networks.py

    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    (cherry picked from commit cc863e07156e3cfe020c5acfcd9875daff477de6)
    (cherry picked from commit ffd82f9ea4e3436eb750803eb7b15770d020f913)

tags: added: in-stable-kilo
tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/296986
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e17f391ab75fa9338f3e7a2c6fa8e49ac43036c0
Submitter: Jenkins
Branch: stable/liberty

commit e17f391ab75fa9338f3e7a2c6fa8e49ac43036c0
Author: Armando Migliaccio <email address hidden>
Date: Fri Mar 11 08:00:34 2016 -0800

    Remove test_external_network_visibility

    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.

    [1] https://review.openstack.org/#/c/280427/

    Related-bug: 1553595
    Related-bug: 1552960

    Conflicts:
     neutron/tests/api/test_networks.py

    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    (cherry picked from commit cc863e07156e3cfe020c5acfcd9875daff477de6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/302570

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)
Download full text (11.4 KiB)

Reviewed: https://review.openstack.org/302570
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=03ccdd6a6d8e4cea9145e39dd7a1614656024aed
Submitter: Jenkins
Branch: master

commit 3213eb124e40b130e174ac3a91067e2b196788dd
Author: Cedric Brandily <email address hidden>
Date: Tue Mar 29 18:06:44 2016 +0200

    Support Routes==2.3

    SubMapper.connect allows in Routes<=2.2 to define routes with/without a
    name:

     map.connect(route_name, route_path)
     map.connect(route_path)

    Routes==2.3 SubMapper.connect now requires a route name (but not
    Mapper.connect). This change adds a route name when needed to support
    Routes<=2.2 and Routes==2.3.

    Closes-Bug: #1563028
    Change-Id: I655bfe7f047945896d63dd94c584acfe60d81b38
    (cherry picked from commit c6efa6292cac9267428366dc3cee00f6fc73e208)

commit 4283a7e36b74823f229400f0f049742f630db28b
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Mar 30 11:48:37 2016 +0200

    Constraint requirements using mitaka upper-constraints.txt file

    Change-Id: I6a908470987ce05769c425c4623df404cef76a10

commit fc690974480611242d4fc6e1deaaf68171fcaed3
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Mar 30 06:21:15 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I78c24969927224277f381532ad2604cfde8741fc

commit 41be555eddb0f9947fdaa4e73fa74a72677d4d11
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Mar 29 06:07:19 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: Iafa43bfcfca4d6af0d0866206c07fa817f567de6

commit b435ec56af98d2c04de37dd779ed3613ec4583ad
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Mar 28 06:07:02 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I9e908579b4704c4294367113523b898839586426

commit bec65f6cdd159ec252c4aa0a63aabc9aea8fcfba
Author: YAMAMOTO Takashi <email address hidden>
Date: Fri Mar 25 15:25:39 2016 +0900

    api tests: Check correct extensions

    auto_allocate -> auto-allocated-topology
    rbac_policies -> rbac-policies

    Closes-Bug: #1561858

    (cherry picked from commit 9d3bab6b8d6b2122732faeb66f831927bee0ac7e)
    Change-Id: Ied13e52bf78afe571f7243e93005bf5c68fe44db

commit f0c3b19165a5ca73ea8f1dc0ddaad290eee35039
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Mar 25 06:06:36 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: Idb22eac32944a088d24f4d44e537d9bba09dbcc2

commit 99915fa76bd4ce13e0d36059616ba357f4f54ee2
Author: Hynek Mlnarik <email address hidden>
Date: Thu Mar...

Revision history for this message
Mohammed Naser (mnaser) wrote :

I'm referencing this bug here. It seems this Tempest test has a bug.

https://review.openstack.org/#/c/288959/
https://github.com/openstack/tempest/commit/6919da4be56850d923345dd870da8ecc57484a72

We run Tempest against our public cloud to make sure everything is running smoothly, our `public_network_id` is an external, shared network. Therefore, as per the Neutron policy, a user *will* be able to retrieve the subnets for that network because it is both shared and external.

I would be more than happy to contribute a fix, but would love guidance on how the Neutron team wants to take this.

Revision history for this message
Kevin Benton (kevinbenton) wrote :

@Mohammed,

I think you will need to propose a change to tempest to stop making that assertion. There isn't anything we can do on the Neutron side.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.