neutron

Stable newton gate is broken

Bug #1680136 reported by Dariusz Smigiel on 2017-04-05

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Critical	Kevin Benton

Bug Description

Stable newton branch is busted.
https://review.openstack.org/#/c/445326/

Py27 and Py35 tests are passing for checks [1] but failing for gate [2]

[1] http://logs.openstack.org/26/445326/2/check/gate-networking-midonet-python27-ubuntu-xenial/57c3b39/testr_results.html.gz
[2] http://logs.openstack.org/26/445326/2/gate/gate-networking-midonet-python27-ubuntu-xenial/5ce12db/testr_results.html.gz

ft21.80: midonet.neutron.tests.unit.test_midonet_plugin.TestMidonetL3NatDBIntTest.test_floatingip_list_with_pagination_reverse_StringException: Empty attachments:
stderr
stdout

pythonlogging:'': {{{
WARNING [stevedore.named] Could not load midonet.neutron.plugin_v1.MidonetPluginV2
WARNING [neutron.api.extensions] Extension address-scope not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension allowed-address-pairs not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension auto-allocated-topology not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension availability_zone not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension default-subnetpools not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension dns-integration not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension dvr not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension extraroute not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension flavors not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension ip_allocation not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension l2_adjacency not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension l3-ha not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension l3-flavors not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension l3_agent_scheduler not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension metering not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension multi-provider not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension net-mtu not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension network_availability_zone not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension network-ip-availability not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension port-security not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension provider not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension qos not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension standard-attr-revisions not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension router_availability_zone not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension router-service-type not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension segment not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension service-type not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension subnet-service-types not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension tag not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension standard-attr-timestamp not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension trunk not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension trunk-details not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension vlan-transparent not supported by any of loaded plugins
   ERROR [neutron.api.extensions] Extension path 'neutron/tests/unit/extensions' doesn't exist!
WARNING [neutron.api.extensions] Extension agent-membership not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension bgp-speaker-router-insertion not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension gateway-device not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension logging-resource not supported by any of loaded plugins
WARNING [neutron.api.extensions] Extension router-interface-fip not supported by any of loaded plugins
WARNING [neutron.scheduler.dhcp_agent_scheduler] No more DHCP agents
WARNING [neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api] Unable to schedule network 810e84a4-e990-4172-be50-7c004a2e18ab: no agents available; will retry on subsequent port and subnet creation events.
WARNING [neutron.scheduler.dhcp_agent_scheduler] No more DHCP agents
WARNING [neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api] Unable to schedule network 1f549549-0bdc-4198-96c6-75360a92edf2: no agents available; will retry on subsequent port and subnet creation events.
WARNING [neutron.scheduler.dhcp_agent_scheduler] No more DHCP agents
WARNING [neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api] Unable to schedule network 90dd6f25-96ae-48d0-ae24-e911b1fd20dd: no agents available; will retry on subsequent port and subnet creation events.
   ERROR [neutron.api.v2.resource] index failed: No details.
Traceback (most recent call last):
  File "/tmp/openstack/neutron/neutron/api/v2/resource.py", line 79, in resource
    result = method(request=request, **args)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 88, in wrapped
    setattr(e, '_RETRY_EXCEEDED', True)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 84, in wrapped
    return f(*args, **kwargs)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
    ectxt.value = e.inner_exc
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
    return f(*args, **kwargs)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 124, in wrapped
    traceback.format_exc())
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 119, in wrapped
    return f(*dup_args, **dup_kwargs)
  File "/tmp/openstack/neutron/neutron/api/v2/base.py", line 361, in index
    return self._items(request, True, parent_id)
  File "/tmp/openstack/neutron/neutron/api/v2/base.py", line 300, in _items
    obj_list = obj_getter(request.context, **kwargs)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 159, in wrapped
    return method(*args, **kwargs)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 88, in wrapped
    setattr(e, '_RETRY_EXCEEDED', True)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 84, in wrapped
    return f(*args, **kwargs)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper
    ectxt.value = e.inner_exc
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper
    return f(*args, **kwargs)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 124, in wrapped
    traceback.format_exc())
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/tmp/openstack/neutron/neutron/db/api.py", line 119, in wrapped
    return f(*dup_args, **dup_kwargs)
  File "/tmp/openstack/neutron/neutron/db/l3_db.py", line 1424, in get_floatingips
    page_reverse=page_reverse)
  File "/tmp/openstack/neutron/neutron/db/common_db_mixin.py", line 329, in _get_collection
    page_reverse=page_reverse)
  File "/tmp/openstack/neutron/neutron/db/common_db_mixin.py", line 303, in _get_collection_query
    sort_dirs=sort_dirs)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/utils.py", line 234, in paginate_query
    crit_attrs.append((model_attr > marker_values[i]))
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/operators.py", line 318, in __gt__
    return self.operate(gt, other)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 175, in operate
    return op(self.comparator, *other, **kwargs)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/operators.py", line 318, in __gt__
    return self.operate(gt, other)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/orm/properties.py", line 269, in operate
    return op(self.__clause_element__(), *other, **kwargs)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/operators.py", line 318, in __gt__
    return self.operate(gt, other)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 742, in operate
    return op(self.comparator, *other, **kwargs)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/operators.py", line 318, in __gt__
    return self.operate(gt, other)
  File "<string>", line 1, in <lambda>
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/type_api.py", line 60, in operate
    return o[0](self.expr, op, *(other + o[1:]), **kwargs)
  File "/home/jenkins/workspace/gate-networking-midonet-python27-ubuntu-xenial/.tox/py27/local/lib/python2.7/site-packages/sqlalchemy/sql/default_comparator.py", line 53, in _boolean_compare
    "Only '=', '!=', 'is_()', 'isnot()' operators can "
ArgumentError: Only '=', '!=', 'is_()', 'isnot()' operators can be used with None/True/False
}}}

Traceback (most recent call last):
  File "/tmp/openstack/neutron/neutron/tests/unit/extensions/test_l3.py", line 2711, in test_floatingip_list_with_pagination_reverse
    ('floating_ip_address', 'asc'), 2, 2)
  File "/tmp/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 749, in _test_list_with_pagination_reverse
    self.assertThat(len(res[resources]),
KeyError: 'floatingips'

Tags:

Revision history for this message

Dariusz Smigiel (smigiel-dariusz) wrote on 2017-04-05:

message:"KeyError: 'floatingips'" and 22 hits in last 7 days

tags:

added: gate-failure

Revision history for this message

Boden R (boden) wrote on 2017-04-05:

If this is by chance related to the facade changes in neutron, it's possible [1] could help in the adoption (once [1] is release with lib 1.4.0).

[1] https://review.openstack.org/#/c/453063/

Dariusz Smigiel (smigiel-dariusz) on 2017-04-05

affects:	networking-midonet → neutron
Changed in neutron:
status:	New → Confirmed
importance:	Undecided → Critical
assignee:	nobody → Kevin Benton (kevinbenton)

Revision history for this message

Dariusz Smigiel (smigiel-dariusz) wrote on 2017-04-05:

Attempt to fix this https://review.openstack.org/#/c/453740/

Ihar Hrachyshka (ihar-hrachyshka) on 2017-04-07

tags:

added: unittest

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-11: Fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/453740
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ed12ceaef185f8ba9560ac22382588fd04de2f62
Submitter: Jenkins
Branch: stable/ocata

commit ed12ceaef185f8ba9560ac22382588fd04de2f62
Author: Kevin Benton <email address hidden>
Date: Wed Apr 5 09:19:16 2017 -0700

Add work-around for older oslo.db

    The min version of oslo.db in Newton at the time of this
    commit is 4.10 and Ocata is 4.15, which do not include
    a commit [1] to skip None values in pagination.
    Rather than trying to get oslo.db globally bumped to
    4.17.0 on both stable branches, this just skips None
    in our code.

This is not proposed to master because it uses a new enough
oslo.db to not be impacted by this.

1. b3869d04cff7071c1226758eb8b58fde9eba5b8d

Closes-Bug: #1680136
Change-Id: Ieef10eebd93f99404dd2fd87ccbab9b75632945a

tags:

added: in-stable-ocata

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-11: Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/453741
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d5bde7fd50d4f2ad9b42f3c75113dc9ad5eaea3e
Submitter: Jenkins
Branch: stable/newton

commit d5bde7fd50d4f2ad9b42f3c75113dc9ad5eaea3e
Author: Kevin Benton <email address hidden>
Date: Wed Apr 5 09:19:16 2017 -0700

Add work-around for older oslo.db

This is not proposed to master because it uses a new enough
oslo.db to not be impacted by this.

1. b3869d04cff7071c1226758eb8b58fde9eba5b8d

Closes-Bug: #1680136
Change-Id: Ieef10eebd93f99404dd2fd87ccbab9b75632945a

tags:

added: in-stable-newton

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-12: Fix included in openstack/neutron 10.0.1

This issue was fixed in the openstack/neutron 10.0.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-04-12: Fix included in openstack/neutron 9.3.1

This issue was fixed in the openstack/neutron 9.3.1 release.

Kevin Benton (kevinbenton) on 2017-04-14

Changed in neutron:
status:	Confirmed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1680858

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.