"FixedIpLimitExceeded: Maximum number of fixed ips exceeded" in tempest nova-network runs since 7/4

[Matt Riedemann]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQnVpbGRBYm9ydEV4Y2VwdGlvbjogQnVpbGQgb2YgaW5zdGFuY2VcIiBBTkQgbWVzc2FnZTpcImFib3J0ZWQ6IEZhaWxlZCB0byBhbGxvY2F0ZSB0aGUgbmV0d29yayhzKSB3aXRoIGVycm9yIE1heGltdW0gbnVtYmVyIG9mIGZpeGVkIGlwcyBleGNlZWRlZCwgbm90IHJlc2NoZWR1bGluZy5cIiBBTkQgdGFnczpcInNjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQwNDc3OTE1MzY1MiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

Saw it here:

http://logs.openstack.org/63/98563/5/check/check-tempest-dsvm-postgres-full/1472e7b/logs/screen-n-cpu.txt.gz?level=TRACE

Looks like it's only in jobs using nova-network.

Started on 7/4, 70 failures in 7 days, check and gate, multiple changes.

Maybe related to https://review.openstack.org/104581.

[Matt Riedemann (mriedem)]

e-r query: https://review.openstack.org/#/c/105312/

[Vish Ishaya (vishvananda)]

I have seen this error a couple of times and it always seems to hit in one of the variations of tempest.api.compute.admin.test_fixed_ips.FixedIPsTest(Json|Xml). Is it possible there is a bug in the test causing us to run out of quota?

[Vish Ishaya (vishvananda)]

Also the fixed_ip quota limit defaults to -1 which should be unlimited which also leads me to suspect a testing error. It wouldn't show up for neutron because if neutron is running then it manages quota internally.

[Joe Gordon (jogo)]

As vish said, It looks like this may be a tempest issue. Perhaps its related to https://review.openstack.org/#/c/100342/ which was merged around the same time this bug cropped up.

tempest.api.compute.admin.test_quotas.QuotaClassesAdminTestJSON.test_update_default_quotas was run right before

setUpClass (tempest.api.compute.admin.test_fixed_ips.FixedIPsTestJson) failed

[Joe Gordon (jogo)]

The offending code is line 183 in https://review.openstack.org/#/c/100342/6/tempest/api/compute/admin/test_quotas.py

Since the default value is -1, adding one sets the quota to 0.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/105651

[Matt Riedemann (mriedem)]

This started up again after the tempest fix, 26 hits in 10 days so something else must be causing quota issues now.

[Ihar Hrachyshka (ihar-hrachyshka)]

Still failed in periodic stable/juno job: http://logs.openstack.org/periodic-stable/periodic-tempest-dsvm-postgres-full-juno/0d88af7/logs/testr_results.html.gz

[Yaroslav Lobankov (ylobankov)]

Guys, is this issue still reproduced? if yes, could anyone please provide some logs or something else? Thanks.

[Bradley Klein (brad-klein)]

Yaroslav,

I haven't seen it in a very long time -- you can close it as far as I'm concerned...

Thanks!
Brad

[Yaroslav Lobankov (ylobankov)]

Brad, thank you for the response! It looks like this bug hasn't been seen by anyone a long time. Moving the bug status back to "Fix released". Feel free to reopen the bug and set the status back to ''Confirmed'' if you encounter the issue.