So the user_id in this case doesn't matter. I'm wondering if the actual project id used is the admin tenant? I don't see 3292f7b35abf4565a53d099ad878a335 in tempest.conf though so it's probably not the admin tenant:
So yeah, we have an admin role and we're using a per-project resource for quotas (fixed_ips), so we need to find out what the quota limit is per project in a tempest run.
I don't see the quota driver set in nova.conf so it defaults to the DB quota driver:
But what are the limits? Because the defaults are 10. I don't see anything in devstack that changes quotas unless the virt driver is the fake driver, which isn't the case here.
Taking a look at the code in this stacktrace:
2014-10-06 08:00:23.848 DEBUG nova.network. manager [req-bf60b9d9- 452c-4fbc- 9f7a-cc90ed7625 36 ServersAdminTes tJSON-122085994 3 ServersAdminTes tJSON-233825707 ] [instance: 50156a3a- 8b14-40ea- affe-97e5a510ec 32] Allocate fixed ip on network cbcee1f9- 13d7-4ec6- a450-d777a3d111 20 allocate_fixed_ip /opt/stack/ new/nova/ nova/network/ manager. py:859 manager [req-bf60b9d9- 452c-4fbc- 9f7a-cc90ed7625 36 ServersAdminTes tJSON-122085994 3 ServersAdminTes tJSON-233825707 ] Quota exceeded for 3292f7b35abf456 5a53d099ad878a3 35, tried to allocate fixed IP allocate_fixed_ip /opt/stack/ new/nova/ nova/network/ manager. py:875 rpc.dispatcher [req-bf60b9d9- 452c-4fbc- 9f7a-cc90ed7625 36 ServersAdminTes tJSON-122085994 3 ServersAdminTes tJSON-233825707 ] Exception during message handling: Maximum number of fixed ips exceeded rpc.dispatcher Traceback (most recent call last): rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 134, in _dispatch_and_reply rpc.dispatcher incoming.message)) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 177, in _dispatch rpc.dispatcher return self._do_ dispatch( endpoint, method, ctxt, args) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 123, in _do_dispatch rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) rpc.dispatcher File "/opt/stack/ new/nova/ nova/network/ floating_ ips.py" , line 114, in allocate_ for_instance rpc.dispatcher **kwargs) rpc.dispatcher File "/opt/stack/ new/nova/ nova/network/ manager. py", line 511, in allocate_ for_instance rpc.dispatcher requested_ networks= requested_ networks) rpc.dispatcher File "/opt/stack/ new/nova/ nova/network/ manager. py", line 192, in _allocate_fixed_ips rpc.dispatcher vpn=vpn, address=address) rpc.dispatcher File "/opt/stack/ new/nova/ nova/network/ manager. py", line 876, in allocate_fixed_ip rpc.dispatcher raise exception. FixedIpLimitExc eeded() rpc.dispatcher FixedIpLimitExc eeded: Maximum number of fixed ips exceeded rpc.dispatcher
2014-10-06 08:00:23.874 DEBUG nova.network.
2014-10-06 08:00:23.874 ERROR oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
2014-10-06 08:00:23.874 25609 TRACE oslo.messaging.
We're using the FlatDHCPManager:
http:// logs.openstack. org/98/ 125398/ 11/check/ check-tempest- dsvm-postgres- full/f0d9495/ logs/etc/ nova/nova. conf.txt. gz
So we're using DB quotas (VlanManager uses the no-op quota driver).
We could be logging the wrong project_id:
LOG.debug("Quota exceeded for %s, tried to allocate "
"fixed IP", context.project_id)
fixed_ips is a per-project quota in the DB API:
PER_PROJECT_QUOTAS = ['fixed_ips', 'floating_ips', 'networks']
So the user_id in this case doesn't matter. I'm wondering if the actual project id used is the admin tenant? I don't see 3292f7b35abf456 5a53d099ad878a3 35 in tempest.conf though so it's probably not the admin tenant:
http:// logs.openstack. org/98/ 125398/ 11/check/ check-tempest- dsvm-postgres- full/f0d9495/ logs/tempest_ conf.txt. gz
From talking with Matt Treinish, it sounds like this test creates a tenant-isolated user/project with an admin role:
http:// git.openstack. org/cgit/ openstack/ tempest/ tree/tempest/ api/compute/ admin/test_ servers. py#n29
So yeah, we have an admin role and we're using a per-project resource for quotas (fixed_ips), so we need to find out what the quota limit is per project in a tempest run.
I don't see the quota driver set in nova.conf so it defaults to the DB quota driver:
http:// logs.openstack. org/98/ 125398/ 11/check/ check-tempest- dsvm-postgres- full/f0d9495/ logs/etc/ nova/nova. conf.txt. gz
But what are the limits? Because the defaults are 10. I don't see anything in devstack that changes quotas unless the virt driver is the fake driver, which isn't the case here.