All CI jobs failing on OVB stack creation

[Gabriele Cerami]

today's job are failing on undercloud install with this error

2016-10-11 06:11:26.000 | [2016/10/11 06:11:26 AM] [INFO] Using config file at: /etc/os-net-config/config.json
2016-10-11 06:11:26.000 | [2016/10/11 06:11:26 AM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
2016-10-11 06:11:26.000 | [2016/10/11 06:11:26 AM] [INFO] Ifcfg net config provider created.
2016-10-11 06:11:26.000 | [2016/10/11 06:11:26 AM] [INFO] nic1 mapped to: eth0
2016-10-11 06:11:26.000 | [2016/10/11 06:11:26 AM] [INFO] adding bridge: br-ctlplane
2016-10-11 06:11:26.000 | [2016/10/11 06:11:26 AM] [ERROR] Unable to read mac address: eth1
2016-10-11 06:11:26.000 | Traceback (most recent call last):
2016-10-11 06:11:26.000 | File "/usr/bin/os-net-config", line 10, in <module>
2016-10-11 06:11:26.000 | sys.exit(main())
2016-10-11 06:11:26.000 | File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 185, in main
2016-10-11 06:11:26.000 | provider.add_object(obj)
2016-10-11 06:11:26.000 | File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 57, in add_object
2016-10-11 06:11:26.000 | self.add_bridge(obj)
2016-10-11 06:11:26.000 | File "/usr/lib/python2.7/site-packages/os_net_config/impl_ifcfg.py", line 399, in add_bridge
2016-10-11 06:11:26.000 | data = self._add_common(bridge)
2016-10-11 06:11:26.000 | File "/usr/lib/python2.7/site-packages/os_net_config/impl_ifcfg.py", line 155, in _add_common
2016-10-11 06:11:26.000 | mac = utils.interface_mac(base_opt.primary_interface_name)
2016-10-11 06:11:26.000 | File "/usr/lib/python2.7/site-packages/os_net_config/utils.py", line 81, in interface_mac
2016-10-11 06:11:26.000 | with open('/sys/class/net/%s/address' % name, 'r') as f:
2016-10-11 06:11:26.000 | IOError: [Errno 2] No such file or directory: '/sys/class/net/eth1/address'

this is caused by OVB failing to create stack correctly.
testenv client returns

testenv-client - INFO - Received job : Failed creating OVB stack

[Derek Higgins (derekh)]

The error is occurring while creating the testenv heat stack

2016-10-11 09:59:12.890 33652 ERROR heat.engine.resource Forbidden: (rule:create_network and rule:create_network:shared) on {'router:external': False, 'availability_zone_hints': [], 'description': '', 'provider:physical_network': <object object at 0x7f487d4f0140>, u'admin_state_up': True, 'tenant_id': u'b79291658f384b7ebbc9019b6349e5c9', 'segments': <object object at 0x7f487d4f0140>, 'dns_domain': '', 'provider:network_type': <object object at 0x7f487d4f0140>, 'is_default': False, 'qos_policy_id': None, 'port_security_enabled': True, u'shared': True, 'provider:segmentation_id': <object object at 0x7f487d4f0140>, u'name': u'public-10372'} by {'domain': None, 'project_name': u'openstack-nodepool', 'tenant_name': u'openstack-nodepool', 'project_domain': None, 'timestamp': '2016-10-11 09:59:12.856425', 'auth_token': 'c6316e0559684f2a8f0f7ea48fb889d9', 'resource_uuid': None, 'is_admin': False, 'user': u'ba119eef29ce49f5b8697f4d63948e3c', 'tenant': u'b79291658f384b7ebbc9019b6349e5c9', 'read_only': False, 'project_id': u'b79291658f384b7ebbc9019b6349e5c9', 'user_id': u'ba119eef29ce49f5b8697f4d63948e3c', 'show_deleted': False, 'roles': [u'_member_'], 'user_identity': 'ba119eef29ce49f5b8697f4d63948e3c b79291658f384b7ebbc9019b6349e5c9 - - -', 'tenant_id': u'b79291658f384b7ebbc9019b6349e5c9', 'request_id': 'req-3fda0674-6a57-4aed-8068-5c03bcc6e32c', 'user_domain': None, 'user_name': u'openstack-nodepool'} disallowed by policy

[Gabriele Cerami (gcerami)]

Proposed https://review.openstack.org/384906 to catch the failure and stop the builds earlier.

[Derek Higgins (derekh)]

I believe what may have happened is that the cron job that refreshes the code on our te-broker was disabled. Somebody re-enabled it last night.

As a result this commit from nearly 2 months ago started to take effect
https://github.com/cybertron/openstack-virtual-baremetal/commit/dfd3a00d0ad8ac46646aef0b1d83112f5bf8cbe5

The neutron policy on the rh1 overcloud doesn't allow users to create shared networks so the stack can't be created.

I've edited the policy and restarted neutron-server
$ diff policy.json_ policy.json
48c48
< "create_network:shared": "rule:admin_only",
---
> "create_network:shared": "rule:admin_or_owner",

[Gabriele Cerami (gcerami)]

The fix worked. OVB started creating stacks correctly again