All OVB jobs in master periodic pipeline fail to install the undercloud while trying to create the ctlplane network with 503 service unavailable on neutron

Bug #1818538 reported by Gabriele Cerami on 2019-03-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Gabriele Cerami

Bug Description

This is happening in all OVB job currently.

Example log at

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/7dc4975/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2019-03-04_11_41_46

Shows:

2019-03-04 11:41:46 | "stderr": [
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,844] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/script < /var/lib/heat-config/deployed/f0d889b9-7280-4073-ac70-315359b3541e.json",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,492] (heat-config) [INFO] |-",
2019-03-04 11:41:46 | " {\"deploy_stdout\": \"ERROR: Network create/update failed.\\n\", \"deploy_stderr\": \"Traceback (most recent call last):\\n File \\\"/var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e\\\", line 281, in <module>\\n network = _ensure_neutron_network(sdk)\\n File \\\"/var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e\\\", line 44, in _ensure_neutron_network\\n network = list(sdk.network.networks(name=CTLPLANE_NETWORK_NAME))\\n File \\\"/usr/lib/python2.7/site-packages/openstack/resource.py\\\", line 1319, in list\\n exceptions.raise_from_response(response)\\n File \\\"/usr/lib/python2.7/site-packages/openstack/exceptions.py\\\", line 229, in raise_from_response\\n http_status=http_status, request_id=request_id\\nopenstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://192.168.24.2:13696/v2.0/networks?name=ctlplane, No server is available to handle this request.: 503 Service Unavailable\\n\", \"deploy_status_code\": 1}",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,493] (heat-config) [DEBUG] [2019-03-04 11:41:41,879] (heat-config) [INFO] config={\"cloud_name\": \"undercloud\", \"enable_routed_networks\": true, \"home_dir\": \"/home/zuul\", \"local_ip\": \"192.168.24.3\", \"local_subnet\": \"ctlplane-subnet\", \"mtu\": 1350, \"nameservers\": [\"1.1.1.1\", \"1.0.0.1\", \"38.145.33.91\"], \"physical_network\": \"ctlplane\", \"subnets\": {\"ctlplane-subnet\": {\"AllocationPools\": [{\"end\": \"192.168.24.30\", \"start\": \"192.168.24.5\"}], \"DhcpRangeEnd\": \"192.168.24.24\", \"DhcpRangeStart\": \"192.168.24.5\", \"NetworkCidr\": \"192.168.24.0/24\", \"NetworkGateway\": \"192.168.24.1\"}}}",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_server_id=a051f8e5-d82c-4aa2-93f5-117fdc55908f",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_action=CREATE",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_stack_id=undercloud-AllNodesDeploySteps-r3ochb4weckr-UndercloudExtraConfigPost-kvq4indzkukw-UndercloudCtlplaneNetworkDeployment-mk7iorqyyayw-0-g33zaq5ybz25/735a1452-4c83-48e7-b921-e927ffcfa5bd",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_resource_name=TripleOSoftwareDeployment",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,879] (heat-config) [INFO] deploy_signal_transport=NO_SIGNAL",
2019-03-04 11:41:46 | "[2019-03-04 11:41:41,880] (heat-config) [DEBUG] Running /var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,486] (heat-config) [INFO] ERROR: Network create/update failed.",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,486] (heat-config) [DEBUG] Traceback (most recent call last):",
2019-03-04 11:41:46 | " File \"/var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e\", line 281, in <module>",
2019-03-04 11:41:46 | " network = _ensure_neutron_network(sdk)",
2019-03-04 11:41:46 | " File \"/var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e\", line 44, in _ensure_neutron_network",
2019-03-04 11:41:46 | " network = list(sdk.network.networks(name=CTLPLANE_NETWORK_NAME))",
2019-03-04 11:41:46 | " File \"/usr/lib/python2.7/site-packages/openstack/resource.py\", line 1319, in list",
2019-03-04 11:41:46 | " exceptions.raise_from_response(response)",
2019-03-04 11:41:46 | " File \"/usr/lib/python2.7/site-packages/openstack/exceptions.py\", line 229, in raise_from_response",
2019-03-04 11:41:46 | " http_status=http_status, request_id=request_id",
2019-03-04 11:41:46 | "openstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://192.168.24.2:13696/v2.0/networks?name=ctlplane, No server is available to handle this request.: 503 Service Unavailable",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,486] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-script/f0d889b9-7280-4073-ac70-315359b3541e. [1]",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,493] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/script",
2019-03-04 11:41:46 | "[2019-03-04 11:41:43,493] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/f0d889b9-7280-4073-ac70-315359b3541e.json < /var/lib/heat-config/deployed/f0d889b9-7280-4073-ac70-315359b3541e.notify.json",
2019-03-04 11:41:46 | "[2019-03-04 11:41:44,314] (heat-config) [INFO] ",
2019-03-04 11:41:46 | "[2019-03-04 11:41:44,315] (heat-config) [DEBUG] "

Digging into service logs, neutron server logs for example at

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/7dc4975/logs/undercloud/var/log/containers/neutron/server.log.txt.gz

is full of errors like this

2019-03-04 11:49:22.866 8 CRITICAL neutron [req-e11a4f6a-c966-4aed-9f02-7d48103f49b1 - - - - -] Unhandled error: TypeError: __init__() got an unexpected keyword argument 'set_proctitle'
2019-03-04 11:49:22.866 8 ERROR neutron Traceback (most recent call last):
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/bin/neutron-server", line 10, in <module>
2019-03-04 11:49:22.866 8 ERROR neutron sys.exit(main())
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main
2019-03-04 11:49:22.866 8 ERROR neutron server.boot_server(wsgi_eventlet.eventlet_wsgi_server)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 68, in boot_server
2019-03-04 11:49:22.866 8 ERROR neutron server_func()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 23, in eventlet_wsgi_server
2019-03-04 11:49:22.866 8 ERROR neutron neutron_api = service.serve_wsgi(service.NeutronApiService)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 91, in serve_wsgi
2019-03-04 11:49:22.866 8 ERROR neutron LOG.exception('Unrecoverable error: please check log '
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-03-04 11:49:22.866 8 ERROR neutron self.force_reraise()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-03-04 11:49:22.866 8 ERROR neutron six.reraise(self.type_, self.value, self.tb)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 88, in serve_wsgi
2019-03-04 11:49:22.866 8 ERROR neutron service.start()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 64, in start
2019-03-04 11:49:22.866 8 ERROR neutron self.wsgi_app = _run_wsgi(self.app_name)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/service.py", line 316, in _run_wsgi
2019-03-04 11:49:22.866 8 ERROR neutron app = config.load_paste_app(app_name)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 125, in load_paste_app
2019-03-04 11:49:22.866 8 ERROR neutron app = loader.load_app(app_name)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app
2019-03-04 11:49:22.866 8 ERROR neutron return deploy.loadapp("config:%s" % self.config_path, name=name)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2019-03-04 11:49:22.866 8 ERROR neutron return loadobj(APP, uri, name=name, **kw)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2019-03-04 11:49:22.866 8 ERROR neutron return context.create()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-03-04 11:49:22.866 8 ERROR neutron return self.object_type.invoke(self)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2019-03-04 11:49:22.866 8 ERROR neutron **context.local_conf)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 58, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron reraise(*exc_info)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/compat.py", line 23, in reraise
2019-03-04 11:49:22.866 8 ERROR neutron exec('raise t, e, tb', dict(t=t, e=e, tb=tb))
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron val = callable(*args, **kw)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2019-03-04 11:49:22.866 8 ERROR neutron app = loader.get_app(app_name, global_conf=global_conf)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2019-03-04 11:49:22.866 8 ERROR neutron name=name, global_conf=global_conf).create()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-03-04 11:49:22.866 8 ERROR neutron return self.object_type.invoke(self)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2019-03-04 11:49:22.866 8 ERROR neutron **context.local_conf)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 58, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron reraise(*exc_info)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/compat.py", line 23, in reraise
2019-03-04 11:49:22.866 8 ERROR neutron exec('raise t, e, tb', dict(t=t, e=e, tb=tb))
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron val = callable(*args, **kw)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 47, in pipeline_factory
2019-03-04 11:49:22.866 8 ERROR neutron app = loader.get_app(pipeline[-1])
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2019-03-04 11:49:22.866 8 ERROR neutron name=name, global_conf=global_conf).create()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2019-03-04 11:49:22.866 8 ERROR neutron return self.object_type.invoke(self)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2019-03-04 11:49:22.866 8 ERROR neutron return fix_call(context.object, context.global_conf, **context.local_conf)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 58, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron reraise(*exc_info)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/compat.py", line 23, in reraise
2019-03-04 11:49:22.866 8 ERROR neutron exec('raise t, e, tb', dict(t=t, e=e, tb=tb))
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2019-03-04 11:49:22.866 8 ERROR neutron val = callable(*args, **kw)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 25, in _factory
2019-03-04 11:49:22.866 8 ERROR neutron return pecan_app.v2_factory(global_config, **local_config)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/app.py", line 47, in v2_factory
2019-03-04 11:49:22.866 8 ERROR neutron startup.initialize_all()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/pecan_wsgi/startup.py", line 39, in initialize_all
2019-03-04 11:49:22.866 8 ERROR neutron manager.init()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 314, in init
2019-03-04 11:49:22.866 8 ERROR neutron NeutronManager.get_instance()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 265, in get_instance
2019-03-04 11:49:22.866 8 ERROR neutron cls._create_instance()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2019-03-04 11:49:22.866 8 ERROR neutron return f(*args, **kwargs)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 251, in _create_instance
2019-03-04 11:49:22.866 8 ERROR neutron cls._instance = cls()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 136, in __init__
2019-03-04 11:49:22.866 8 ERROR neutron plugin_provider)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 175, in _get_plugin_instance
2019-03-04 11:49:22.866 8 ERROR neutron plugin_inst = plugin_class()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 128, in wrapper
2019-03-04 11:49:22.866 8 ERROR neutron return f(*args, **kwargs)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 240, in __init__
2019-03-04 11:49:22.866 8 ERROR neutron self._setup_dhcp()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 264, in _setup_dhcp
2019-03-04 11:49:22.866 8 ERROR neutron self.add_periodic_dhcp_agent_status_check()
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 193, in add_periodic_dhcp_agent_status_check
2019-03-04 11:49:22.866 8 ERROR neutron self.remove_networks_from_down_agents
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/db/agentschedulers_db.py", line 94, in add_agent_status_check_worker
2019-03-04 11:49:22.866 8 ERROR neutron initial_delay)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/worker.py", line 35, in __init__
2019-03-04 11:49:22.866 8 ERROR neutron super(PeriodicWorker, self).__init__(worker_process_count=0)
2019-03-04 11:49:22.866 8 ERROR neutron File "/usr/lib/python2.7/site-packages/neutron/worker.py", line 24, in __init__
2019-03-04 11:49:22.866 8 ERROR neutron set_proctitle=set_proctitle
2019-03-04 11:49:22.866 8 ERROR neutron TypeError: __init__() got an unexpected keyword argument 'set_proctitle'
2019-03-04 11:49:22.866 8 ERROR neutron

Gabriele Cerami (gcerami) wrote :

Looks like this has been caused by this change

https://review.openstack.org/637019

not sure how this started to affect TripleO though

summary: All OVB jobs in master periodic pipeline fail to install the undercloud
- while trying to create the ctlplane networ with 503 service unavailable
+ while trying to create the ctlplane network with 503 service unavailable
+ on neutron
Alex Schultz (alex-schultz) wrote :

We need neutron-lib 1.25.0. It added https://review.openstack.org/#/c/637119/ which was required for https://review.openstack.org/#/c/637019/

Gabriele Cerami (gcerami) wrote :

https://review.rdoproject.org/r/19100 has been proposed and merged to bump neutron-lib release

Changed in tripleo:
status: Triaged → Fix Committed
Gabriele Cerami (gcerami) wrote :

Jobs are now passing. No additional fixes required.

Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers