[9.x][updates] an env update has failed on the "openstack-network-routers" task

[Ruslan Khozinov]

Env description:
 - compute
 - cinder+controller
 - elasticsearch_kibana+influxdb_grafana

Detailed bug description:
Deployment has failed. All nodes are finished. Failed tasks: Task[openstack-network-routers/7] Stopping the deployment process!

ERROR neutron.db.l3_agentschedulers_db [req-5c1118f0-9e98-48ca-98a2-3f38e89f4ec1 - - - - -] Failed to reschedule router 19179bdc-09ac-49a2-84fc-879a287f236f
ERROR neutron.db.l3_agentschedulers_db Traceback (most recent call last):
ERROR neutron.db.l3_agentschedulers_db File "/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line 136, in reschedule_routers_from_down_agents
ERROR neutron.db.l3_agentschedulers_db self.reschedule_router(context, binding.router_id)
ERROR neutron.db.l3_agentschedulers_db File "/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line 310, in reschedule_router
ERROR neutron.db.l3_agentschedulers_db router_id=router_id)
ERROR neutron.db.l3_agentschedulers_db RouterReschedulingFailed: Failed rescheduling router 19179bdc-09ac-49a2-84fc-879a287f236f: no eligible l3 agent found.
ERROR neutron.db.l3_agentschedulers_db

Steps to reproduce:
- Get Fuel 9.1
- Configure
 - compute
 - cinder+controller
 - elasticsearch_kibana+influxdb_grafana (stacklight plugin from the oficial mirantis catalog)
- Set the next proposed repo for fuel:
  [mos9.2]
  name=mos9.2
  #snapshot681
  baseurl=http://mirror.seed-cz1.fuel-infra.org/mos-repos/centos/mos9.0-centos7/snapshots/proposed-2016-12-24-130321/x86_64
  enabled=1
  gpgcheck=1
  gpgkey=file:///etc/pki/fuel-gpg/RPM-GPG-KEY-mos
  skip_if_unavailable=1
- Run the playbooks/mos9_prepare_fuel.yml playbook
- Run the playbooks/mos9_prepare_env.yml -e '{"env_id":2, "snapshot_repo":'9.0-2016-12-24-130321'}' -vv | tee prepare_env.log
- Run the playbooks/update_fuel.yml -e '{"rebuild_bootstrap":false}' -vv | tee update_fuel.log
- Run fuel2 update --env 2 install --repos mos9.2 --restart-rabbit --restart-mysql

[Ruslan Khozinov (rkhozinov)]

Diagnostic snapshot: https://drive.google.com/file/d/0B9nJDnRluay9OVVkT0FkVjBuMkE/view?usp=sharing

[Ruslan Khozinov (rkhozinov)]

systemd-shim has already been installed during 9.0->9.1

[Sergii Rizvan (srizvan)]

There was a lot of errors in neutron logs like this:

2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 686, in _report_state
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent True)
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 87, in report_state
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent return method(context, 'report_state', **kwargs)
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent retry=self.retry)
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 91, in _send
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent timeout=timeout, retry=retry)
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent retry=retry)
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 503, in _send
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent raise result
2016-12-26 08:26:15.827 2080 ERROR neutron.agent.l3.agent RemoteError: Remote error: DBConnectionError (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0") [SQL: u'SELECT 1']

For some reason MySQL server was restarted when "openstack-network-routers" task had been executing. Seems that caused the issue.

After that bug haven't been reproduced.

[Sergii Rizvan (srizvan)]

Status set to Invalid because bug wasn't reproduced anymore. If bug will be reproduced again, feel free to reopen it.