Juniper Openstack

R3.0- HA-nova services are down after upgrade from build 39 to 40

Bug #1583842 reported by shajuvk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Invalid
Low
shajuvk
R3.0
Invalid
Low
shajuvk
Juniper Openstack r3.0.2.0

Bug Description

After upgrading from R3.0.3 build 39 t0 40 some of the nova services on one of the node shows down. SQL db is in sync on all the nodes. Upgrade task was executed in parallel

Please check the node a6s32 (1st cfgm) for the testbed details

----+------------------+-------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+-------+----------+---------+-------+----------------------------+-----------------+
| 3 | nova-scheduler | a6s32 | internal | enabled | up | 2016-05-19T23:22:02.000000 | - |
| 6 | nova-console | a6s32 | internal | enabled | up | 2016-05-19T23:22:15.000000 | - |
| 9 | nova-consoleauth | a6s32 | internal | enabled | up | 2016-05-19T23:22:02.000000 | - |
| 12 | nova-conductor | a6s32 | internal | enabled | up | 2016-05-19T23:22:17.000000 | - |
| 15 | nova-scheduler | a6s33 | internal | enabled | up | 2016-05-19T23:22:10.000000 | - |
| 18 | nova-console | a6s33 | internal | enabled | up | 2016-05-19T23:22:10.000000 | - |
| 21 | nova-consoleauth | a6s33 | internal | enabled | up | 2016-05-19T23:22:10.000000 | - |
| 24 | nova-conductor | a6s33 | internal | enabled | up | 2016-05-19T23:22:09.000000 | - |
| 30 | nova-scheduler | a6s38 | internal | enabled | down | 2016-05-19T22:09:48.000000 | - |
| 33 | nova-console | a6s38 | internal | enabled | down | 2016-05-19T22:09:51.000000 | - |
| 36 | nova-consoleauth | a6s38 | internal | enabled | down | 2016-05-19T22:09:50.000000 | - |
| 39 | nova-conductor | a6s38 | internal | enabled | down | 2016-05-19T22:09:48.000000 | - |
| 42 | nova-compute | a6s2 | nova | enabled | up | 2016-05-19T23:22:16.000000 | - |
| 45 | nova-compute | a6s38 | nova | enabled | up | 2016-05-19T23:22:17.000000 | - |
+----+------------------+-------+----------+---------+-------+----------------------------+--

mysql> show status like '%wsrep%';
+----------------------------+----------------------------------------------------+
| Variable_name | Value |
+----------------------------+----------------------------------------------------+
| wsrep_local_state_uuid | 7c6a7822-1d95-11e6-93f5-a6102df31469 |
| wsrep_protocol_version | 4 |
| wsrep_last_committed | 92403 |
| wsrep_replicated | 82509 |
| wsrep_replicated_bytes | 88401231 |
| wsrep_received | 8545 |
| wsrep_received_bytes | 43584369 |
| wsrep_local_commits | 81898 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.000000 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 116.117949 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_cert_index_size | 267 |
| wsrep_causal_reads | 1663330 |
| wsrep_incoming_addresses | 10.84.13.38:3306,10.84.13.33:3306,10.84.13.32:3306 |
| wsrep_cluster_conf_id | 19 |
| wsrep_cluster_size | 3 |
| wsrep_cluster_state_uuid | 7c6a7822-1d95-11e6-93f5-a6102df31469 |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_index | 2 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <email address hidden> |
| wsrep_provider_version | 25.2.9(r172) |
| wsrep_ready | ON |
+----------------------------+----------------------------------------------------+
40 rows in set (0.00 sec)

See original description
Tags: ha sanity
shajuvk (shajuvk)
summary: - R3.0- HA-nova services are down afterupgrade from build 39 to 40
+ R3.0- HA-nova services are down after upgrade from build 39 to 40
tags: added: sanity
shajuvk (shajuvk)
Changed in juniperopenstack:
importance: Undecided → Critical
Jeba Paulaiyan (jebap)
information type: Proprietary → Public
tags: added: blocker
Revision history for this message
shajuvk (shajuvk) wrote :
Download full text (6.4 KiB)

Nova scheduler log:
===
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py", line 224, in run_periodic_tasks
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task task(self, context)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 70, in _expire_reservations
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task QUOTAS.expire(context)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 1444, in expire
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task self._driver.expire(context)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 690, in expire
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task db.reservation_expire(context)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 1174, in reservation_expire
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task return IMPL.reservation_expire(context)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 215, in wrapper
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task return f(*args, **kwargs)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 271, in wrapped
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task return f(*args, **kwargs)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 3742, in reservation_expire
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task reservation_query.soft_delete(synchronize_session=False)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/session.py", line 600, in soft_delete
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task synchronize_session=synchronize_session)
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2796, in update
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task update_op.exec_()
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 897, in exec_
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task self._do_exec()
2016-05-19 21:44:30.507 31340 TRACE nova.openstack.common.periodic_task File "/usr/lib/pyt...

Read more...

description: updated
Revision history for this message
shajuvk (shajuvk) wrote :

logs are copied to : /auto/cores/shajuvk/1583842

Jeba Paulaiyan (jebap)
Changed in juniperopenstack:
importance: Critical → High
Sachin Bansal (sbansal)
Changed in juniperopenstack:
assignee: nobody → Rudra Rugge (rudrarugge)
Revision history for this message
Ashish Ranjan (aranjan-n) wrote :

Not a blocker.

tags: removed: blocker
Revision history for this message
shajuvk (shajuvk) wrote :

Comments from Rudra:
======

Looks like /etc/nova/nova.conf is not in sync. This is missing on 38.

root@a6s33:~# cd /etc/nova
root@a6s33:/etc/nova# grep -R mysql *
nova.conf:connection = mysql://nova:c0ntrail123@10.84.13.201:33306/nova
root@a6s33:/etc/nova#

Rudra Rugge (rrugge)
Changed in juniperopenstack:
assignee: Rudra Rugge (rudrarugge) → Nagendra Prasath (npchandran)
Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

Verified upgrade in same setup from build 3.0.2.0-44 to 3.0.2.0-45, didn't hit the issue.

Changed in juniperopenstack:
importance: High → Low
status: New → Incomplete
assignee: Nagendra Prasath (npchandran) → shajuvk (shajuvk)
Jeba Paulaiyan (jebap)
Changed in juniperopenstack:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.