7.0 MU1
Swarm test: https://product-ci.infra.mirantis.net/view/7.0_swarm/job/7.0.system_test.ubuntu.cic_maintenance_mode/103/testReport/junit/(root)/auto_cic_maintenance_mode/auto_cic_maintenance_mode/
1. Revert snapshot 3 ['controller', 'mongo'] +2 ['compute', 'cinder']
2. reboot --force controller
3. Wait until controller is switching in maintenance mode
4. Exit maintenance mode
5. Check the controller become available
6. Run OSTF and repeat for other 2 nodes.
For the 3rd node OSTF smoke finally didn't pass
{
"Create volume and boot instance from it (failure)": "Failed to get to expected status. In error state. Please refer to OpenStack logs for more details."
},
{
"Create volume and attach it to instance (failure)": "Failed to get to expected status. In error state. Please refer to OpenStack logs for more details."
},
{
"Check network connectivity from instance via floating IP (failure)": "Failed to get to expected status. In error state. Please refer to OpenStack logs for more details."
},
{
"Launch instance (failure)": "Failed to get to expected status. In error state. Please refer to OpenStack logs for more details."
},
{
"Launch instance with file injection (failure)": "Failed to get to expected status. In error state. Please refer to OpenStack logs for more details."
},
{
"Launch instance, create snapshot, launch instance from snapshot (failure)": "Failed to get to expected status. In error state. Please refer to OpenStack logs for more details."
}
]
Errors in nova-compute "NovaException: Unexpected vif_type=binding_failed" :
https://paste.mirantis.net/show/1405/
Open vSwitch agents are marked as dead in neutron agent-list the same as others:
root@node-5:~# neutron agent-list
+--------------------------------------+--------------------+--------------------------+-------+----------------+---------------------------+
| id | agent_type | host | alive | admin_state_up | binary |
+--------------------------------------+--------------------+--------------------------+-------+----------------+---------------------------+
| 122b0a93-0b84-42d2-adf1-0aed651d4344 | L3 agent | node-5.test.domain.local | xxx | True | neutron-l3-agent |
| 3b1e0177-3458-4a1e-b1aa-c32df12a9cd3 | DHCP agent | node-5.test.domain.local | xxx | True | neutron-dhcp-agent |
| 49b5dac6-75eb-492d-9a01-d39679ac6a75 | Metadata agent | node-5.test.domain.local | xxx | True | neutron-metadata-agent |
| 4b9259b6-87f8-423c-bfa3-ad984c4c4937 | L3 agent | node-4.test.domain.local | xxx | True | neutron-l3-agent |
| 577098e7-8565-4456-b2e8-1feeb133f983 | DHCP agent | node-4.test.domain.local | xxx | True | neutron-dhcp-agent |
| 70b2841c-1c31-4ef8-ab8a-35d63883c072 | L3 agent | node-1.test.domain.local | :-) | True | neutron-l3-agent |
| 992c731e-2f37-4498-be62-9b969b0aa4b9 | Open vSwitch agent | node-1.test.domain.local | :-) | True | neutron-openvswitch-agent |
| 99eabcd2-7192-4c0c-a717-62a49b4ff0f3 | Metadata agent | node-4.test.domain.local | xxx | True | neutron-metadata-agent |
| a066f867-1f44-4524-925c-3de92df5cc3f | DHCP agent | node-1.test.domain.local | :-) | True | neutron-dhcp-agent |
| c6028891-c029-47cd-9081-5d7644717d56 | Open vSwitch agent | node-5.test.domain.local | :-) | True | neutron-openvswitch-agent |
| ccc0bcb2-f1de-48bb-bd71-82bac0a519dc | Open vSwitch agent | node-3.test.domain.local | xxx | True | neutron-openvswitch-agent |
| d84f6b13-9b4a-4969-9219-29aeb13cf091 | Open vSwitch agent | node-2.test.domain.local | xxx | True | neutron-openvswitch-agent |
| f923b54d-fe16-43b2-9d71-9c0179f3e5b5 | Open vSwitch agent | node-4.test.domain.local | xxx | True | neutron-openvswitch-agent |
| fbd47cae-5990-4af4-863d-9c565cc029af | Metadata agent | node-1.test.domain.local | :-) | True | neutron-metadata-agent |
+--------------------------------------+--------------------+--------------------------+-------+----------------+---------------------------+
Workaround: Kill neutron-openvswitch-agent on each compute
Also we can see crush report in rabbit logs during the test execution(https:/
2015-11-10 12:06:21,821 - INFO fuel_web_
"Check if amount of tables in databases is the same on each node": "success",
"RabbitMQ replication": "success",
"Check pacemaker status": "success",
"RabbitMQ availability": "success",
"Check galera environment state": "success",
"Check data replication over mysql": "success"
}
and then start ['smoke', 'sanity']
In reverted snapshot I've seen that rabbitmq was not stopped