[acceptance 9.2] "cinder_ceph_for_images_ephemeral" failed

[Dmitry Belyaninov]

Detailed bug description:
There is failed test case:
https://product-ci.infra.mirantis.net/view/9.x_acceptance/job/9.x.acceptance.ubuntu.ha_vlan_group_2/18/console

<<<<<##############################[ cinder_ceph_for_images_ephemeral ]##############################>>>>>
Deployment with 3 controllers, NeutronVLAN, with Ceph for
           images and ephemeral

        Scenario:
            1. Create new environment
            2. Choose Neutron, VLAN
            3. Choose Ceph for images and ceph for ephemeral
            4. Add 3 controller
            5. Add 2 compute
            6. Add 1 cinder
            7. Add 3 ceph
            8. Untag management and storage networks and move them to separate
               interfaces
            9. Verify networks
            10. Deploy the environment
            11. Verify networks
            12. Run OSTF tests

AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[openstack-network-networks/2] Stopping the deployment process!'

neutron-openswitch-agent
2016-10-21 06:37:46 ERR neutron.agent.linux.ovsdb_monitor [req-bfce47ae-b413-47c5-98fe-fcab3b7e75ee - - - - -] Interface monitor is not active
2016-10-21 06:37:45 ERR neutron.agent.ovsdb.impl_vsctl [req-bfce47ae-b413-47c5-98fe-fcab3b7e75ee - - - - -] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=type', 'list', 'Interface', 'int-br-floating']. Exception: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: no row "int-br-floating" in table Interface
2016-10-21 06:37:45 ERR neutron.agent.ovsdb.impl_vsctl [req-bfce47ae-b413-47c5-98fe-fcab3b7e75ee - - - - -] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=type', 'list', 'Interface', 'int-br-prv']. Exception: Exit code: 1; Stdin: ; Stdout: ; Stderr: ovs-vsctl: no row "int-br-prv" in table Interface

puppet
2016-10-21 06:39:10 ERR Could not prefetch neutron_subnet provider 'neutron': Can't retrieve subnet-list because Neutron or Keystone API is not available.
2016-10-21 06:38:59 ERR (/Stage[main]/Openstack_tasks::Openstack_network::Networks/Neutron_network[admin_internal_net]/ensure) change from absent to present failed: Not managing Neutron_network[admin_internal_net] due to earlier Neutron API failures.

Steps to reproduce:
run the test
Expected results:
pass
Actual result:
fail
Reproducibility:
yes
Workaround:
-
Impact:
 <put your information here>
Description of the environment:
 Operation system: <put your information here>
 Versions of components: <put your information here>
 Reference architecture: <put your information here>
 Network model: <put your information here>
 Related projects installed: <put your information here>
Additional information:
 <put your information here>

[Dmitry Belyaninov (dbelyaninov)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.install_plugin_after_cluster_create/132/testReport/(root)/three_ctrl_install_enable_after_create/three_ctrl_install_enable_after_create/

[Oleksiy Molchanov (omolchanov)]

Neutron team, can you check?

<163>Nov 15 04:36:48 node-1 neutron-server: 2016-11-15 04:36:48.618 32095 ERROR oslo_messaging._drivers.common [req-482e5ace-7fc8-45b0-82bd-ee6e3650e339 - - - - -] Returning exception (_mysql_exceptions.OperationalError) (1054, "Unknown column 'agents.resource_versions' in 'field list'") [SQL: u'SELECT agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents.`binary` AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.availability_zone AS agents_availability_zone, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.started_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations, agents.resource_versions AS agents_resource_versions, agents.`load` AS agents_load \nFROM agents \nWHERE agents.agent_type = %s AND agents.host = %s'] [parameters: ('Open vSwitch agent', 'node-1.test.domain.local')] to caller

[Ann Taraday (akamyshnikova)]

I've checked logs of https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.install_plugin_after_cluster_create/132/testReport/(root)/three_ctrl_install_enable_after_create/three_ctrl_install_enable_after_create/

In puppet logs on the node-1 I find the following http://paste.openstack.org/show/590197/
This means that on this environment migrations were applied before and now upgrade started from the middle from migration that was partially applied.

[Ann Taraday (akamyshnikova)]

Can be duplicate of bug https://bugs.launchpad.net/fuel/+bug/1636611