Ubuntu
neutron package

SQL error during alembic.migration when populating Neutron database on MariaDB 10.0

Bug #1384555 reported by Robert Campbell on 2014-10-23

This bug affects 3 people

	Status	Importance	Assigned to	Milestone
neutron	Fix Released	Medium	Jakub Libosvar	neutron 2015.1.0 "kilo"
Juno	Fix Released	Medium	Jakub Libosvar	neutron 2014.2.1
neutron (Ubuntu)	Fix Released	Medium	Unassigned

Bug Description

On a fresh installation of Juno, it seems that that the database is not being populated correctly on a fresh install. This is the output of the log (I also demonstrated the DB had no tables to begin with):

MariaDB [(none)]> use neutron
Database changed
MariaDB [neutron]> show tables;
Empty set (0.00 sec)

MariaDB [neutron]> quit
Bye
root@vm-1:~# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini current
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
Current revision for mysql://neutron:XXXXX@10.10.10.1/neutron: None
root@vm-1:~# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
INFO [alembic.migration] Running upgrade None -> havana, havana_initial
INFO [alembic.migration] Running upgrade havana -> e197124d4b9, add unique constraint to members
INFO [alembic.migration] Running upgrade e197124d4b9 -> 1fcfc149aca4, Add a unique constraint on (agent_type, host) columns to prevent a race condition when an agent entry is 'upserted'.
INFO [alembic.migration] Running upgrade 1fcfc149aca4 -> 50e86cb2637a, nsx_mappings
INFO [alembic.migration] Running upgrade 50e86cb2637a -> 1421183d533f, NSX DHCP/metadata support
INFO [alembic.migration] Running upgrade 1421183d533f -> 3d3cb89d84ee, nsx_switch_mappings
INFO [alembic.migration] Running upgrade 3d3cb89d84ee -> 4ca36cfc898c, nsx_router_mappings
INFO [alembic.migration] Running upgrade 4ca36cfc898c -> 27cc183af192, ml2_vnic_type
INFO [alembic.migration] Running upgrade 27cc183af192 -> 50d5ba354c23, ml2 binding:vif_details
INFO [alembic.migration] Running upgrade 50d5ba354c23 -> 157a5d299379, ml2 binding:profile
INFO [alembic.migration] Running upgrade 157a5d299379 -> 3d2585038b95, VMware NSX rebranding
INFO [alembic.migration] Running upgrade 3d2585038b95 -> abc88c33f74f, lb stats
INFO [alembic.migration] Running upgrade abc88c33f74f -> 1b2580001654, nsx_sec_group_mapping
INFO [alembic.migration] Running upgrade 1b2580001654 -> e766b19a3bb, nuage_initial
INFO [alembic.migration] Running upgrade e766b19a3bb -> 2eeaf963a447, floatingip_status
INFO [alembic.migration] Running upgrade 2eeaf963a447 -> 492a106273f8, Brocade ML2 Mech. Driver
INFO [alembic.migration] Running upgrade 492a106273f8 -> 24c7ea5160d7, Cisco CSR VPNaaS
INFO [alembic.migration] Running upgrade 24c7ea5160d7 -> 81c553f3776c, bsn_consistencyhashes
INFO [alembic.migration] Running upgrade 81c553f3776c -> 117643811bca, nec: delete old ofc mapping tables
INFO [alembic.migration] Running upgrade 117643811bca -> 19180cf98af6, nsx_gw_devices
INFO [alembic.migration] Running upgrade 19180cf98af6 -> 33dd0a9fa487, embrane_lbaas_driver
INFO [alembic.migration] Running upgrade 33dd0a9fa487 -> 2447ad0e9585, Add IPv6 Subnet properties
INFO [alembic.migration] Running upgrade 2447ad0e9585 -> 538732fa21e1, NEC Rename quantum_id to neutron_id
INFO [alembic.migration] Running upgrade 538732fa21e1 -> 5ac1c354a051, n1kv segment allocs for cisco n1kv plugin
INFO [alembic.migration] Running upgrade 5ac1c354a051 -> icehouse, icehouse
INFO [alembic.migration] Running upgrade icehouse -> 54f7549a0e5f, set_not_null_peer_address
INFO [alembic.migration] Running upgrade 54f7549a0e5f -> 1e5dd1d09b22, set_not_null_fields_lb_stats
INFO [alembic.migration] Running upgrade 1e5dd1d09b22 -> b65aa907aec, set_length_of_protocol_field
INFO [alembic.migration] Running upgrade b65aa907aec -> 33c3db036fe4, set_length_of_description_field_metering
INFO [alembic.migration] Running upgrade 33c3db036fe4 -> 4eca4a84f08a, Remove ML2 Cisco Credentials DB
INFO [alembic.migration] Running upgrade 4eca4a84f08a -> d06e871c0d5, set_admin_state_up_not_null_ml2
INFO [alembic.migration] Running upgrade d06e871c0d5 -> 6be312499f9, set_not_null_vlan_id_cisco
INFO [alembic.migration] Running upgrade 6be312499f9 -> 1b837a7125a9, Cisco APIC Mechanism Driver
INFO [alembic.migration] Running upgrade 1b837a7125a9 -> 10cd28e692e9, nuage_extraroute
INFO [alembic.migration] Running upgrade 10cd28e692e9 -> 2db5203cb7a9, nuage_floatingip
INFO [alembic.migration] Running upgrade 2db5203cb7a9 -> 5446f2a45467, set_server_default
INFO [alembic.migration] Running upgrade 5446f2a45467 -> db_healing, Include all tables and make migrations unconditional.
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected server default on column 'cisco_ml2_apic_epgs.provider'
INFO [alembic.autogenerate.compare] Detected removed index 'cisco_n1kv_vlan_allocations_ibfk_1' on 'cisco_n1kv_vlan_allocations'
INFO [alembic.autogenerate.compare] Detected server default on column 'cisco_n1kv_vxlan_allocations.allocated'
INFO [alembic.autogenerate.compare] Detected removed index 'cisco_n1kv_vxlan_allocations_ibfk_1' on 'cisco_n1kv_vxlan_allocations'
INFO [alembic.autogenerate.compare] Detected removed index 'embrane_pool_port_ibfk_2' on 'embrane_pool_port'
INFO [alembic.autogenerate.compare] Detected removed index 'firewall_rules_ibfk_1' on 'firewall_rules'
INFO [alembic.autogenerate.compare] Detected removed index 'firewalls_ibfk_1' on 'firewalls'
INFO [alembic.autogenerate.compare] Detected server default on column 'meteringlabelrules.excluded'
INFO [alembic.autogenerate.compare] Detected server default on column 'ml2_port_bindings.host'
INFO [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.destination'
INFO [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.nexthop'
INFO [alembic.autogenerate.compare] Detected server default on column 'poolmonitorassociations.status'
INFO [alembic.autogenerate.compare] Detected added index 'ix_quotas_tenant_id' on '['tenant_id']'
INFO [alembic.autogenerate.compare] Detected NULL on column 'tz_network_bindings.phy_uuid'
INFO [alembic.autogenerate.compare] Detected NULL on column 'tz_network_bindings.vlan_id'
INFO [neutron.db.migration.alembic_migrations.heal_script] Detected removed foreign key u'nuage_floatingip_pool_mapping_ibfk_2' on table u'nuage_floatingip_pool_mapping'
INFO [alembic.migration] Running upgrade db_healing -> 3927f7f7c456, L3 extension distributed mode
INFO [alembic.migration] Running upgrade 3927f7f7c456 -> 2026156eab2f, L2 models to support DVR
INFO [alembic.migration] Running upgrade 2026156eab2f -> 37f322991f59, removing_mapping_tables
INFO [alembic.migration] Running upgrade 37f322991f59 -> 31d7f831a591, add constraint for routerid
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 173, in main
    CONF.command.func(config, CONF.command.name)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 83, in do_upgrade_downgrade
    do_alembic_command(config, cmd, revision, sql=CONF.command.sql)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 61, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/alembic/command.py", line 125, in upgrade
    script.run_env()
  File "/usr/lib/python2.7/dist-packages/alembic/script.py", line 203, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/dist-packages/alembic/util.py", line 212, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/dist-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/env.py", line 108, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/env.py", line 100, in run_migrations_online
    context.run_migrations()
  File "<string>", line 7, in run_migrations
  File "/usr/lib/python2.7/dist-packages/alembic/environment.py", line 688, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python2.7/dist-packages/alembic/migration.py", line 258, in run_migrations
    change(**kw)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/versions/31d7f831a591_add_constraint_for_routerid.py", line 85, in upgrade
    cols=['router_id', 'l3_agent_id']
  File "<string>", line 7, in create_primary_key
  File "/usr/lib/python2.7/dist-packages/alembic/operations.py", line 518, in create_primary_key
    schema)
  File "/usr/lib/python2.7/dist-packages/alembic/ddl/impl.py", line 135, in add_constraint
    self._exec(schema.AddConstraint(const))
  File "/usr/lib/python2.7/dist-packages/alembic/ddl/impl.py", line 76, in _exec
    conn.execute(construct, *multiparams, **params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/ddl.py", line 69, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 783, in _execute_ddl
    compiled
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
sqlalchemy.exc.OperationalError: (OperationalError) (1832, "Cannot change column 'l3_agent_id': used in a foreign key constraint 'routerl3agentbindings_ibfk_1'") 'ALTER TABLE routerl3agentbindings ADD CONSTRAINT pk_routerl3agentbindings PRIMARY KEY (router_id, l3_agent_id)' ()

Robert Campbell (camprr) on 2014-10-23

summary:	- Openstack Neutron Database error. + Openstack Neutron Database error on package install
summary:	- Openstack Neutron Database error on package install + Openstack Neutron Database error on filling database

Ann Taraday (akamyshnikova) on 2014-10-23

Changed in neutron:
assignee:	nobody → Ann Kamyshnikova (akamyshnikova)

Revision history for this message

Ann Taraday (akamyshnikova) wrote on 2014-10-23: Re: Openstack Neutron Database error on filling database

Checked this on MySQL and MariaDB. In both cases this error does not appear.
Logs of running with MariaDB - http://paste.openstack.org/show/123532/.
I used Juno and master neutron code also, both works fine.

Changed in neutron:
status:	New → Invalid

Revision history for this message

Robert Campbell (camprr) wrote on 2014-10-23:

Yes, with the standard neutron services it works, but we have some additional l2 and l3 plugins. WIll post config files. Here are the packages:

We are using Ubuntu 14.04 LTS and the staging-Juno packages:
root@vm-1:/var/log/mysql# dpkg --list |grep neutron
ii neutron-common 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-lbaas-agent 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - LBaaS agent
ii neutron-metadata-agent 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-metering-agent 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metering agent
ii neutron-plugin-ml2 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-plugin-openvswitch-agent 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-server 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - server
ii neutron-vpn-agent 1:2014.2-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - VPN agent
ii python-neutron 1:2014.2-0ubuntu1~cloud0 all Neutron is a virutal network service for Openstack - Python library
ii python-neutronclient 1:2.3.8-0ubuntu1~cloud0 all client - Neutron is a virtual network service for Openstack

Yes, with the standard neutron services it works, but we have some additional l2 and l3 plugins. WIll post config files. Here are the packages:

We are using Ubuntu 14.04 LTS and the staging-Juno packages:
root@vm-1:/var/log/mysql# dpkg --list |grep neutron
ii  neutron-common                           1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - common
ii  neutron-dhcp-agent                       1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - DHCP agent
ii  neutron-lbaas-agent                      1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - LBaaS agent
ii  neutron-metadata-agent                   1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - metadata agent
ii  neutron-metering-agent                   1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - metering agent
ii  neutron-plugin-ml2                       1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - ML2 plugin
ii  neutron-plugin-openvswitch-agent         1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii  neutron-server                           1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - server
ii  neutron-vpn-agent                        1:2014.2-0ubuntu1~cloud0              all          Neutron is a virtual network service for Openstack - VPN agent
ii  python-neutron                           1:2014.2-0ubuntu1~cloud0              all          Neutron is a virutal network service for Openstack - Python library
ii  python-neutronclient                     1:2.3.8-0ubuntu1~cloud0               all          client - Neutron is a virtual network service for Openstack

Revision history for this message

Robert Campbell (camprr) wrote on 2014-10-23:

neutron.conf Edit (23.8 KiB, text/plain)

Revision history for this message

Robert Campbell (camprr) wrote on 2014-10-23:

plugin.ini Edit (3.1 KiB, text/plain)

Robert Campbell (camprr) on 2014-10-23

Changed in neutron:
status:	Invalid → Incomplete

Tom Verdaat (tom-verdaat) on 2014-10-23

Changed in neutron:
status:	Incomplete → Confirmed

Revision history for this message

Tom Verdaat (tom-verdaat) wrote on 2014-10-23:

Can confirm that I've got the same problem on Ubuntu 14.04, Openstack Juno using the cloud-archive repository (juno-staging), ML2 with OVS and L2population plugins, flat and vxlan network(s), L3 HA and DVR enabled. Used puppet-neutron to deploy.

Revision history for this message

Ann Taraday (akamyshnikova) wrote on 2014-10-24:

This is very interesting. I installed neutron from cloud-archive repository (juno-staging) http://paste.openstack.org/show/123954/ on Ubuntu 14.04 I used plugin.ini and neutron.conf that was provided but still migrations run ok.

Also I'm not sure how this problem could depends on adding l2 and l3 agents as in juno migrations are unconditional, so running on empty database should be the same all the time.

In common such problem can appear if wrong engine is used ( but it set automatically for all tables as InnoDB) or if this "alter column" try to change size of the column for example, but here router_id, l3_agent_id has the same size and it is not changed.

So, I'm not sure how this could be fixed as I cannot reproduce this bug. Could you please verify that this bug can be reproduced in clean-room install and provide concrete steps to reproduce it?

Revision history for this message

Robert Campbell (camprr) wrote on 2014-10-24:

Download full text (11.0 KiB)

1. Install a virtual machine with 14.04.1 LTS server AMD-x64
2. Add juno-staging from https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/juno-staging
3. Add http://ftp.osuosl.org/pub/mariadb/repo/10.0/ubuntu and apt-get install galera mariadb-server-10.0 mariadb-server-core-10.0 mariadb-client-10.0 (because controller-node should be built using galera to enable multiple-controller setups with active/active replication)
4. Install neutron-common neutron-dhcp-agent neutron-lbaas-agent neutron-metadata-agent neutron-metering-agent neutron-plugin-ml2 neutron-plugin-openvswitch-agent neutron-server neutron-vpn-agent python-neutron python-neutronclient
5. apt-get install python-mysqldb
6. Add config files from this bug-report to /etc/neutron/
7. mysql -e "create database neutron;"
8. mysql -e "show databases;"
9. add user and flush privileges:
a) create user 'neutron'@'localhost' identified by 'C2qDdWD2Ij1LxullUS9nzSvS3ePGXDLA';
b) grant all privileges on *.* to 'neutron'@'localhost';
c) FLUSH PRIVILEGES;

1. Install a virtual machine with 14.04.1 LTS server AMD-x64
2. Add juno-staging from https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/juno-staging
3. Add http://ftp.osuosl.org/pub/mariadb/repo/10.0/ubuntu and apt-get install galera  mariadb-server-10.0 mariadb-server-core-10.0 mariadb-client-10.0 (because controller-node should be built using galera to enable multiple-controller setups with active/active replication)
4. Install neutron-common neutron-dhcp-agent neutron-lbaas-agent neutron-metadata-agent neutron-metering-agent neutron-plugin-ml2 neutron-plugin-openvswitch-agent neutron-server neutron-vpn-agent python-neutron python-neutronclient
5. apt-get install python-mysqldb
6. Add config files from this bug-report to /etc/neutron/
7. mysql -e "create database neutron;"
8. mysql -e "show databases;"
9. add user and flush privileges: 
a) create user 'neutron'@'localhost' identified by 'C2qDdWD2Ij1LxullUS9nzSvS3ePGXDLA';
b) grant all privileges on *.* to 'neutron'@'localhost';
c) FLUSH PRIVILEGES;

10. Change connection string in neutron.conf (127.0.0.1)
11. Run the command: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head
12. Observe result:
root@vm-1:/etc/neutron# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.migration] Running upgrade None -> havana, havana_initial
INFO  [alembic.migration] Running upgrade havana -> e197124d4b9, add unique constraint to members
INFO  [alembic.migration] Running upgrade e197124d4b9 -> 1fcfc149aca4, Add a unique constraint on (agent_type, host) columns to prevent a race
condition when an agent entry is 'upserted'.
INFO  [alembic.migration] Running upgrade 1fcfc149aca4 -> 50e86cb2637a, nsx_mappings
INFO  [alembic.migration] Running upgrade 50e86cb2637a -> 1421183d533f, NSX DHCP/metadata support
INFO  [alembic.migration] Running upgrade 1421183d533f -> 3d3cb89d84ee, nsx_switch_mappings
INFO  [alembic.migration] Running upgrade 3d3cb89d84ee -> 4ca36cfc898c, nsx_router_mappings
INFO  [alembic.migration] Running upgrade 4ca36cfc898c -> 27cc183af192, ml2_vnic_type
INFO  [alembic.migration] Running upgrade 27cc183af192 -> 50d5ba354c23, ml2 binding:vif_details
INFO  [alembic.migration] Running upgrade 50d5ba354c23 -> 157a5d299379, ml2 binding:profile
INFO  [alembic.migration] Running upgrade 157a5d299379 -> 3d2585038b95, VMware NSX rebranding
INFO  [alembic.migration] Running upgrade 3d2585038b95 -> abc88c33f74f, lb stats
INFO  [alembic.migration] Running upgrade abc88c33f74f -> 1b2580001654, nsx_sec_group_mapping
INFO  [alembic.migration] Running upgrade 1b2580001654 -> e766b19a3bb, nuage_initial
INFO  [alembic.migration] Running upgrade e766b19a3bb -> 2eeaf963a447, floatingip_status
INFO  [alembic.migration] Running upgrade 2eeaf963a447 -> 492a106273f8, Brocade ML2 Mech. Driver
INFO  [alembic.migration] Running upgrade 492a106273f8 -> 24c7ea5160d7, Cisco CSR VPNaaS
INFO  [alembic.migration] Running upgrade 24c7ea5160d7 -> 81c553f3776c, bsn_consistencyhashes
INFO  [alembic.migration] Running upgrade 81c553f3776c -> 117643811bca, nec: delete old ofc mapping tables
INFO  [alembic.migration] Running upgrade 117643811bca -> 19180cf98af6, nsx_gw_devices
INFO  [alembic.migration] Running upgrade 19180cf98af6 -> 33dd0a9fa487, embrane_lbaas_driver
INFO  [alembic.migration] Running upgrade 33dd0a9fa487 -> 2447ad0e9585, Add IPv6 Subnet properties
INFO  [alembic.migration] Running upgrade 2447ad0e9585 -> 538732fa21e1, NEC Rename quantum_id to neutron_id
INFO  [alembic.migration] Running upgrade 538732fa21e1 -> 5ac1c354a051, n1kv segment allocs for cisco n1kv plugin
INFO  [alembic.migration] Running upgrade 5ac1c354a051 -> icehouse, icehouse
INFO  [alembic.migration] Running upgrade icehouse -> 54f7549a0e5f, set_not_null_peer_address
INFO  [alembic.migration] Running upgrade 54f7549a0e5f -> 1e5dd1d09b22, set_not_null_fields_lb_stats
INFO  [alembic.migration] Running upgrade 1e5dd1d09b22 -> b65aa907aec, set_length_of_protocol_field
INFO  [alembic.migration] Running upgrade b65aa907aec -> 33c3db036fe4, set_length_of_description_field_metering
INFO  [alembic.migration] Running upgrade 33c3db036fe4 -> 4eca4a84f08a, Remove ML2 Cisco Credentials DB
INFO  [alembic.migration] Running upgrade 4eca4a84f08a -> d06e871c0d5, set_admin_state_up_not_null_ml2
INFO  [alembic.migration] Running upgrade d06e871c0d5 -> 6be312499f9, set_not_null_vlan_id_cisco
INFO  [alembic.migration] Running upgrade 6be312499f9 -> 1b837a7125a9, Cisco APIC Mechanism Driver
INFO  [alembic.migration] Running upgrade 1b837a7125a9 -> 10cd28e692e9, nuage_extraroute
INFO  [alembic.migration] Running upgrade 10cd28e692e9 -> 2db5203cb7a9, nuage_floatingip
INFO  [alembic.migration] Running upgrade 2db5203cb7a9 -> 5446f2a45467, set_server_default
INFO  [alembic.migration] Running upgrade 5446f2a45467 -> db_healing, Include all tables and make migrations unconditional.
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_ml2_apic_epgs.provider'
INFO  [alembic.autogenerate.compare] Detected removed index 'cisco_n1kv_vlan_allocations_ibfk_1' on 'cisco_n1kv_vlan_allocations'
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_n1kv_vxlan_allocations.allocated'
INFO  [alembic.autogenerate.compare] Detected removed index 'cisco_n1kv_vxlan_allocations_ibfk_1' on 'cisco_n1kv_vxlan_allocations'
INFO  [alembic.autogenerate.compare] Detected removed index 'embrane_pool_port_ibfk_2' on 'embrane_pool_port'
INFO  [alembic.autogenerate.compare] Detected removed index 'firewall_rules_ibfk_1' on 'firewall_rules'
INFO  [alembic.autogenerate.compare] Detected removed index 'firewalls_ibfk_1' on 'firewalls'
INFO  [alembic.autogenerate.compare] Detected server default on column 'meteringlabelrules.excluded'
INFO  [alembic.autogenerate.compare] Detected server default on column 'ml2_port_bindings.host'
INFO  [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.destination'
INFO  [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.nexthop'
INFO  [alembic.autogenerate.compare] Detected server default on column 'poolmonitorassociations.status'
INFO  [alembic.autogenerate.compare] Detected added index 'ix_quotas_tenant_id' on '['tenant_id']'
INFO  [alembic.autogenerate.compare] Detected NULL on column 'tz_network_bindings.phy_uuid'
INFO  [alembic.autogenerate.compare] Detected NULL on column 'tz_network_bindings.vlan_id'
INFO  [neutron.db.migration.alembic_migrations.heal_script] Detected removed foreign key u'nuage_floatingip_pool_mapping_ibfk_2' on table u'nuage_floatingip_pool_mapping'
INFO  [alembic.migration] Running upgrade db_healing -> 3927f7f7c456, L3 extension distributed mode
INFO  [alembic.migration] Running upgrade 3927f7f7c456 -> 2026156eab2f, L2 models to support DVR
INFO  [alembic.migration] Running upgrade 2026156eab2f -> 37f322991f59, removing_mapping_tables
INFO  [alembic.migration] Running upgrade 37f322991f59 -> 31d7f831a591, add constraint for routerid
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 173, in main
    CONF.command.func(config, CONF.command.name)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 83, in do_upgrade_downgrade
    do_alembic_command(config, cmd, revision, sql=CONF.command.sql)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 61, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/alembic/command.py", line 125, in upgrade
    script.run_env()
  File "/usr/lib/python2.7/dist-packages/alembic/script.py", line 203, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/dist-packages/alembic/util.py", line 212, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/dist-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/env.py", line 108, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/env.py", line 100, in run_migrations_online
    context.run_migrations()
  File "<string>", line 7, in run_migrations
  File "/usr/lib/python2.7/dist-packages/alembic/environment.py", line 688, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python2.7/dist-packages/alembic/migration.py", line 258, in run_migrations
    change(**kw)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/versions/31d7f831a591_add_constraint_for_routerid.py", line 85, in upgrade
    cols=['router_id', 'l3_agent_id']
  File "<string>", line 7, in create_primary_key
  File "/usr/lib/python2.7/dist-packages/alembic/operations.py", line 518, in create_primary_key
    schema)
  File "/usr/lib/python2.7/dist-packages/alembic/ddl/impl.py", line 135, in add_constraint
    self._exec(schema.AddConstraint(const))
  File "/usr/lib/python2.7/dist-packages/alembic/ddl/impl.py", line 76, in _exec
    conn.execute(construct, *multiparams, **params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/ddl.py", line 69, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 783, in _execute_ddl
    compiled
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
sqlalchemy.exc.OperationalError: (OperationalError) (1832, "Cannot change column 'l3_agent_id': used in a foreign key constraint 'routerl3agentbindings_ibfk_1'") 'ALTER TABLE routerl3agentbindings ADD CONSTRAINT pk_routerl3agentbindings PRIMARY KEY (router_id, l3_agent_id)' ()

Revision history for this message

Robert Campbell (camprr) wrote on 2014-10-24:

Download full text (8.0 KiB)

Solved:
Just tried with Server version: 5.5.39-MariaDB-0ubuntu0.14.04.1-log (Ubuntu):

Solved:
Just tried with Server version: 5.5.39-MariaDB-0ubuntu0.14.04.1-log (Ubuntu):

root@vm-1:/var/lib# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.migration] Running upgrade None -> havana, havana_initial
INFO  [alembic.migration] Running upgrade havana -> e197124d4b9, add unique constraint to members
INFO  [alembic.migration] Running upgrade e197124d4b9 -> 1fcfc149aca4, Add a unique constraint on (agent_type, host) columns to prevent a race
condition when an agent entry is 'upserted'.
INFO  [alembic.migration] Running upgrade 1fcfc149aca4 -> 50e86cb2637a, nsx_mappings
INFO  [alembic.migration] Running upgrade 50e86cb2637a -> 1421183d533f, NSX DHCP/metadata support
INFO  [alembic.migration] Running upgrade 1421183d533f -> 3d3cb89d84ee, nsx_switch_mappings
INFO  [alembic.migration] Running upgrade 3d3cb89d84ee -> 4ca36cfc898c, nsx_router_mappings
INFO  [alembic.migration] Running upgrade 4ca36cfc898c -> 27cc183af192, ml2_vnic_type
INFO  [alembic.migration] Running upgrade 27cc183af192 -> 50d5ba354c23, ml2 binding:vif_details
INFO  [alembic.migration] Running upgrade 50d5ba354c23 -> 157a5d299379, ml2 binding:profile
INFO  [alembic.migration] Running upgrade 157a5d299379 -> 3d2585038b95, VMware NSX rebranding
INFO  [alembic.migration] Running upgrade 3d2585038b95 -> abc88c33f74f, lb stats
INFO  [alembic.migration] Running upgrade abc88c33f74f -> 1b2580001654, nsx_sec_group_mapping
INFO  [alembic.migration] Running upgrade 1b2580001654 -> e766b19a3bb, nuage_initial
INFO  [alembic.migration] Running upgrade e766b19a3bb -> 2eeaf963a447, floatingip_status
INFO  [alembic.migration] Running upgrade 2eeaf963a447 -> 492a106273f8, Brocade ML2 Mech. Driver
INFO  [alembic.migration] Running upgrade 492a106273f8 -> 24c7ea5160d7, Cisco CSR VPNaaS
INFO  [alembic.migration] Running upgrade 24c7ea5160d7 -> 81c553f3776c, bsn_consistencyhashes
INFO  [alembic.migration] Running upgrade 81c553f3776c -> 117643811bca, nec: delete old ofc mapping tables
INFO  [alembic.migration] Running upgrade 117643811bca -> 19180cf98af6, nsx_gw_devices
INFO  [alembic.migration] Running upgrade 19180cf98af6 -> 33dd0a9fa487, embrane_lbaas_driver
INFO  [alembic.migration] Running upgrade 33dd0a9fa487 -> 2447ad0e9585, Add IPv6 Subnet properties
INFO  [alembic.migration] Running upgrade 2447ad0e9585 -> 538732fa21e1, NEC Rename quantum_id to neutron_id
INFO  [alembic.migration] Running upgrade 538732fa21e1 -> 5ac1c354a051, n1kv segment allocs for cisco n1kv plugin
INFO  [alembic.migration] Running upgrade 5ac1c354a051 -> icehouse, icehouse
INFO  [alembic.migration] Running upgrade icehouse -> 54f7549a0e5f, set_not_null_peer_address
INFO  [alembic.migration] Running upgrade 54f7549a0e5f -> 1e5dd1d09b22, set_not_null_fields_lb_stats
INFO  [alembic.migration] Running upgrade 1e5dd1d09b22 -> b65aa907aec, set_length_of_protocol_field
INFO  [alembic.migration] Running upgrade b65aa907aec -> 33c3db036fe4, set_length_of_description_field_metering
INFO  [alembic.migration] Running upgrade 33c3db036fe4 -> 4eca4a84f08a, Remove ML2 Cisco Credentials DB
INFO  [alembic.migration] Running upgrade 4eca4a84f08a -> d06e871c0d5, set_admin_state_up_not_null_ml2
INFO  [alembic.migration] Running upgrade d06e871c0d5 -> 6be312499f9, set_not_null_vlan_id_cisco
INFO  [alembic.migration] Running upgrade 6be312499f9 -> 1b837a7125a9, Cisco APIC Mechanism Driver
INFO  [alembic.migration] Running upgrade 1b837a7125a9 -> 10cd28e692e9, nuage_extraroute
INFO  [alembic.migration] Running upgrade 10cd28e692e9 -> 2db5203cb7a9, nuage_floatingip
INFO  [alembic.migration] Running upgrade 2db5203cb7a9 -> 5446f2a45467, set_server_default
INFO  [alembic.migration] Running upgrade 5446f2a45467 -> db_healing, Include all tables and make migrations unconditional.
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_ml2_apic_epgs.provider'
INFO  [alembic.autogenerate.compare] Detected removed index 'cisco_n1kv_vlan_allocations_ibfk_1' on 'cisco_n1kv_vlan_allocations'
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_n1kv_vxlan_allocations.allocated'
INFO  [alembic.autogenerate.compare] Detected removed index 'cisco_n1kv_vxlan_allocations_ibfk_1' on 'cisco_n1kv_vxlan_allocations'
INFO  [alembic.autogenerate.compare] Detected removed index 'embrane_pool_port_ibfk_2' on 'embrane_pool_port'
INFO  [alembic.autogenerate.compare] Detected removed index 'firewall_rules_ibfk_1' on 'firewall_rules'
INFO  [alembic.autogenerate.compare] Detected removed index 'firewalls_ibfk_1' on 'firewalls'
INFO  [alembic.autogenerate.compare] Detected server default on column 'meteringlabelrules.excluded'
INFO  [alembic.autogenerate.compare] Detected server default on column 'ml2_port_bindings.host'
INFO  [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.destination'
INFO  [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.nexthop'
INFO  [alembic.autogenerate.compare] Detected server default on column 'poolmonitorassociations.status'
INFO  [alembic.autogenerate.compare] Detected added index 'ix_quotas_tenant_id' on '['tenant_id']'
INFO  [alembic.autogenerate.compare] Detected NULL on column 'tz_network_bindings.phy_uuid'
INFO  [alembic.autogenerate.compare] Detected NULL on column 'tz_network_bindings.vlan_id'
INFO  [neutron.db.migration.alembic_migrations.heal_script] Detected removed foreign key u'nuage_floatingip_pool_mapping_ibfk_2' on table u'nuage_floatingip_pool_mapping'
INFO  [alembic.migration] Running upgrade db_healing -> 3927f7f7c456, L3 extension distributed mode
INFO  [alembic.migration] Running upgrade 3927f7f7c456 -> 2026156eab2f, L2 models to support DVR
INFO  [alembic.migration] Running upgrade 2026156eab2f -> 37f322991f59, removing_mapping_tables
INFO  [alembic.migration] Running upgrade 37f322991f59 -> 31d7f831a591, add constraint for routerid
INFO  [alembic.migration] Running upgrade 31d7f831a591 -> 5589aa32bf80, L3 scheduler additions to support DVR
INFO  [alembic.migration] Running upgrade 5589aa32bf80 -> 884573acbf1c, Drop NSX table in favor of the extra_attributes one
INFO  [alembic.migration] Running upgrade 884573acbf1c -> 4eba2f05c2f4, correct Vxlan Endpoint primary key
INFO  [alembic.migration] Running upgrade 4eba2f05c2f4 -> 327ee5fde2c7, set_innodb_engine
INFO  [alembic.migration] Running upgrade 327ee5fde2c7 -> 3b85b693a95f, Drop unused servicedefinitions and servicetypes tables.
INFO  [alembic.migration] Running upgrade 3b85b693a95f -> aae5706a396, nuage_provider_networks
INFO  [alembic.migration] Running upgrade aae5706a396 -> 32f3915891fd, cisco_apic_driver_update
INFO  [alembic.migration] Running upgrade 32f3915891fd -> 58fe87a01143, cisco_csr_routing
INFO  [alembic.migration] Running upgrade 58fe87a01143 -> 236b90af57ab, ml2_type_driver_refactor_dynamic_segments
INFO  [alembic.migration] Running upgrade 236b90af57ab -> 86d6d9776e2b, Cisco APIC Mechanism Driver
INFO  [alembic.migration] Running upgrade 86d6d9776e2b -> 16a27a58e093, ext_l3_ha_mode
INFO  [alembic.migration] Running upgrade 16a27a58e093 -> 3c346828361e, metering_label_shared
INFO  [alembic.migration] Running upgrade 3c346828361e -> 1680e1f0c4dc, Remove Cisco Nexus Monolithic Plugin
/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py:436: Warning: Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. Statements writing to a table with an auto-increment column after selecting from another table are unsafe because the order in which rows are retrieved determines what (if any) rows will be written. This order cannot be predicted and may differ on master and the slave.
  cursor.execute(statement, parameters)
INFO  [alembic.migration] Running upgrade 1680e1f0c4dc -> 544673ac99ab, add router port relationship
INFO  [alembic.migration] Running upgrade 544673ac99ab -> juno, juno

Revision history for this message

Tom Verdaat (tom-verdaat) wrote on 2014-10-24:

So this is indeed a bug, but the real problem is not so much the neutron database model itself but SQLAlchemy/Alembic generating SQL that doesn't work on MariaDB 10.0. Has MariaDB 10.0 broken SQL syntax?

Ann Taraday (akamyshnikova) on 2014-10-24

Changed in neutron:
status:	Confirmed → Invalid
status:	Invalid → Opinion

Tom Verdaat (tom-verdaat) on 2014-10-24

summary:

- Openstack Neutron Database error on filling database
+ SQL error during alembic.migration when populating Neutron database on
+ MariaDB 10.0

Revision history for this message

Eugene Nikanorov (enikanorov) wrote on 2014-10-24:

#10

I think since this is problem with specific version of sql backend while other versions work correctly there's no need to fix this.

Changed in neutron:
status:	Opinion → Won't Fix

Revision history for this message

Tom Verdaat (tom-verdaat) wrote on 2014-10-24:

#11

Due to this bug Neutron will only work with version <= 5.5 of mysql/mariadb, e.g. older versions. Downgrading to 5.5 isn't a solution, it's a workaround! It also means there is no database upgrade path for OpenStack, and this version limitation isn't mentioned anywhere in the documentation.

I'd say that's a pretty big deal that somebody should look at! If not a Neutron core developer, then somebody that works on SQLAlchemy. If nobody is willing to address the actual technical issue then at least the version limitation should be added to the documentation.

Can't force you to work on this, just saying that I think ignoring this would be a mistake.

Revision history for this message

Ann Taraday (akamyshnikova) wrote on 2014-10-24:

#12

I used 10.0.14 and migrations work. I this is problem of current 10.0 versionof MariaDB. MySQL works fine and it is tested a lot.

Revision history for this message

Jakub Libosvar (libosvar) wrote on 2014-10-30:

#13

We're seeing this issue on Fedora 21 with mariadb-10.0.14-7.fc21.x86_64, it doesn't reproduce on Fedora 20

Revision history for this message

Jakub Libosvar (libosvar) wrote on 2014-10-30:

#14

The problem since MySQL 5.6 forbids to alter column with FK constraint. I tested solution we drop both FKs (router_id, l3_agent_id), set new primary key for l3_agent_id and then re-create FKs and it works. I'll send patch soon.

Revision history for this message

Ann Taraday (akamyshnikova) wrote on 2014-10-31:

#15

Interesting as on MySQL 5.6 I still don't have any errors, tried mariadb 10.1 and migrations still worked.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-31: Fix proposed to neutron (master)

#16

Fix proposed to branch: master
Review: https://review.openstack.org/132273

Changed in neutron:
assignee:	Ann Kamyshnikova (akamyshnikova) → Jakub Libosvar (libosvar)
status:	Won't Fix → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-31:

#17

Fix proposed to branch: master
Review: https://review.openstack.org/132275

Changed in neutron:
assignee:	Jakub Libosvar (libosvar) → Lars Kellogg-Stedman (larsks)

Revision history for this message

Tim Flink (tflink) wrote on 2014-10-31:

#18

I tried the patch from c#17 in a packstack install on fedora 21. After dropping the old neutron database and rebuilding it, the migration now works for me

OpenStack Infra (hudson-openstack) on 2014-10-31

Changed in neutron:
assignee:	Lars Kellogg-Stedman (larsks) → Jakub Libosvar (libosvar)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-10-31: Change abandoned on neutron (master)

#19

Change abandoned by Lars Kellogg-Stedman (<email address hidden>) on branch: master
Review: https://review.openstack.org/132275
Reason: See https://review.openstack.org/132273

Armando Migliaccio (armando-migliaccio) on 2014-11-20

Changed in neutron:
milestone:	none → kilo-1
importance:	Undecided → Medium
tags:	added: juno-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-21: Fix merged to neutron (master)

#20

Reviewed: https://review.openstack.org/132273
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b5f1cc48b877caaebc944154d56a8bb4060aa9b0
Submitter: Jenkins
Branch: master

commit b5f1cc48b877caaebc944154d56a8bb4060aa9b0
Author: Jakub Libosvar <email address hidden>
Date: Fri Oct 31 15:49:40 2014 +0100

Drop and recreate FK if adding new PK to routerl3bindings

Since MySQL 5.6 FK columns cannot be altered. As a workaround we can
drop the FK before ALTER COLUMN and then recreate it.

Change-Id: Icd1b3075cf29a6b0c477a4ddea2e6ebee91baef8
Closes-Bug: 1384555

Changed in neutron:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-24: Fix proposed to neutron (stable/juno)

#21

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/136710

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-26: Fix merged to neutron (stable/juno)

#22

Reviewed: https://review.openstack.org/136710
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=467a4496cfcd74b473730d2baa667fd633e4d0b3
Submitter: Jenkins
Branch: stable/juno

commit 467a4496cfcd74b473730d2baa667fd633e4d0b3
Author: Jakub Libosvar <email address hidden>
Date: Fri Oct 31 15:49:40 2014 +0100

Drop and recreate FK if adding new PK to routerl3bindings

Since MySQL 5.6 FK columns cannot be altered. As a workaround we can
drop the FK before ALTER COLUMN and then recreate it.

    Change-Id: Icd1b3075cf29a6b0c477a4ddea2e6ebee91baef8
    Closes-Bug: 1384555
    (cherry picked from commit b5f1cc48b877caaebc944154d56a8bb4060aa9b0)

tags:

added: in-stable-juno

Alan Pevec (apevec) on 2014-12-08

tags:	removed: in-stable-juno juno-backport-potential
tags:	removed: clean database error install networking neutron

Revision history for this message

Launchpad Janitor (janitor) wrote on 2014-12-15:

#23

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in neutron (Ubuntu):
status:	New → Confirmed

Thierry Carrez (ttx) on 2014-12-18

Changed in neutron:
status:	Fix Committed → Fix Released

Revision history for this message

Kevin Bringard (kbringard) wrote on 2015-02-19:

#24

Download full text (11.6 KiB)

I'm not sure this is entirely fixed. I'm seeing very similar behavior running Percona XtraDB Cluster 5.5.39-36.0-55, and neutron 2014.2.1 (the migrations come from: python-neutron-2014.2.1-6.el7ost.noarch):

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 11549
Server version: 5.5.39-36.0-55 Percona XtraDB Cluster (GPL), Release rel36.0, Revision 824, WSREP version 25.11, wsrep_25.11.r4023

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show tables;
Empty set (0.00 sec)

mysql>

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini current
INFO [alembic.migration] Context impl MySQLImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
Current revision for mysql://neutron:XXXXX@10.202.117.5/neutron: None

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11549
Server version: 5.5.39-36.0-55 Percona XtraDB Cluster (GPL), Release rel36.0, Revision 824, WSREP version 25.11, wsrep_25.11.r4023

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show tables;
Empty set (0.00 sec)

mysql>

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini current
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
Current revision for mysql://neutron:XXXXX@10.202.117.5/neutron: None

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.migration] Running upgrade None -> havana, havana_initial
INFO  [alembic.migration] Running upgrade havana -> e197124d4b9, add unique constraint to members
INFO  [alembic.migration] Running upgrade e197124d4b9 -> 1fcfc149aca4, Add a unique constraint on (agent_type, host) columns to prevent a race
condition when an agent entry is 'upserted'.
INFO  [alembic.migration] Running upgrade 1fcfc149aca4 -> 50e86cb2637a, nsx_mappings
INFO  [alembic.migration] Running upgrade 50e86cb2637a -> 1421183d533f, NSX DHCP/metadata support
INFO  [alembic.migration] Running upgrade 1421183d533f -> 3d3cb89d84ee, nsx_switch_mappings
INFO  [alembic.migration] Running upgrade 3d3cb89d84ee -> 4ca36cfc898c, nsx_router_mappings
INFO  [alembic.migration] Running upgrade 4ca36cfc898c -> 27cc183af192, ml2_vnic_type
INFO  [alembic.migration] Running upgrade 27cc183af192 -> 50d5ba354c23, ml2 binding:vif_details
INFO  [alembic.migration] Running upgrade 50d5ba354c23 -> 157a5d299379, ml2 binding:profile
INFO  [alembic.migration] Running upgrade 157a5d299379 -> 3d2585038b95, VMware NSX rebranding
INFO  [alembic.migration] Running upgrade 3d2585038b95 -> abc88c33f74f, lb stats
INFO  [alembic.migration] Running upgrade abc88c33f74f -> 1b2580001654, nsx_sec_group_mapping
INFO  [alembic.migration] Running upgrade 1b2580001654 -> e766b19a3bb, nuage_initial
INFO  [alembic.migration] Running upgrade e766b19a3bb -> 2eeaf963a447, floatingip_status
INFO  [alembic.migration] Running upgrade 2eeaf963a447 -> 492a106273f8, Brocade ML2 Mech. Driver
INFO  [alembic.migration] Running upgrade 492a106273f8 -> 24c7ea5160d7, Cisco CSR VPNaaS
INFO  [alembic.migration] Running upgrade 24c7ea5160d7 -> 81c553f3776c, bsn_consistencyhashes
INFO  [alembic.migration] Running upgrade 81c553f3776c -> 117643811bca, nec: delete old ofc mapping tables
INFO  [alembic.migration] Running upgrade 117643811bca -> 19180cf98af6, nsx_gw_devices
INFO  [alembic.migration] Running upgrade 19180cf98af6 -> 33dd0a9fa487, embrane_lbaas_driver
INFO  [alembic.migration] Running upgrade 33dd0a9fa487 -> 2447ad0e9585, Add IPv6 Subnet properties
INFO  [alembic.migration] Running upgrade 2447ad0e9585 -> 538732fa21e1, NEC Rename quantum_id to neutron_id
INFO  [alembic.migration] Running upgrade 538732fa21e1 -> 5ac1c354a051, n1kv segment allocs for cisco n1kv plugin
INFO  [alembic.migration] Running upgrade 5ac1c354a051 -> icehouse, icehouse
INFO  [alembic.migration] Running upgrade icehouse -> 54f7549a0e5f, set_not_null_peer_address
INFO  [alembic.migration] Running upgrade 54f7549a0e5f -> 1e5dd1d09b22, set_not_null_fields_lb_stats
INFO  [alembic.migration] Running upgrade 1e5dd1d09b22 -> b65aa907aec, set_length_of_protocol_field
INFO  [alembic.migration] Running upgrade b65aa907aec -> 33c3db036fe4, set_length_of_description_field_metering
INFO  [alembic.migration] Running upgrade 33c3db036fe4 -> 4eca4a84f08a, Remove ML2 Cisco Credentials DB
INFO  [alembic.migration] Running upgrade 4eca4a84f08a -> d06e871c0d5, set_admin_state_up_not_null_ml2
INFO  [alembic.migration] Running upgrade d06e871c0d5 -> 6be312499f9, set_not_null_vlan_id_cisco
INFO  [alembic.migration] Running upgrade 6be312499f9 -> 1b837a7125a9, Cisco APIC Mechanism Driver
INFO  [alembic.migration] Running upgrade 1b837a7125a9 -> 10cd28e692e9, nuage_extraroute
INFO  [alembic.migration] Running upgrade 10cd28e692e9 -> 2db5203cb7a9, nuage_floatingip
INFO  [alembic.migration] Running upgrade 2db5203cb7a9 -> 5446f2a45467, set_server_default
INFO  [alembic.migration] Running upgrade 5446f2a45467 -> db_healing, Include all tables and make migrations unconditional.
INFO  [alembic.migration] Context impl MySQLImpl.
INFO  [alembic.migration] Will assume non-transactional DDL.
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_ml2_apic_epgs.provider'
INFO  [alembic.autogenerate.compare] Detected removed index 'cisco_n1kv_vlan_allocations_ibfk_1' on 'cisco_n1kv_vlan_allocations'
INFO  [alembic.autogenerate.compare] Detected server default on column 'cisco_n1kv_vxlan_allocations.allocated'
INFO  [alembic.autogenerate.compare] Detected removed index 'cisco_n1kv_vxlan_allocations_ibfk_1' on 'cisco_n1kv_vxlan_allocations'
INFO  [alembic.autogenerate.compare] Detected removed index 'embrane_pool_port_ibfk_2' on 'embrane_pool_port'
INFO  [alembic.autogenerate.compare] Detected removed index 'firewall_rules_ibfk_1' on 'firewall_rules'
INFO  [alembic.autogenerate.compare] Detected removed index 'firewalls_ibfk_1' on 'firewalls'
INFO  [alembic.autogenerate.compare] Detected server default on column 'meteringlabelrules.excluded'
INFO  [alembic.autogenerate.compare] Detected server default on column 'ml2_port_bindings.host'
INFO  [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.destination'
INFO  [alembic.autogenerate.compare] Detected added column 'nuage_routerroutes_mapping.nexthop'
INFO  [alembic.autogenerate.compare] Detected server default on column 'poolmonitorassociations.status'
INFO  [alembic.autogenerate.compare] Detected added index 'ix_quotas_tenant_id' on '['tenant_id']'
INFO  [alembic.autogenerate.compare] Detected NULL on column 'tz_network_bindings.phy_uuid'
INFO  [alembic.autogenerate.compare] Detected NULL on column 'tz_network_bindings.vlan_id'
INFO  [neutron.db.migration.alembic_migrations.heal_script] Detected removed foreign key u'nuage_floatingip_pool_mapping_ibfk_2' on table u'nuage_floatingip_pool_mapping'
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 173, in main
    CONF.command.func(config, CONF.command.name)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 83, in do_upgrade_downgrade
    do_alembic_command(config, cmd, revision, sql=CONF.command.sql)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 61, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/alembic/command.py", line 125, in upgrade
    script.run_env()
  File "/usr/lib/python2.7/site-packages/alembic/script.py", line 203, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/site-packages/alembic/util.py", line 212, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/site-packages/alembic/compat.py", line 58, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 108, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 100, in run_migrations_online
    context.run_migrations()
  File "<string>", line 7, in run_migrations
  File "/usr/lib/python2.7/site-packages/alembic/environment.py", line 688, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python2.7/site-packages/alembic/migration.py", line 258, in run_migrations
    change(**kw)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/versions/1d6ee1ae5da5_db_healing.py", line 32, in upgrade
    heal_script.heal()
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/heal_script.py", line 80, in heal
    execute_alembic_command(el)
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/heal_script.py", line 92, in execute_alembic_command
    METHODS[command[0]](*command[1:])
  File "/usr/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/heal_script.py", line 175, in add_column
    op.add_column(table_name, column.copy(), schema=schema)
  File "<string>", line 7, in add_column
  File "/usr/lib/python2.7/site-packages/alembic/operations.py", line 428, in add_column
    schema=schema
  File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 127, in add_column
    self._exec(base.AddColumn(table_name, column, schema=schema))
  File "/usr/lib/python2.7/site-packages/alembic/ddl/impl.py", line 76, in _exec
    conn.execute(construct, *multiparams, **params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 69, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 783, in _execute_ddl
    compiled
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
    util.raise_from_cause(newraise, exc_info)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
oslo.db.exception.DBConnectionError: (OperationalError) (2013, 'Lost connection to MySQL server during query') 'ALTER TABLE nuage_routerroutes_mapping ADD COLUMN destination VARCHAR(64) NOT NULL' ()

The reason the connection to the DB is being lost is because it's crashing every node in the cluster.

mysql neutron
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)

systemctl status mysql
mysql.service - LSB: start and stop MySQL (Percona XtraDB Cluster)
   Loaded: loaded (/etc/rc.d/init.d/mysql)
   Active: failed (Result: exit-code) since Thu 2015-02-19 13:45:23 EST; 24min ago
  Process: 18961 ExecStart=/etc/rc.d/init.d/mysql start (code=exited, status=1/FAILURE)

I am able to manually go to havana and/or icehouse, but as soon as I try to make the last step to juno or head, I always get the same error. This happens with both a new DB or when attempting to upgrade an existing one.

James Page (james-page) on 2015-03-24

Changed in neutron (Ubuntu):
importance:	Undecided → Medium
status:	Confirmed → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in neutron:
milestone:	kilo-1 → 2015.1.0

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Ubuntuneutron package

SQL error during alembic.migration when populating Neutron database on MariaDB 10.0

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
neutron package