DBError storing build_requests.instance with user_data too long

Bug #1635446 reported by Abhishek Chanda on 2016-10-20
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Magnum
Invalid
Undecided
Unassigned
OpenStack Compute (nova)
Critical
Matt Riedemann
Newton
Critical
Matt Riedemann

Bug Description

The cluster goes to CREATE_FAILED

# magnum $AUTH cluster-show k8s-cluster-fedora
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status | CREATE_FAILED |
| cluster_template_id | cd70e24d-717c-40b4-8bcf-fd56d0cf23bb |
| uuid | 5df0d9f3-de35-4557-881c-efac6bc7e489 |
| stack_id | 497e8905-f425-454b-8f77-d87ff41cc7dd |
| status_reason | Resource CREATE failed: ClientException: resources.kube_minions.resources[0].resources.kube-minion: Unexpected API Error. Please report this at http: //bugs.launchpad.net/nova/ and attach the Nova API log if possible. |
| | <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-c064df25-a47c-439e-bdc5-99e1cfbd44c5) |
| created_at | 2016-10-20T22:26:28+00:00 |
| name | k8s-cluster-fedora |
| updated_at | 2016-10-20T22:28:00+00:00 |
| discovery_url | https://discovery.etcd.io/703919614eb4688507382ff891fa86a9 |
| faults | {'0': "ClientException: resources[0].resources.kube-minion: Unexpected API Error. Please report this at http: //bugs.launchpad.net/nova/ and attach the Nova API log if possible. |
| | <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-c064df25-a47c-439e-bdc5-99e1cfbd44c5)", 'kube_minions': "ClientException: resources.kube_minions.resources[0].resources.kube-minion: Unexpected API Error. Please report this at http: //bugs.launchpad.net/nova/ and attach the Nova API log if possible. |
| | <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-c064df25-a47c-439e-bdc5-99e1cfbd44c5)", 'kube-minion': "ClientException: resources.kube-minion: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. |
| | <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-c064df25-a47c-439e-bdc5-99e1cfbd44c5)"} |
| api_address | https://10.240.127.136:6443 |
| coe_version | v1.2.0 |
| master_addresses | ['10.240.127.136'] |
| create_timeout | 18000 |
| node_addresses | ['10.240.127.141'] |
| master_count | 1 |
| container_version | 1.9.1 |
| node_count | 1 |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

nova-api logs attached

Abhishek Chanda (abhishek-i) wrote :
Abhishek Chanda (abhishek-i) wrote :

We did modify some of the scripts in fragments here and there. The image used is http://tarballs.openstack.org/magnum/images/fedora-atomic-f23-dib.qcow2

Abhishek Chanda (abhishek-i) wrote :

The instance column in the build_requests table in the nova_api db has the type TEXT, which is 65535 chars in Mariadb. The length of the blob that is in the nova api logs is around 67000 chars. This makes the db insert fail. This looks like more of a nova issue than magnum. In the case of Magnum, the workaround is to not include some of the SoftwareConfig files.

Matt Riedemann (mriedem) wrote :

The instance is serialized into the build_requests table:

https://github.com/openstack/nova/blob/14.0.1/nova/db/sqlalchemy/api_models.py#L243

From the nova-api log attached to this bug, it looks like the user_data is blowing it up:

https://github.com/openstack/nova/blob/14.0.1/nova/db/sqlalchemy/models.py#L281

Looking at the models, the build_requests.instance column is Text and the instances.user_data is MediumText, so that needs to change.

http://dev.mysql.com/doc/refman/5.7/en/string-type-overview.html

Changed in magnum:
status: New → Invalid
summary: - Magnum fails to create fedora minions
+ DBError storing build_requests.instance with user_data too long
Changed in nova:
status: New → Triaged
importance: Undecided → High
Matt Riedemann (mriedem) on 2016-10-26
Changed in nova:
assignee: nobody → Matt Riedemann (mriedem)
Matt Riedemann (mriedem) wrote :
Changed in nova:
status: Triaged → In Progress
importance: High → Critical

Reviewed: https://review.openstack.org/390812
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fd7b3d64c70eeae34dcf38a847180c60fddfed5c
Submitter: Jenkins
Branch: master

commit fd7b3d64c70eeae34dcf38a847180c60fddfed5c
Author: Matt Riedemann <email address hidden>
Date: Wed Oct 26 06:23:42 2016 -0400

    Make build_requests.instance MediumText

    The build_requests.instance column is a serialized
    instance object, and the instances.user_data column
    is MediumText, so the build_requests.instance column
    itself needs to be at least MediumText in size for MySQL.

    Change-Id: I7d65df37c02750593037744543ad15e5bc64e913
    Closes-Bug: #1635446

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/391086
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ee086460db947ea0dcd2cb27fd2aecf738b47c20
Submitter: Jenkins
Branch: stable/newton

commit ee086460db947ea0dcd2cb27fd2aecf738b47c20
Author: Matt Riedemann <email address hidden>
Date: Wed Oct 26 06:23:42 2016 -0400

    Make build_requests.instance MediumText

    The build_requests.instance column is a serialized
    instance object, and the instances.user_data column
    is MediumText, so the build_requests.instance column
    itself needs to be at least MediumText in size for MySQL.

    Conflicts:
            nova/tests/functional/db/api/test_migrations.py

    NOTE(mriedem): The conflict is due to not having the 026
    API DB migration test in newton.

    Change-Id: I7d65df37c02750593037744543ad15e5bc64e913
    Closes-Bug: #1635446
    (cherry picked from commit fd7b3d64c70eeae34dcf38a847180c60fddfed5c)

tags: added: in-stable-newton

Reviewed: https://review.openstack.org/391897
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=867661d51bdb0cf2a6f326cb18f26bbc1f04eb15
Submitter: Jenkins
Branch: stable/newton

commit 867661d51bdb0cf2a6f326cb18f26bbc1f04eb15
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 31 11:54:30 2016 -0400

    Add release note for bug 1635446

    Since we're adding a database schema migration in
    a stable release, we should have a release note to
    document it.

    Change-Id: Ic3f507efb3a5b39283f49ae36fc8fbd727b9d445
    Related-Bug: #1635446

This issue was fixed in the openstack/nova 14.0.2 release.

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.

This issue was fixed in the openstack/nova 14.0.2 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments