DBError storing build_requests.instance with user_data too long

[Abhishek Chanda]

The cluster goes to CREATE_FAILED

# magnum $AUTH cluster-show k8s-cluster-fedora
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| status | CREATE_FAILED |
| cluster_template_id | cd70e24d-717c-40b4-8bcf-fd56d0cf23bb |
| uuid | 5df0d9f3-de35-4557-881c-efac6bc7e489 |
| stack_id | 497e8905-f425-454b-8f77-d87ff41cc7dd |
| status_reason | Resource CREATE failed: ClientException: resources.kube_minions.resources[0].resources.kube-minion: Unexpected API Error. Please report this at http: //bugs.launchpad.net/nova/ and attach the Nova API log if possible. |
| | <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-c064df25-a47c-439e-bdc5-99e1cfbd44c5) |
| created_at | 2016-10-20T22:26:28+00:00 |
| name | k8s-cluster-fedora |
| updated_at | 2016-10-20T22:28:00+00:00 |
| discovery_url | https://discovery.etcd.io/703919614eb4688507382ff891fa86a9 |
| faults | {'0': "ClientException: resources[0].resources.kube-minion: Unexpected API Error. Please report this at http: //bugs.launchpad.net/nova/ and attach the Nova API log if possible. |
| | <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-c064df25-a47c-439e-bdc5-99e1cfbd44c5)", 'kube_minions': "ClientException: resources.kube_minions.resources[0].resources.kube-minion: Unexpected API Error. Please report this at http: //bugs.launchpad.net/nova/ and attach the Nova API log if possible. |
| | <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-c064df25-a47c-439e-bdc5-99e1cfbd44c5)", 'kube-minion': "ClientException: resources.kube-minion: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. |
| | <class 'oslo_db.exception.DBError'> (HTTP 500) (Request-ID: req-c064df25-a47c-439e-bdc5-99e1cfbd44c5)"} |
| api_address | https://10.240.127.136:6443 |
| coe_version | v1.2.0 |
| master_addresses | ['10.240.127.136'] |
| create_timeout | 18000 |
| node_addresses | ['10.240.127.141'] |
| master_count | 1 |
| container_version | 1.9.1 |
| node_count | 1 |
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

nova-api logs attached

[Abhishek Chanda (abhishek-i)]

We did modify some of the scripts in fragments here and there. The image used is http://tarballs.openstack.org/magnum/images/fedora-atomic-f23-dib.qcow2

[Abhishek Chanda (abhishek-i)]

The instance column in the build_requests table in the nova_api db has the type TEXT, which is 65535 chars in Mariadb. The length of the blob that is in the nova api logs is around 67000 chars. This makes the db insert fail. This looks like more of a nova issue than magnum. In the case of Magnum, the workaround is to not include some of the SoftwareConfig files.

[Matt Riedemann (mriedem)]

The instance is serialized into the build_requests table:

https://github.com/openstack/nova/blob/14.0.1/nova/db/sqlalchemy/api_models.py#L243

From the nova-api log attached to this bug, it looks like the user_data is blowing it up:

https://github.com/openstack/nova/blob/14.0.1/nova/db/sqlalchemy/models.py#L281

Looking at the models, the build_requests.instance column is Text and the instances.user_data is MediumText, so that needs to change.

http://dev.mysql.com/doc/refman/5.7/en/string-type-overview.html

[Matt Riedemann (mriedem)]

Patch: https://review.openstack.org/#/c/390812/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/391086

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/390812
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fd7b3d64c70eeae34dcf38a847180c60fddfed5c
Submitter: Jenkins
Branch: master

commit fd7b3d64c70eeae34dcf38a847180c60fddfed5c
Author: Matt Riedemann <email address hidden>
Date: Wed Oct 26 06:23:42 2016 -0400

    Make build_requests.instance MediumText

    The build_requests.instance column is a serialized
    instance object, and the instances.user_data column
    is MediumText, so the build_requests.instance column
    itself needs to be at least MediumText in size for MySQL.

    Change-Id: I7d65df37c02750593037744543ad15e5bc64e913
    Closes-Bug: #1635446

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/391897

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391086
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ee086460db947ea0dcd2cb27fd2aecf738b47c20
Submitter: Jenkins
Branch: stable/newton

commit ee086460db947ea0dcd2cb27fd2aecf738b47c20
Author: Matt Riedemann <email address hidden>
Date: Wed Oct 26 06:23:42 2016 -0400

    Make build_requests.instance MediumText

    The build_requests.instance column is a serialized
    instance object, and the instances.user_data column
    is MediumText, so the build_requests.instance column
    itself needs to be at least MediumText in size for MySQL.

    Conflicts:
            nova/tests/functional/db/api/test_migrations.py

    NOTE(mriedem): The conflict is due to not having the 026
    API DB migration test in newton.

    Change-Id: I7d65df37c02750593037744543ad15e5bc64e913
    Closes-Bug: #1635446
    (cherry picked from commit fd7b3d64c70eeae34dcf38a847180c60fddfed5c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391897
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=867661d51bdb0cf2a6f326cb18f26bbc1f04eb15
Submitter: Jenkins
Branch: stable/newton

commit 867661d51bdb0cf2a6f326cb18f26bbc1f04eb15
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 31 11:54:30 2016 -0400

    Add release note for bug 1635446

    Since we're adding a database schema migration in
    a stable release, we should have a release note to
    document it.

    Change-Id: Ic3f507efb3a5b39283f49ae36fc8fbd727b9d445
    Related-Bug: #1635446

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.2 release.