[RFE] Stop storing configdrive in the database

[David Edery]

In the Ironic database the 'instance_info' field as stored as a TEXT field. With MySQL this has a limit of 64KiB. The configdrive is stored in the instance_info field and so can not be larger than 64KiB. We have reports from multiple users who would like to be able to use configdrives larger than 64KIB.

Proposal is to switch to using the MEDIUMTEXT field type for instance_info for MySQL databases, which would allow up to 16MiB for the instance_info.

Also propose to have a configuration setting for the maximum size for the configdrive to by default be set to 64KiB. Database entry would still be MEDIUMTTEXT. Any configdrive that exceeds the config setting would be rejected. This would maintain the current size limitation and operators would not be surprised by their database growing unexpectedly. Operators could then change the config setting to increase the maximum size of the configdrive.

Update from dtantsur: since bug #1575935 is fixed, we no longer need to store the configdrive in the database at all.

[David Edery (david-edery)]

I'm going to follow Mistral's solution to the same text-is-too-short-for-me issue:
https://github.com/openstack/mistral/blob/master/mistral/db/sqlalchemy/types.py#L86

TBD: medium vs long text type

[David Edery (david-edery)]

TBD: other db flavors in which the same solution is needed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/334967

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/369617

[Jim Rollenhagen (jim-rollenhagen)]

We've discussed this, and the real solution is to do this: https://bugs.launchpad.net/ironic/+bug/1575935

Going to close this RFE.

[Ruby Loo (rloo)]

The discussion was at Monday's ironic meeting. Starting at 17:31:51: http://eavesdrop.openstack.org/meetings/ironic/2016/ironic.2016-11-21-17.00.log.html

[OpenStack Infra (hudson-openstack)]

Change abandoned by John L. Villalovos (<email address hidden>) on branch: master
Review: https://review.openstack.org/369617

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/334967
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=8fdf752ae9688c581fe8c977e8a5ffb9432583f3
Submitter: Zuul
Branch: master

commit 8fdf752ae9688c581fe8c977e8a5ffb9432583f3
Author: David Edery <email address hidden>
Date: Sun Sep 4 13:18:53 2016 +0300

    Increase the instance_info column size to LONGTEXT on MySQL/MariaDB

    In MySQL, the maximum size of TEXT is 64KiB. The instance_info column
    is defined as TEXT and can potentially hold big amount of data
    (partly affected by cloud-init scripts for the node).

    This patch-set resizes nodes.instance_info to LONGTEXT which allows up
    to 4GiB of data to be stored in the column. This change is relevant only
    to MySQL and does not affect PostgreSQL or any other DB flavor. The fix
    also addresses the upgrade process of the column.

    Change-Id: Ifb9a49d4258a559cf2175d902e9424a3f98065c5
    Closes-Bug: #1596421

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/546551

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546551
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=b777d3a517d4b1576a24ec7453eb5351a5893b71
Submitter: Zuul
Branch: stable/queens

commit b777d3a517d4b1576a24ec7453eb5351a5893b71
Author: David Edery <email address hidden>
Date: Sun Sep 4 13:18:53 2016 +0300

    Increase the instance_info column size to LONGTEXT on MySQL/MariaDB

    In MySQL, the maximum size of TEXT is 64KiB. The instance_info column
    is defined as TEXT and can potentially hold big amount of data
    (partly affected by cloud-init scripts for the node).

    This patch-set resizes nodes.instance_info to LONGTEXT which allows up
    to 4GiB of data to be stored in the column. This change is relevant only
    to MySQL and does not affect PostgreSQL or any other DB flavor. The fix
    also addresses the upgrade process of the column.

    Change-Id: Ifb9a49d4258a559cf2175d902e9424a3f98065c5
    Closes-Bug: #1596421
    (cherry picked from commit 8fdf752ae9688c581fe8c977e8a5ffb9432583f3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 10.1.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ironic 11.0.0 release.