Instance stuck on building status if user has many roles

Bug #2075100 reported by Masahito Muroi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Undecided
Masahito Muroi

Bug Description

Description:

Instance boot operation is stuck in BUILDING status if user has some roles which total name length is over 255 chars.

Details:

The nova-compute manager tries to store all roles list of the request context in InstanceSystemMetadata in the __build_and_run_instance method[1], but the metadata value is limited to 255 chars as DB model[2]. So the total role name string length including ","-joined is over 255 chars, the instance boot operation gets fail due to the DB operation failure.

1. https://opendev.org/openstack/nova/src/commit/7a7427691e0bd4818bb7a2c5f5371e0244addbbb/nova/compute/manager.py#L2584-L2587
2. https://opendev.org/openstack/nova/src/commit/7a7427691e0bd4818bb7a2c5f5371e0244addbbb/nova/db/main/models.py#L980

Step to reproduce:

Create some roles which name length is over 255 characters.
In this sample, use three 100 characters roles.

$ openstack role create aaaaaa...<omit 90 a chars>..aaaa
$ openstack role create bbbbbb...<omit 90 b chars>..bbbb
$ openstack role create cccccc...<omit 90 b chars>..cccc

$ openstack role add --project admin --user admin <aaaa role id>
$ openstack role add --project admin --user admin <bbbb role id>
$ openstack role add --project admin --user admin <cccc role id>

$ openstack server create --image <image-id> --nic net-id=<net-id> --flavor <flavor-id> stuck-vm

The server, stuck-vm, gets stuck in status is "BUILD" and vm_state is "building" status.

Error log by nova-compute:

DBDataError (pymysql.err.DataError) (1406, "Data too long for column 'value' at row 1")

[SQL: INSERT INTO instance_system_metadata (created_at, updated_at, deleted_at, deleted, `key`, value, instance_uuid) VALUES (%(created_at)s, %(updated_at)s, %(deleted_at)s, %(deleted)s, %(key)s, %(value)s, %(instance_uuid)s)]
[parameters: {'created_at': datetime.datetime(2024, 7, 12, 18, 57, 35, 750579), 'updated_at': None, 'deleted_at': None, 'deleted': 0, 'key': 'boot_roles', 'value': 'ccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbbccccccccccbbbbbbbbbb,ddddddddddbbbbbbbbbbddddddddddbbbbbbbbbbdddddddd ... (32 characters truncated) ... ddddddddddbbbbbbbbbb,manager,admin,reader,member,aaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbbaaaaaaaaaabbbbbbbbbb', 'instance_uuid': '1bbd12b0-62d1-4b03-88e2-0edc77517261'}]
(Background on this error at: https://sqlalche.me/e/14/9h9h)

Environment:

Caracal release (2024.1) and master branch

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/925163

Changed in nova:
status: New → In Progress
Changed in nova:
assignee: nobody → Masahito Muroi (muroi-masahito)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.