Nova fails to boot from volume

[Dirk Mueller]

When booting from volume via Horizon webui, the block device mapping that is passed down to nova contains "volume_size": "", which causes a DB integrity error later:

2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack Traceback (most recent call last):
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/api/openstack/__init__.py", line 81, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return req.get_response(self.application)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/webob/request.py", line 1296, in send
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack application, catch_exc_info=False)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/webob/request.py", line 1260, in call_application
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/webob/dec.py", line 144, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return resp(environ, start_response)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 457, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return self.app(env, start_response)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/webob/dec.py", line 144, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return resp(environ, start_response)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/webob/dec.py", line 144, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return resp(environ, start_response)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/routes/middleware.py", line 131, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/webob/dec.py", line 144, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return resp(environ, start_response)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/webob/dec.py", line 130, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/webob/dec.py", line 195, in call_func
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/api/openstack/wsgi.py", line 890, in __call__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack content_type, body, accept)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/api/openstack/wsgi.py", line 942, in _process_stack
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1022, in dispatch
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return method(req=request, **action_args)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/api/openstack/compute/servers.py", line 898, in create
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack scheduler_hints=scheduler_hints)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/hooks.py", line 85, in inner
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack rv = f(*args, **kwargs)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/compute/api.py", line 961, in create
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack scheduler_hints=scheduler_hints)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/compute/api.py", line 675, in _create_instance
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack reservation_id, scheduler_hints)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/compute/api.py", line 633, in _validate_and_provision_instance
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack QUOTAS.rollback(context, quota_reservations)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack self.gen.next()
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/compute/api.py", line 615, in _validate_and_provision_instance
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack num_instances, i)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/compute/api.py", line 910, in create_db_entry_for_new_instance
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack instance_type, image, block_device_mapping)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/compute/api.py", line 806, in _populate_instance_for_bdm
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack instance_type, instance_uuid, mapping)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/compute/api.py", line 772, in _update_block_device_mapping
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack values)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/db/api.py", line 1032, in block_device_mapping_update_or_create
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return IMPL.block_device_mapping_update_or_create(context, values)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 113, in wrapper
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack return f(*args, **kwargs)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2962, in block_device_mapping_update_or_create
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack bdm_ref.save(session=session)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/models.py", line 54, in save
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack session.flush()
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack File "/usr/lib64/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 437, in _wrap
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack raise exception.DBError(e)
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack DBError: (DataError) invalid input syntax for integer: ""
2013-07-10 12:54:03.089 4920 TRACE nova.api.openstack LINE 1: ...LL, NULL, '6d6ad4f0-2ce9-49ab-a94f-71fc1b74ec7f', '', NULL, ...

the volume_size parameter should be validated to be usefol. for compatibility, the empty volume_size parameter should be ignored, as in the case of boot from volume it is not used anyway.

[Matt Riedemann (mriedem)]

I'm not sure if this should be routed to Horizon to validate this or as something to validate in the API, but it is failing on bad data, so to fix, just don't use the bad data (which is one way to resolve this bug). The thing about this is I'm not sure what precedent there is for validating user input on data types in the UI (or via CLI to the APIs for that matter).

[Matt Riedemann (mriedem)]

I sent a note to the developer mailing list to get some opinions on how to handle this.

[Gabriel Hurley (gabriel-hurley)]

This should absolutely be caught before it raises a database error. A useful, human-friendly error message should be returned via the API. Obviously anything using the API (such as Horizon) should do its best to pre-validate the input, but if invalid input is sent it should be handled well. The best way to let Horizon devs know what the problem is is for the API to return an intelligent failure.

[Dirk Mueller (dmllr)]

It is not an error in Horizon, Horizon is just triggering the problem by using Novaclient. In the case of Boot from Volume the Volume Size is not used (it is by definition the size of the volume), so Horizon is correct in not passing it down Novaclient.

https://review.openstack.org/#/c/34769/

https://review.openstack.org/#/c/34749/

[Sean Dague (sdague)]

Is this actually a novaclient problem, or is it a nova problem? I've seen these kinds of issues before when we make assumptions about how mysql does type conversions.

[David Ripton (dripton)]

The server-side error is #1171190, which I fixed in 188a94c898bcf0, which was merged to Havana mainline on May 10. I think it was only a problem in PostgreSQL, which is more strongly typed than MySQL or SQLite.

So I think this is a duplicate.

[David Ripton (dripton)]

I don't think we need to fix anything in novaclient for this. The bug was on the server side.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/34749
Committed: http://github.com/openstack/nova/commit/3ff667620c2d735536b26c024691dfb9193975a2
Submitter: Jenkins
Branch: master

commit 3ff667620c2d735536b26c024691dfb9193975a2
Author: Dirk Mueller <email address hidden>
Date: Thu Jun 27 17:23:41 2013 +0200

    Validate volume_size in block_device_mapping

    Reject invalid values for volume_size upfront.
    For compatibility with old python-novaclient, an
    empty volume_size is ignored.

    Create a common _validate_int_value() function for
    validating that a parameter is an integer in a given
    range. Use it also for min_count/max_count to share code.

    Fixes LP Bug#1199539

    Change-Id: I18ac285d5b3c8200d7706c7d577809b15d6ab9e8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/34769
Committed: http://github.com/openstack/python-novaclient/commit/f50ff361e27a8ca688c0f1ba448bbc8bfb284905
Submitter: Jenkins
Branch: master

commit f50ff361e27a8ca688c0f1ba448bbc8bfb284905
Author: Dirk Mueller <email address hidden>
Date: Thu Jun 27 19:13:01 2013 +0200

    Skip setting volume_size if not given

    When the block-device parameters skip volume_size,
    don't set it. Setting to an empty volume_size
    would be invalid as it has to be an integer,
    and Nova API will reject the request if api validation
    is implemented. (proposed e.g. at
    https://review.openstack.org/#/c/34749/)

    Fixes bug LP #1199539

    Change-Id: I7ab518886abf8d449caf1c70563a79a990d7765e