[api] when deploying environment, failed package validation error results in misleading exception

[Ankur Rishi]

When deploying an environment with a package that contained a wrongly named .png file for the logo which was actually a .jpeg file, the murano engine wrongly reports a NoClassFound exception.

In my case, I get the following output:

2014-05-09 15:52:13.360 14584 DEBUG muranoclient.common.http [-]
HTTP/1.1 200 OK
date: Fri, 09 May 2014 22:52:13 GMT
content-length: 427
content-type: application/json

{"packages": [{"class_definitions": ["marketplace.linux.apache.Apache"], "description": "Apache on linux\n", "tags": ["Linux"], "updated": "2014-05-09T19:38:09", "is_public": true, "categories": [], "name": "Apache", "author": "Ankur Rishi", "created": "2014-05-09T19:38:09", "enabled": true, "id": "243a7b3ec9a04119bd2328815d83333c", "fully_qualified_name": "marketplace.linux.apache", "type": "Application", "owner_id": ""}]}
 log_http_response /var/www/murano-api/.venv/local/lib/python2.7/site-packages/muranoclient/common/http.py:159
2014-05-09 15:52:41.927 14584 DEBUG muranoclient.common.http [-] curl -i -X GET -H 'X-Auth-Token: f580d4f8b2eb4348460c6ccff502ae9b' -H 'Content-Type: application/octet-stream' -H 'User-Agent: python-muranoclient' http://localhost:8082/v1/catalog/packages/243a7b3ec9a04119bd2328815d83333c/download log_curl_request /var/www/murano-api/.venv/local/lib/python2.7/site-packages/muranoclient/common/http.py:149
2014-05-09 15:52:41.958 14584 DEBUG muranoclient.common.http [-]
HTTP/1.1 200 OK
date: Fri, 09 May 2014 22:52:41 GMT
content-length: 18427
content-type: application/octet-stream
 log_http_response /var/www/murano-api/.venv/local/lib/python2.7/site-packages/muranoclient/common/http.py:159
2014-05-09 15:55:32.590 14584 ERROR muranoapi.common.engine [-] Error during task execution for tenant e5dd1875d685424b94b3891c1e59a5d2
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine Traceback (most recent call last):
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/common/engine.py", line 63, in handle_task
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine obj = exc.load(task['model'])
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/executor.py", line 237, in load
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine None, self._root_context)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/object_store.py", line 80, in load
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine obj.initialize(**value)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/murano_object.py", line 64, in initialize
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine self.set_property(property_name, property_value)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/murano_object.py", line 119, in set_property
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine self.__set_property(key, value, caller_class)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/murano_object.py", line 143, in __set_property
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine value, self, self.__context, self.__object_store, default)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/typespec.py", line 43, in validate
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine self._namespace_resolver, default)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/type_scheme.py", line 275, in __call__
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine result = self._map(data, self._spec, context)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/type_scheme.py", line 257, in _map
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine return self._map_list(data, spec, child_context)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/type_scheme.py", line 240, in _map_list
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine result.append(self._map(item, spec_item, context))
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/type_scheme.py", line 253, in _map
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine return spec.evaluate(context=child_context)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/yaql_expression.py", line 51, in evaluate
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine return self._parsed_expression.evaluate(context=context)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/.venv/local/lib/python2.7/site-packages/yaql/expressions.py", line 35, in evaluate
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine return f()
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/.venv/local/lib/python2.7/site-packages/yaql/expressions.py", line 73, in __call__
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine this = self.obj_wrapper()
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/.venv/local/lib/python2.7/site-packages/yaql/expressions.py", line 73, in __call__
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine this = self.obj_wrapper()
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/.venv/local/lib/python2.7/site-packages/yaql/expressions.py", line 100, in __call__
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine return self._try_invoke(fs, args_to_pass, context)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/.venv/local/lib/python2.7/site-packages/yaql/expressions.py", line 112, in _try_invoke
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine return func(*args_to_pass)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/type_scheme.py", line 124, in _class
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine return _class2(value, name, None)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/type_scheme.py", line 152, in _class2
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine defaults=default)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/object_store.py", line 59, in load
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine class_obj = self._class_loader.get_class(obj_type)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/dsl/class_loader.py", line 50, in get_class
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine data = self.load_definition(name)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine File "/var/www/murano-api/muranoapi/engine/package_class_loader.py", line 39, in load_definition
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine raise exceptions.NoClassFound(name)
2014-05-09 15:55:32.590 14584 TRACE muranoapi.common.engine NoClassFound: Class marketplace.linux.apache.Apache is not found

The function where the exception is caught is defined in muranoapi/engine/system/package_class_loader.py, in class PackageClassLoader:

def load_definition(self, name):
        try:
            package = self.package_loader.get_package_by_class(name)
            return package.get_class(name)
        except Exception, e:
            raise exceptions.NoClassFound(name)

Because the code is written to capture *any* exception, a misleading message is collected by the traceback above.

A closer look revels that the loading error occurs on the murano-api endpoint side, when the application package is being loaded. Specifically, the following function, for validating the logo file, results in an exception:

muranoapi/packages/application_package.py, in class ApplicationPackage:

    def _load_logo(self, validate=False):
        logo_file = self._logo or 'logo.png'
        full_path = os.path.join(self._source_directory, logo_file)
        if not os.path.isfile(full_path) and logo_file == 'logo.png':
            self._logo_cache = None
            return
        try:
            if validate:
                if imghdr.what(full_path) != 'png':
                    raise e.PackageLoadError("Logo is not in PNG format")
            with open(full_path) as stream:
                self._logo_cache = stream.read()
        except Exception as ex:
            trace = sys.exc_info()[2]
            raise e.PackageLoadError(
                "Unable to load logo: " + str(ex)), None, trace

This is, in a way, 2 bugs:

(1) The error reporting is a bit misleading in that the murano-api endpoint should return a more meaningful error which is passed back up the chain on the engine side and

(2) The validation being done on the logo file here should also take place when the package is being imported to begin with.

[Timur Nurlygayanov (tnurlygayanov)]

<tsufiev> the right solution is to validate package at the import stage
<ativelkov> It should be done, and, I think, it is done
<ativelkov> however if you import the package with manage.py, then no validation will be run
<sjmc7> i think we can fix this one
<sjmc7> at least validate the image
<ativelkov> So, what should be the correct fix for it?
<ruhe> ativelkov: i'm going to remove the import logic from manage.py soon
<ativelkov> It is validated, I believe\
<sjmc7> where we can, we should not allow invalid packages
<ativelkov> https://github.com/stackforge/murano/blob/master/murano/packages/application_package.py#L163
<ativelkov> This is the validation for PNG

[Steve McLellan (sjmc7)]

muranoclient actually does validate packages (returning 400 bad request), and the old manage.py import-package path is going away, so this particular issue is resolved.

The ClassNotFound exception is a separate problem (rolled up into engine error reporting). There's a separate issue that muranoclient cannot display error messages in https://bugs.launchpad.net/murano/+bug/1328662