Boot resource upload failed: error: length too large

[Jason Hobbs]

This is MAAS 1.7.0~beta2+bzr2849-0ubuntu1~ppa1. When I attempt to upload a generated hyperv image (1.6 G) I get an error of "length too large".

ubuntu@trusty-maas6:~$ maas local boot-resources create name=windows/win2012hvr2 architecture=amd64/generic content@=/home/ubuntu/windows-win2012hvr2-amd64-root-dd
usage: /usr/lib/python2.7/dist-packages/maascli/__main__.py [-h] COMMAND ...
/usr/lib/python2.7/dist-packages/maascli/__main__.py: error: length too large
ubuntu@trusty-maas6:~$ ls -alh /home/ubuntu/windows-win2012hvr2-amd64-root-dd
-rw-r--r-- 1 ubuntu ubuntu 1.6G Aug 31 19:40 /home/ubuntu/windows-win2012hvr2-amd64-root-dd

[Jason Hobbs (jason-hobbs)]

It's failing somewhere in encode_multipart_message() in apiclient/multipart.py

The base64 encoded size of a 1.6GB file will be over 2GB - I wonder if this is what's causing it.

There is some code in cypthon's cStringIO with a matching error and matching size:

static int
IO_cread(PyObject *self, char **output, Py_ssize_t n) {
    Py_ssize_t l;

    if (!IO__opencheck(IOOOBJECT(self))) return -1;
    assert(IOOOBJECT(self)->pos >= 0);
    assert(IOOOBJECT(self)->string_size >= 0);
    l = ((IOobject*)self)->string_size - ((IOobject*)self)->pos;
    if (n < 0 || n > l) {
        n = l;
        if (n < 0) n=0;
    }
    if (n > INT_MAX) {
        PyErr_SetString(PyExc_OverflowError,
                        "length too large");
        return -1;
    }

    *output=((IOobject*)self)->buf + ((IOobject*)self)->pos;
    ((IOobject*)self)->pos += n;
    return (int)n;
}

[Jason Hobbs (jason-hobbs)]

More detail on where it's hitting in encode_multipart_message() in apiclient/multipart.py

> # Flatten the message without headers.
> buf = BytesIO()
> generator = Generator(buf, False) # Don't mangle "^From".
> generator._write_headers = lambda self: None # Ignore.
> generator.flatten(message)

^^ Failure is happening in the generator.flatten() call.

[Jason Hobbs (jason-hobbs)]

Here's a full stacktrace:

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/dist-packages/maascli/__main__.py", line 20, in <module>
    main()
  File "/usr/lib/python2.7/dist-packages/maascli/__init__.py", line 46, in main
    options.execute(options)
  File "/usr/lib/python2.7/dist-packages/maascli/api.py", line 171, in __call__
    self.op, self.method, uri, options.data)
  File "/usr/lib/python2.7/dist-packages/maascli/api.py", line 257, in prepare_payload
    headers, body = encode_multipart_message(message)
  File "/usr/lib/python2.7/dist-packages/apiclient/multipart.py", line 132, in encode_multipart_message
    generator.flatten(message)
  File "/usr/lib/python2.7/email/generator.py", line 83, in flatten
    self._write(msg)
  File "/usr/lib/python2.7/email/generator.py", line 108, in _write
    self._dispatch(msg)
  File "/usr/lib/python2.7/email/generator.py", line 134, in _dispatch
    meth(msg)
  File "/usr/lib/python2.7/email/generator.py", line 203, in _handle_multipart
    g.flatten(part, unixfrom=False)
  File "/usr/lib/python2.7/email/generator.py", line 83, in flatten
    self._write(msg)
  File "/usr/lib/python2.7/email/generator.py", line 108, in _write
    self._dispatch(msg)
  File "/usr/lib/python2.7/email/generator.py", line 134, in _dispatch
    meth(msg)
  File "/usr/lib/python2.7/email/generator.py", line 180, in _handle_text
    self._fp.write(payload)
OverflowError: length too large

[Jason Hobbs (jason-hobbs)]

Better formatted stack trace http://paste.ubuntu.com/8200089/

[Jason Hobbs (jason-hobbs)]

Email Generator is using cStringIO's StringIO internally, which is hitting the error (n > INT_MAX) from the C code in the above comment.

[Jason Hobbs (jason-hobbs)]

This is a bug/limitation in the standard library code we're using in apiclient to generate the request - anything 1.5GB or larger isn't going to upload with the current code.

A couple of options to fix this:

1. Try to get the email.generator code fixed in upstream python - it could use BytesIO instead of StringIO, maybe.
2. Provide our own Generator that uses BytesIO instead, or performs multiple writes() that are less than 2GB each.

[Gavin Panella (allenap)]

We could also change the CLI tool to stream files into MAAS. Stream the file to a temporary on-server location, then call the API with a file token.

[Blake Rouse (blake-rouse)]

The way the boot resource model is designed, it allows the ability for a resource to exist and not have all the content. This would give us the ability to support multiple api calls to complete an upload. This was something I wanted to implement in the future, looks like the future is sooner than expected.