UnicodeDecodeError when uploading a file and other fields in form don't validate

[Chris]

I found when I create a form and the form validates correctly the file
passed will upload fine. However if for example the e-mail field doesn't validate I get the
following error:

2010-04-07 14:55:59: (mod_fastcgi.c.2610) FastCGI-stderr: Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/web/application.py", line 242, in process
    return self.handle()
  File "/usr/lib/python2.5/site-packages/web/application.py", line 233, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.5/site-packages/web/application.py", line 415, in _delegate
    return handle_class(cls)
  File "/usr/lib/python2.5/site-packages/web/application.py", line 390, in handle_class
    return tocall(*args)
  File "/var/www/webapp/webapp.py", line 36, in POST
    return render.upload(f)
  File "/usr/lib/python2.5/site-packages/web/template.py", line 1024, in template
    return self._base(t(*a, **kw))
  File "/usr/lib/python2.5/site-packages/web/template.py", line 898, in __call__
    return BaseTemplate.__call__(self, *a, **kw)
  File "/usr/lib/python2.5/site-packages/web/template.py", line 804, in __call__
    return self.t(*a, **kw)
  File "templates/upload.html", line 6, in __template__
  File "/usr/lib/python2.5/site-packages/web/template.py", line 821, in _escape
    value = safeunicode(value)
  File "/usr/lib/python2.5/site-packages/web/utils.py", line 326, in safeunicode
    return obj.decode(encoding)
  File "/usr/lib/python2.5/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 321-322: invalid data

-----------------------------------------
Code snippet:

vpass = form.regexp(r".{3,20}$", 'must be between 3 and 20 characters')
vemail = form.regexp(r".*@.*", "must be a valid email address")

upload_form = form.Form(
    form.Textbox("email", vemail, description="E-Mail"),
    form.File("page", description="JPEG image of handwriting"),
    form.Button("submit", type="submit", description="Register")
)

class upload:

    def GET(self):
        f = upload_form()
        return render.upload(f)

    def POST(self):

        f = upload_form()
        if not f.validates():
                return render.upload(f)
        else:
                p = web.input(_unicode=False,page={})

[LKRaider (paul-eipper)]

I have a similar issue, but no validation is required, just a simple form with one File field:

plugin_form = web.form.Form(
        web.form.File('plugin_file', web.form.notnull),
)

upon uploading a file from this form, this is the error:

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/web/application.py", line 209, in process
    return p(lambda: process(processors))
  File "/usr/lib/pymodules/python2.6/web/application.py", line 537, in processor
    return handler()
  File "/usr/lib/pymodules/python2.6/web/application.py", line 209, in <lambda>
    return p(lambda: process(processors))
  File "/usr/lib/pymodules/python2.6/web/application.py", line 216, in process
    raise self.internalerror()
  File "/usr/lib/pymodules/python2.6/web/application.py", line 449, in internalerror
    return debugerror.debugerror()
  File "/usr/lib/pymodules/python2.6/web/debugerror.py", line 299, in debugerror
    return web._InternalError(djangoerror())
  File "/usr/lib/pymodules/python2.6/web/debugerror.py", line 289, in djangoerror
    return t(exception_type, exception_value, frames)
  File "/usr/lib/pymodules/python2.6/web/template.py", line 850, in __call__
    return BaseTemplate.__call__(self, *a, **kw)
  File "/usr/lib/pymodules/python2.6/web/template.py", line 768, in __call__
    return self._join_output(out)
  File "/usr/lib/pymodules/python2.6/web/template.py", line 774, in _join_output
    for name, value in out:
  File "/usr/lib/pymodules/python2.6/web/debugerror.pyc", line 183, in __template__
    $if ctx.output or ctx.headers:
  File "/usr/lib/pymodules/python2.6/web/webapi.py", line 222, in input
    return storify(out, *requireds, **defaults)
  File "/usr/lib/pymodules/python2.6/web/utils.py", line 141, in storify
    value = getvalue(value)
  File "/usr/lib/pymodules/python2.6/web/utils.py", line 128, in getvalue
    return unicodify(x.value)
  File "/usr/lib/pymodules/python2.6/web/utils.py", line 123, in unicodify
    if _unicode and isinstance(s, str): return safeunicode(s)
  File "/usr/lib/pymodules/python2.6/web/utils.py", line 231, in safeunicode
    return obj.decode(encoding)
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 12-13: invalid data

I don't know what it is trying to decode, the uploaded binary data maybe?

[LKRaider (paul-eipper)]

Correction: it was the validation

debugging before and after the line 221 on webapi.py:

       defaults.setdefault('_unicode', True)

the contents of the "defaults" dict varies (1 is before, 2 is after that line):

1 {'_unicode': False, 'myfile': {}} -> ['myfile'] 992166
2 {'_unicode': False, 'myfile': {}} -> ['myfile'] 992166
1 {} -> ['myfile'] 992166
2 {'_unicode': True} -> ['myfile'] 992166

This is the output for 1 upload POST request, running through validation and accessing it later. It seems the contents are emptied somehow, and then the default _unicode value is applied to the uploaded file, which causes the break.

[LKRaider (paul-eipper)]

Oh, and turning off validation and just accessing the input makes it work, that part of the code is then run only once:

1 {'_unicode': False, 'myfile': {}} -> ['myfile'] 992166
2 {'_unicode': False, 'myfile': {}} -> ['myfile'] 992166

(the debug is printing the defaults dict and the size of the upload)