overcloud deploy errors with 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte from swiftclient

[Alex Schultz]

Running a deploy under python3 errors.

openstack --debug overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml -e ~/containers-prepare-parameter.yaml

When you run with --debug it shows the error being thrown from deep within the swiftclient.

CMD "/usr/bin/tar -C /tmp/tripleoclient-l1_7_rma/tripleo-heat-templates -czf /tmp/tmpgq5r53me --exclude .git --exclude .tox --exclude *.pyc --exclude *.pyo ." returned: 0 in 0.092s
Uploading filename /tmp/tmpgq5r53me to Swift container overcloud
Starting new HTTPS connection (1): 192.168.24.2
'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 867, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 358, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 389, in _deploy_tripleo_heat_templates
    parsed_args.networks_file)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 164, in create_plan_from_templates
    plan_env_file, networks_file)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 41, in _upload_templates
    swift_client, tmp_tarball.name, container_name)
  File "/usr/lib/python3.6/site-packages/tripleo_common/utils/tarball.py", line 44, in tarball_extract_to_swift_container
    headers={'X-Detect-Content-Type': 'true'}
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1878, in put_object
    response_dict=response_dict)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1722, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1346, in put_object
    conn.putrequest(path, headers=headers, data=data)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 454, in putrequest
    return self.request('PUT', full_path, data, headers, files)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 444, in request
    files=files, **self.requests_args)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 427, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 460, in send
    for i in request.body:
  File "/usr/lib/python3.6/site-packages/swiftclient/utils.py", line 298, in __next__
    chunk = self.content.read(self.chunk_size)
  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
clean_up DeployOvercloud: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3.6/site-packages/osc_lib/shell.py", line 175, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python3.6/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.6/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 867, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 358, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 389, in _deploy_tripleo_heat_templates
    parsed_args.networks_file)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 164, in create_plan_from_templates
    plan_env_file, networks_file)
  File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/plan_management.py", line 41, in _upload_templates
    swift_client, tmp_tarball.name, container_name)
  File "/usr/lib/python3.6/site-packages/tripleo_common/utils/tarball.py", line 44, in tarball_extract_to_swift_container
    headers={'X-Detect-Content-Type': 'true'}
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1878, in put_object
    response_dict=response_dict)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1722, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1346, in put_object
    conn.putrequest(path, headers=headers, data=data)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 454, in putrequest
    return self.request('PUT', full_path, data, headers, files)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 444, in request
    files=files, **self.requests_args)
  File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 427, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 460, in send
    for i in request.body:
  File "/usr/lib/python3.6/site-packages/swiftclient/utils.py", line 298, in __next__
    chunk = self.content.read(self.chunk_size)
  File "/usr/lib64/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

END return value: 1

[Alex Schultz (alex-schultz)]

Seems to be blowing up in https://github.com/openstack/python-swiftclient/blob/master/swiftclient/utils.py#L267 which has references to python3 and binary mode which would make sense why it's complaing about not being able to be decoded.

[Alex Schultz (alex-schultz)]

turns out it's in tripleo_common[0]. We were opening the tarball using Text I/O instead of Binary I/O[1].

[0] https://github.com/openstack/tripleo-common/blob/master/tripleo_common/utils/tarball.py#L38
[1] https://docs.python.org/3/library/io.html#text-i-o

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/594560

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/594560
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=714145c563e82ce4f5f2610950baba39df892d88
Submitter: Zuul
Branch: master

commit 714145c563e82ce4f5f2610950baba39df892d88
Author: Alex Schultz <email address hidden>
Date: Tue Aug 21 16:16:57 2018 -0600

    Open tarball in binary mode

    Under python3 this open call fails in swiftclient because it tries to
    decode the data using utf-8. Since we're uploading a binary tarball
    anyway we should us binary i/o instead.

    Change-Id: I5b8aff53e44024e916b9a491e52ffd9362ca39e4
    Closes-Bug: #1788288

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 9.3.0 release.