tempest cleanup fails on py35

[Jay Pipes]

Not entirely sure what's going on, but `tempest cleanup` is broken after doing an initial `tempest cleanup --init-saved-state` successfully...

(venv) :~/tempest$ tempest cleanup
2018-11-30 16:57:05.950 12641 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf
2018-11-30 16:57:06.720 12641 INFO tempest.lib.common.rest_client [req-cea1d96d-4dbc-4a6e-892b-aaf6b95bad23 ] Request (main): 201 POST http://localhost:5000/v3/auth/tokens
2018-11-30 16:57:07.134 12641 INFO tempest.lib.common.rest_client [req-b3dc7afb-f16d-4685-afc8-8df07802fde0 ] Request (main): 201 POST http://localhost:5000/v3/auth/tokens
2018-11-30 16:57:21.713 12641 INFO tempest.lib.common.rest_client [req-313bc14d-a057-450c-bc2e-a9080a941d33 ] Request (main): 200 GET http://bearfair.corp.ne1.yahoo.com:9696/v2.0/networks 14.577s
2018-11-30 16:57:22.157 12641 INFO tempest.lib.common.rest_client [req-2841f850-13c8-4b41-8413-2d87f95249e2 ] Request (main): 200 GET http://bearfair.corp.ne1.yahoo.com:35357/v3/projects?name=admin 0.361s
2018-11-30 16:57:22.675 12641 INFO tempest.lib.common.rest_client [req-5853c881-632b-4afa-bf75-2aa4cf603081 ] Request (main): 201 POST http://localhost:5000/v3/auth/tokens
2018-11-30 16:57:23.118 12641 INFO tempest.lib.common.rest_client [req-8c7a1f4b-0f3e-4fab-8db0-ddc36249d645 ] Request (main): 201 POST http://localhost:5000/v3/auth/tokens
2018-11-30 16:57:44.242 12641 INFO tempest.lib.common.rest_client [req-89ba37ab-4895-4ab8-9044-a0b64477225b ] Request (main): 200 GET http://bearfair.corp.ne1.yahoo.com:35357/v3/projects?name=admin 21.122s
2018-11-30 16:57:44.797 12641 INFO tempest.lib.common.rest_client [req-cb52cd9b-04bb-4f9c-88dd-314d299f5b43 ] Request (main): 200 GET http://bearfair.corp.ne1.yahoo.com:35357/v3/users?name=bootstrap_admin 0.547s
2018-11-30 16:57:45.059 12641 INFO tempest.lib.common.rest_client [req-e0f85649-99b4-4fa3-b61d-2f36fee10237 ] Request (main): 200 GET http://bearfair.corp.ne1.yahoo.com:35357/v3/role_assignments?scope.project.id=a789d2d4ef6144df88749fcdf4804472 0.260s
2018-11-30 16:57:45.337 12641 INFO tempest.lib.common.rest_client [req-70a82005-617a-4a44-b6ec-f6c512312ff7 ] Request (main): 200 GET http://bearfair.corp.ne1.yahoo.com:35357/v3/roles 0.274s
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup [-] Exception parsing saved state json : can't concat str to bytes: TypeError: can't concat str to bytes
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup Traceback (most recent call last):
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup File "/home/jpipes/tempest/venv/lib/python3.5/site-packages/tempest/cmd/cleanup.py", line 315, in _load_json
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup self.json_data = json.load(json_file)
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup File "/home/jpipes/tempest/venv/lib/python3.5/site-packages/oslo_serialization/jsonutils.py", line 276, in load
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup return json.load(codecs.getreader(encoding)(fp), **kwargs)
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup File "/opt/python/lib/python3.5/json/__init__.py", line 265, in load
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup return loads(fp.read(),
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup File "/home/jpipes/tempest/venv/lib/python3.5/codecs.py", line 497, in read
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup data = self.bytebuffer + newdata
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup TypeError: can't concat str to bytes
2018-11-30 16:57:45.340 12641 ERROR tempest.cmd.cleanup
can't concat str to bytes

[Martin Kopec (mkopec)]

I've reproduced this issue on packstack rocky deployment:
1. clone tempest master branch
2. Install tox python3 tox environment and source it:
  $ tox -epy36
  $ source .tox/py36/bin/activate
3. Create saved_state.json and run cleanup:
  $ tempest cleanup --init-saved-state
  $ tempest cleanup
<the traceback as in the comment above>

The issue seems to be in python3 codecs library which expects to have new data in byte format not in string one like it is in python2.
I've found the same problem described on stackoverflow:
https://stackoverflow.com/questions/45037676/python-3-6-urllib-typeerror-cant-concat-bytes-to-str

So I guess, we just need to change the mode for reading the saved_state.json file.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/631982

[Martin Kopec (mkopec)]

Hmm, my proposed solution may be wrong, I did a small experiment. I created the following script:
```
$ cat script.py

import json
import sys

with open(sys.argv[1]) as json_file:
    json_data = json.load(json_file)
    print(json_data)
```

The script should mock the tempest cleanup call which currently fails. When I execute the script by using python3.6, it works, but running tempest cleanup by the same python3.6 leads to the traceback .. that's very strange. Also I noticed that a different codecs file is used.
It could mean that something is overridden/hardcoded in tempest .. I need to investigate that more.

[Martin Kopec (mkopec)]

Let's say, I have the following script and I made the changes in Tempest showed below:

```
$ cat script.py

import json
import sys

def func(f):
    with open(f) as json_file:
        json_data = json.load(json_file)
        print(json_data)

if __name__ == '__main__':
    func(sys.argv[1])

$ git diff
diff --git a/tempest/cmd/cleanup.py b/tempest/cmd/cleanup.py
index 2f54f9a..f71b2b0 100644
--- a/tempest/cmd/cleanup.py
+++ b/tempest/cmd/cleanup.py
@@ -311,7 +311,11 @@ class TempestCleanup(command.Command):

     def _load_json(self):
         try:
             with open(SAVED_STATE_JSON) as json_file:
+ import pdb; pdb.set_trace()
+ import script
+ script.func(SAVED_STATE_JSON)
                 self.json_data = json.load(json_file)

         except IOError as ex:

$ tempest cleanup
> /home/centos/tempest/tempest/cmd/cleanup.py(317)_load_json()
-> import script
(Pdb) n
> /home/centos/tempest/tempest/cmd/cleanup.py(318)_load_json()
-> script.func(SAVED_STATE_JSON)
(Pdb) n
{'domains': {'default': 'Default'}, 'flavors': {'1': 'm1.tiny', '161c6e9a-a5a0-4bd2-abb2-eb4a19a29afc': 'm1.micro', '2': 'm1.small', '3': 'm1.medium', '4': 'm1.large', '5': 'm1.xlarge', '79388ef4-b963-4057-8d11-a0ce4b156e1f': 'm1.nano'}, 'images': {}, 'projects': {'268bcb63488b4aa2942ecaac0f85ed62': 'demo', 'c40e248acd22492abf4ca0cfad13aca3': 'admin', 'f2f2bb73de1e4edf9c465249cb000e2e': 'services', 'f81c157b99184c53a08c42edcf25664e': 'alt_demo'}, 'roles': {}, 'users': {'023e65a5922a454585a91c6af8310968': 'demo', '26b295116d354bd7b6d6534a874788b7': 'nova', '2bca7128607946c38b61e1555dc0632a': 'placement', '405bb53f68c84841b454cf31ecd6fc62': 'ceilometer', '4d7d436275204d119bdb5b1053add033': 'glance', '7ec1aac2c02e4653a920b4e28c64c166': 'swift', '9092e14ceb0e432a9ef48723e4a980ec': 'neutron', 'ae1e668149c4445ab837a23fefe1ec30': 'admin', 'bd2782e336b44e019a62cb8f56a19adb': 'gnocchi', 'f334a6c7949b4ceaaf5d78e2e7240950': 'aodh', 'f3e9262159d247d5956b9f82d29d6543': 'cinder', 'f4952230adb949d5a4140e87b51764d5': 'alt_demo'}}
> /home/centos/tempest/tempest/cmd/cleanup.py(319)_load_json()
-> self.json_data = json.load(json_file)
(Pdb) n
TypeError: can't concat str to bytes

```

I can't explain why `json.load(json_file)` in the script works, but the one other one no. Hmmmmm

[Martin Kopec (mkopec)]

oh, clenaup.py uses json from jsonutils from oslo_serialization:
https://github.com/openstack/tempest/blob/d47a79c2b54848d101ade3b753e547b49158e5ce/tempest/cmd/cleanup.py#L81

here's the commit:
https://github.com/openstack/tempest/commit/2190551738eb0672b52d0ed324b2ca1476b6f819

Now need to check it the description in the commit message is still true.

[Martin Kopec (mkopec)]

https://bugs.launchpad.net/oslo.serialization/+bug/1515231

[Martin Kopec (mkopec)]

Based on the following comment, it seems, that if we opened the file in a byte mode (as the review suggests), it would solve our issue.
https://bugs.launchpad.net/oslo.serialization/+bug/1515231/comments/2

[Martin Kopec (mkopec)]

For now it can be solved by the opening file in the binary mode, but when we fully change to python3, I would omit the oslo.serialization library and would use json directly.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631982
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=6caf3fabbbded0790142edfd51dc693166d669c2
Submitter: Zuul
Branch: master

commit 6caf3fabbbded0790142edfd51dc693166d669c2
Author: Martin Kopec <email address hidden>
Date: Sun Jan 20 15:24:10 2019 +0000

    Fix tempest cleanup for python3

    When using python3 the file which is read and
    loaded by jsonutils needs to be opened in binary
    mode, otherwise, it's causing problems.
    It is a known bug [1] in jsonutils from
    oslo.serialization module.
    The patch also adds a unit test which tests
    the loading of a json file.

    [1] https://bugs.launchpad.net/oslo.serialization/+bug/1515231

    Closes-Bug: #1806089
    Change-Id: I3db71c6662d9137a4b75381de7d6dbddd3de4e09

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 20.0.0 release.