openstack CLI object save - MemoryError

[Dmitry Sutyagin]

Reproduced on Mirantis OpenStack 9.2 (mitaka)

Issue summary: CLI command "openstack object save ..." silently fails (error only visible with --debug).

Steps to reproduce:
- Deploy controllers with 2GB RAM (virtual lab), then:
root@node-6:/mnt/backup/swift-glance# free -m
             total used free shared buffers cached
Mem: 2000 1353 647 7 10 93
-/+ buffers/cache: 1249 751
root@node-6:/mnt/backup/swift-glance# openstack object show glance 18c0fae2-4e31-464d-9776-b3033945224e
+----------------+---------------------------------------+
| Field | Value |
+----------------+---------------------------------------+
| account | AUTH_231d9fe5c973475581cac208e6de4534 |
| container | glance |
| content-length | 341311488 |
| content-type | application/octet-stream |
| etag | 3209808b650928a389f4c94a67f5f3a9 |
| last-modified | Wed, 08 Feb 2017 00:38:16 GMT |
| object | 18c0fae2-4e31-464d-9776-b3033945224e |
+----------------+---------------------------------------+
root@node-6:/mnt/backup/swift-glance# openstack --debug object save glance 18c0fae2-4e31-464d-9776-b3033945224e
...
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/common/command.py", line 38, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/object/v1/object.py", line 204, in take_action
    file=parsed_args.file,
  File "/usr/lib/python2.7/dist-packages/openstackclient/api/object_store_v1.py", line 369, in object_save
    stream=True,
  File "/usr/lib/python2.7/dist-packages/openstackclient/api/api.py", line 83, in _request
    return session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/openstackclient/common/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 452, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 517, in _send_request
    self._http_log_response(response=resp, logger=logger)
  File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 263, in _http_log_response
    text = self._remove_service_catalog(response.text)
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 769, in text
    encoding = self.apparent_encoding
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 643, in apparent_encoding
    return chardet.detect(self.content)['encoding']
  File "/usr/lib/python2.7/dist-packages/chardet/__init__.py", line 24, in detect
    u.feed(aBuf)
  File "/usr/lib/python2.7/dist-packages/chardet/universaldetector.py", line 115, in feed
    if prober.feed(aBuf) == constants.eFoundIt:
  File "/usr/lib/python2.7/dist-packages/chardet/charsetgroupprober.py", line 59, in feed
    st = prober.feed(aBuf)
  File "/usr/lib/python2.7/dist-packages/chardet/sjisprober.py", line 53, in feed
    for i in range(0, aLen):
MemoryError

Expected results: openstack client successfully downloads a 300MB file from Swift.

Actual result: silent failure (error visible only if --debug is specified).

Reproducibility: 100% on this env.

Impact: cannot use openstack object CLI.

[Dmitry Sutyagin (dsutyagin)]

downstream MOS bug - https://bugs.launchpad.net/mos/+bug/1663771

[John Dickinson (notmyname)]

Can you please try the download with curl and/or python-swiftclient? If those work, then it shows that the error is likely with the openstackclient project, and we can tag that appropriately.

[Dmitry Sutyagin (dsutyagin)]

@John Dickinson
downloading via "swift download glance" works fine (had to switch auth URL to v2.0 though".

P.S. I have another interesting phenomena now:
root@node-6:~/test# swift list
glance
root@node-6:~/test# openstack container list
Service Unavailable (HTTP 503)

"swift list" works while "openstack container" fails with 503 ("openstack object list glance" also fails with 503). Anyway, that's another story.

[Vitalii Gridnev (vgridnev)]

I think that this issue is about https://bugs.launchpad.net/python-keystoneclient/+bug/1616105

[Vitalii Gridnev (vgridnev)]

See ^^, this issue is already fixed in the upstream

[Dmitry Sutyagin (dsutyagin)]

Ok, so maybe this error with --debug is different from the issue without --debug

Without debug the CLI exits silently without downloading the object, which is why I added --debug in the first place. If I hit that issue again I will create a new bug I guess.