Fails with "BadStatusLine: ''"

[Francis Ginther]

There are several Tracebacks in this log file, but the last one caused the sync to give up. This resulted in a failed openstack autopilot deployment.

Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 444, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 215, in do_sync
    tmirror.sync(smirror, path=initial_path)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
    self.sync(reader, path=epath)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 341, in sync_products
    self.insert_item(item, src, target, pgree, ipath_cs)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 303, in insert_item
    ret = self.gclient.images.create(**create_kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 253, in create
    'POST', '/v1/images', headers=hdrs, body=image_data)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 289, in raw_request
    return self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 226, in _http_request
    resp = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 415, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

The full log is attached.

This was part of a Landscape OSA deployment using Landscape version 16.05~bzr10061~release43+jenkins120-1.

[Francis Ginther (fginther)]

Here's another failure from a landscape OSA deployment. Both this and the original failure in this bug are from the same maas cluster.

[Данило Шеган (danilo)]

glance-simplestreams-sync charm handles a bunch of "expected" exceptions, since they are thrown when all of the dependencies are not fully set up yet: it is designed to retry in such cases.

The core question here is if this was something that would have resolved itself with time, when it would have made sense to retry, or not. Do you perhaps have rest of the logs for this cloud deployment? Can you check the state of ceph-radosgw (unit state, logs in /var/log/ceph/radosgw.log, is radosgw service even running)? If you've still got the cloud up, a manual run after the issue has been resolved should then work:

  JUJU_HOME=/var/lib/landscape/juju-homes/1 juju ssh glance-simplestreams-sync/0 sudo /etc/cron.daily/glance-simplestreams-sync

I was going to make the script ignore and retry on all errors (since we now retry on a number of them: Chad just fixed another one), but if this one in particular will never lead to a success without manual intervention, there is no sense in retrying, and the problem is somewhere else.

[Данило Шеган (danilo)]

FWIW, in my test, I get exactly this error if I stop ceph-radosgw process. If this deployment is using swift, then maybe swift is not up.

[Данило Шеган (danilo)]

We've confirmed that ceph-radosgw was not up and running when Francis hit this. A separate bug will be filed for that since ceph-radosgw 10.2.0 was just pushed up yesterday.

Glance simplestreams sync charm can't do anything about that though, and it's correct to fatally fail. The attached branch might still be useful for other cases, but I won't push it for 16.05 and will get it landed into trunk later.

[Данило Шеган (danilo)]

The first case here was not really something that would have fixed itself, but Francis hit a case where a retry would have helped: https://bugs.launchpad.net/simplestreams/+bug/1583276/comments/10