Fails with "BadStatusLine: ''"

Bug #1580745 reported by Francis Ginther on 2016-05-11
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance - Simplestreams Sync Charm
High
Данило Шеган
Landscape Server
High
Данило Шеган
16.06
High
Данило Шеган

Bug Description

There are several Tracebacks in this log file, but the last one caused the sync to give up. This resulted in a failed openstack autopilot deployment.

Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 444, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 215, in do_sync
    tmirror.sync(smirror, path=initial_path)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
    self.sync(reader, path=epath)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 341, in sync_products
    self.insert_item(item, src, target, pgree, ipath_cs)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 303, in insert_item
    ret = self.gclient.images.create(**create_kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 253, in create
    'POST', '/v1/images', headers=hdrs, body=image_data)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 289, in raw_request
    return self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 226, in _http_request
    resp = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 415, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

The full log is attached.

This was part of a Landscape OSA deployment using Landscape version 16.05~bzr10061~release43+jenkins120-1.

Related branches

Revision history for this message
Francis Ginther (fginther) wrote :
tags: added: landscape
description: updated
Revision history for this message
Francis Ginther (fginther) wrote :

Here's another failure from a landscape OSA deployment. Both this and the original failure in this bug are from the same maas cluster.

Revision history for this message
Данило Шеган (danilo) wrote :

glance-simplestreams-sync charm handles a bunch of "expected" exceptions, since they are thrown when all of the dependencies are not fully set up yet: it is designed to retry in such cases.

The core question here is if this was something that would have resolved itself with time, when it would have made sense to retry, or not. Do you perhaps have rest of the logs for this cloud deployment? Can you check the state of ceph-radosgw (unit state, logs in /var/log/ceph/radosgw.log, is radosgw service even running)? If you've still got the cloud up, a manual run after the issue has been resolved should then work:

  JUJU_HOME=/var/lib/landscape/juju-homes/1 juju ssh glance-simplestreams-sync/0 sudo /etc/cron.daily/glance-simplestreams-sync

I was going to make the script ignore and retry on all errors (since we now retry on a number of them: Chad just fixed another one), but if this one in particular will never lead to a success without manual intervention, there is no sense in retrying, and the problem is somewhere else.

Revision history for this message
Данило Шеган (danilo) wrote :

FWIW, in my test, I get exactly this error if I stop ceph-radosgw process. If this deployment is using swift, then maybe swift is not up.

Revision history for this message
Данило Шеган (danilo) wrote :

We've confirmed that ceph-radosgw was not up and running when Francis hit this. A separate bug will be filed for that since ceph-radosgw 10.2.0 was just pushed up yesterday.

Glance simplestreams sync charm can't do anything about that though, and it's correct to fatally fail. The attached branch might still be useful for other cases, but I won't push it for 16.05 and will get it landed into trunk later.

Changed in glance-simplestreams-sync-charm:
status: New → Invalid
Changed in landscape:
status: New → Invalid
Changed in landscape:
status: Invalid → Triaged
Changed in glance-simplestreams-sync-charm:
status: Invalid → In Progress
assignee: nobody → Данило Шеган (danilo)
importance: Undecided → High
Changed in landscape:
importance: Undecided → High
Revision history for this message
Данило Шеган (danilo) wrote :

The first case here was not really something that would have fixed itself, but Francis hit a case where a retry would have helped: https://bugs.launchpad.net/simplestreams/+bug/1583276/comments/10

Changed in glance-simplestreams-sync-charm:
status: In Progress → Fix Committed
Changed in landscape:
assignee: nobody → Данило Шеган (danilo)
status: Triaged → In Progress
Changed in landscape:
milestone: none → 16.06
status: In Progress → Fix Committed
Changed in landscape:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers