API client blocks when deleting a resource

Bug #1313556 reported by Raphaël Badin on 2014-04-28
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Critical
Jeroen T. Vermeulen
1.5
Critical
Jeroen T. Vermeulen
maas (Ubuntu)
Undecided
Unassigned
Trusty
Undecided
Unassigned

Bug Description

When using the API client to delete a resource, the resource is deleted all right but the client blocks forever.

[Test Case]
= How to reproduce =

Assuming you've got a MAAS server installed on localhost:

# Create admin user.
sudo maas-region-admin createadmin --username admin --password test --email <email address hidden>
# Set up maas cli.
sudo maas login maas http://localhost/MAAS/api/1.0/ `sudo maas-region-admin apikey --username admin`
# Create a zone object.
sudo maas maas zones create name=test_zone
# Get the api key for the admin user.
sudo maas-region-admin apikey --username admin
<grab the API key>

# Now, in a python shell:
python

Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from apiclient.creds import convert_string_to_tuple
>>> from apiclient.maas_client import (
... MAASClient,
... MAASDispatcher,
... MAASOAuth,
... urllib2,
... )
>>> credentials = convert_string_to_tuple('<Replace with the API key>')
>>> auth = MAASOAuth(*credentials)
>>> client = MAASClient(auth, MAASDispatcher(), "http://localhost/MAAS")
>>> zone_uri = u'/api/1.0/zones/test_zone/'
>>> client.delete(zone_uri)
<blocked forever>

= Notes =

- The same happens when deleting a node so I suspect this is a general problem with the `delete()` method.
- If you CTL-C the last command (the deletion of the zone), `sudo maas maas zones read` will show that the zone has effectively be deleted.

= Workaround =

A (pretty ugly) workaround is to set a timeout to `urlopen` used by the API client.

>>> from functools import partial
>>> urlopen_with_timeout = partial(urllib2.urlopen, timeout=4)
>>> from apiclient import maas_client
>>> maas_client.urllib2.urlopen = urlopen_with_timeout

Related branches

Raphaël Badin (rvb) on 2014-04-28
description: updated
Jeroen T. Vermeulen (jtv) wrote :

All of the API's DELETE implementations return piston.utils.rc.DELETED: status code 204 (“No Content”) and an empty body. It sounds as if the error is in that area somewhere... What should happen AFAICS is that Django (in conditional_content_removal) strips the empty text body from the response, but as Raphaël has pointed out elsewhere, perhaps that leaves a file-like object unclosed or something.

Raphaël Badin (rvb) on 2014-04-28
Changed in maas:
importance: Undecided → Critical
status: New → Triaged
Jeroen T. Vermeulen (jtv) wrote :

There doesn't seem to be anything special about the response: all DELETE methods currently return 204: No Content. But returning OK with a body, or even an error with a body, produces the same problem.

This could be a result of a limitation of urllib2: its Request class assumes that all requests are either GET or POST. Our client code splices in support for other methods, but that may not be enough.

To help further debugging, here's a traceback from a blocked client:

  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 244, in delete
    url, method="DELETE", headers=headers)
  File "/usr/lib/python2.7/dist-packages/apiclient/maas_client.py", line 113, in dispatch_query
    res = urllib2.urlopen(req)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1187, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)

Changed in maas:
assignee: nobody → Jeroen T. Vermeulen (jtv)
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
Jeroen T. Vermeulen (jtv) wrote :

Fix has landed in trunk. Thanks for an _excellent_ bug report. The easy recipe for reproducing the problem made it much easier, and much more attractive, to fix.

Changed in maas:
milestone: none → 14.10
Changed in maas:
milestone: 14.10 → none
description: updated
Changed in maas:
status: Fix Committed → Fix Released
Adam Conrad (adconrad) on 2014-05-09
Changed in maas (Ubuntu):
status: New → Fix Released

Hello Raphaël, or anyone else affected,

Accepted maas into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/maas/1.5.1+bzr2269-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in maas (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package maas - 1.5.1+bzr2269-0ubuntu0.1

---------------
maas (1.5.1+bzr2269-0ubuntu0.1) trusty; urgency=medium

  * Stable Release Update (LP: #1317601):
    - Hardware Enablement for Cisco B-Series. (LP: #1300476)
    - Allow AMT power type to specify IP Address. (LP: #1308772)
    - Spurious failure when starting and creating lock files. (LP: 1308069)
    - Fix usage of hardware enablement kernels by fixing the preseeds
      (LP: #1310082, LP: #1310076, LP: #1310082)
    - Fix parallel juju deployments. (LP: #1314409)
    - Clear distro_series when stopping node from WebUI (LP: #1316396)
    - Fix click hijacking (LP: #1298784)
    - Fix blocking API client when deleting a resource (LP: #1313556)
    - Do not import Trusty RC images by default (LP: #1311151)
    - debian/control: Add missing dep on python-crochet for
      python-maas-provisioningserver (LP: #1311765)
 -- Andres Rodriguez <email address hidden> Fri, 09 May 2014 22:35:43 -0500

Changed in maas (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for maas has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers