api calls fail while launchpad is readonly

[Martin Pool]

  affects launchpad
  status triaged
  importance high
  tags api readonly-mode

When lp is readonly, API requests that ought to also be readonly still fail:

+/home/mbp/bin/update-kanbans:3> ./bin/kanban generate-person-kanban
canonical-bazaar
Traceback (most recent call last):
  File "./bin/kanban", line 14, in <module>
    main(sys.argv)
  File "./kanban/entry_point.py", line 25, in main
    controller.run(argv[1:])
  File "/home/mbp/lib/python/commandant/controller.py", line 205, in run
    run_bzr(argv)
  File "/usr/lib/pymodules/python2.6/bzrlib/commands.py", line 1123, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/pymodules/python2.6/bzrlib/commands.py", line 691, in
run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/pymodules/python2.6/bzrlib/commands.py", line 710, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/pymodules/python2.6/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/pymodules/python2.6/bzrlib/cleanup.py", line 165, in
_do_with_cleanups
    result = func(*args, **kwargs)
  File "./kanban/commands.py", line 66, in run
    bugs = get_person_assigned_bugs(launchpad, person_name)
  File "./kanban/launchpad.py", line 129, in get_person_assigned_bugs
    person = launchpad.people[person_name]
  File "/usr/lib/python2.6/dist-packages/lazr/restfulclient/resource.py",
line 917, in __getitem__
    unicode(self._root._browser.get(url)))
  File "/usr/lib/python2.6/dist-packages/lazr/restfulclient/_browser.py",
line 287, in get
    response, content = self._request(url, extra_headers=headers)
  File "/usr/lib/python2.6/dist-packages/lazr/restfulclient/_browser.py",
line 277, in _request
    raise HTTPError(response, content)
lazr.restfulclient.errors.HTTPError: HTTP Error 503: Service Unavailable
Response headers:
---
-content-encoding: gzip
connection: close
content-length: 5667
content-type: text/html;charset=utf-8
date: Wed, 09 Feb 2011 23:17:42 GMT
server: zope.server.http (HTTP)
status: 503
transfer-encoding: chunked
vary: Accept-Encoding
via: 1.1 wildcard.edge.launchpad.net
x-powered-by: Zope (www.zope.org), Python (www.python.org)
---
Response body:
---

[Robert Collins (lifeless)]

Is there an OOPS in the body?

Without an OOPS its a little tricky to figure out whats going on, but I'll take a stab and say its the nonce table rejecting writes.

-> we cannot fix this while using oauth and other nonce based systems, unless we move the nonces out of postgresql (and even then we may still be taking the services in question down).

[Martin Pool (mbp)]

No, there's no OOPS.

I agree this it's better to just reduce the readonly windows.

[Robert Collins (lifeless)]

I'm marking this low for a couple of reasons:
 - we're simply not likely to get to it within 2011
 - we're working on reducing downtime windows with a goal of eventually removing readonly mode, which will remove the need to directly fix this bug at all
 - moving the nonce out of the db has tricky implications on points of failure, retrying of requests.

[William Grant (wgrant)]

It is the consumer check attempting to get the master store. This happens even for anonymous requests.

[Robert Collins (lifeless)]

There is now not a readonly mode.

[Martin Pool (mbp)]

\o/