add-apt-repository intermittently fails on krillin

[Para Siva]

Running 'sudo add-apt-repository --yes ppa:ci-train-ppa-service/stable-phone-overlay' on krillin devices (witth vivid devel-proposed images) sometimes fails to add the ppa and fails with the following error:

Cannot add PPA: 'ppa:~ci-train-ppa-service/ubuntu/stable-phone-overlay'.
The team named '~ci-train-ppa-service' has no PPA named 'ubuntu/stable-phone-overlay'
Please choose from the following available PPAs:
 * 'landing-000': Landing PPA 000
 * 'landing-000': Landing PPA 000 (RTM)
 * 'landing-001': Landing PPA 001 (RTM)
 * 'landing-001': Landing PPA 001
 * 'landing-002': Landing PPA 002
 * 'landing-002': Landing PPA 002 (RTM)
 * 'landing-003': Landing PPA 003
 * 'landing-003': Landing PPA 003 (RTM)
 * 'landing-004': Landing PPA 004 (RTM)
 * 'landing-004': Landing PPA 004
 * 'landing-005': Landing PPA 005
 * 'landing-005': Landing PPA 005 (RTM)
 * 'landing-006': Landing PPA 006 (RTM)
 * 'landing-006': Landing PPA 006
 * 'landing-007': Landing PPA 007
 * 'landing-007': Landing PPA 007 (RTM)
 * 'landing-008': Landing PPA 008 (RTM)
 * 'landing-008': Landing PPA 008
 * 'landing-009': Landing PPA 009 (RTM)
 * 'landing-009': Landing PPA 009
 * 'landing-010': Landing PPA 010 (RTM)
 * 'landing-010': Landing PPA 010
 * 'landing-011': Landing PPA 011
 * 'landing-011': Landing PPA 011 (RTM)
 * 'landing-012': Landing PPA 012 (RTM)
 * 'landing-012': Landing PPA 012
 * 'landing-013': Landing PPA 013 (RTM)
 * 'landing-013': Landing PPA 013
 * 'landing-014': Landing PPA 014 (RTM)
 * 'landing-014': Landing PPA 014
 * 'landing-015': Landing PPA 015
 * 'landing-015': Landing PPA 015 (RTM)
 * 'landing-016': Landing PPA 016 (RTM)
 * 'landing-016': Landing PPA 016
 * 'landing-017': Landing PPA 017
 * 'landing-017': Landing PPA 017 (RTM)
 * 'landing-018': Landing PPA 018 (RTM)
 * 'landing-018': Landing PPA 018
 * 'landing-019': Landing PPA 019 (RTM)
 * 'landing-019': Landing PPA 019
 * 'landing-020': Landing PPA 020
 * 'landing-020': Landing PPA 020 (RTM)
 * 'landing-021': Landing PPA 021 (RTM)
 * 'landing-021': Landing PPA 021
 * 'landing-022': Landing PPA 022 (RTM)
 * 'landing-022': Landing PPA 022
 * 'landing-023': Landing PPA 023 (RTM)
 * 'landing-023': Landing PPA 023
 * 'landing-024': Landing PPA 024
 * 'landing-024': Landing PPA 024 (RTM)
 * 'landing-025': Landing PPA 025
 * 'landing-025': Landing PPA 025 (RTM)
 * 'landing-026': Landing PPA 026
 * 'landing-026': Landing PPA 026 (RTM)
 * 'landing-027': Landing PPA 027
 * 'landing-027': Landing PPA 027 (RTM)
 * 'landing-028': Landing PPA 028 (RTM)
 * 'landing-028': Landing PPA 028
 * 'landing-029': Landing PPA 029
 * 'landing-029': Landing PPA 029 (RTM)
 * 'landing-030': Landing PPA 030 (RTM)
 * 'landing-030': Landing PPA 030
 * 'landing-031': Landing PPA 031
 * 'landing-031': Landing PPA 031 (RTM)
 * 'landing-032': Landing PPA 032
 * 'landing-032': Landing PPA 032 (RTM)
 * 'landing-033': Landing PPA 033
 * 'landing-033': Landing PPA 033 (RTM)
 * 'landing-034': Landing PPA 034 (RTM)
 * 'landing-034': Landing PPA 034
 * 'landing-035': Landing PPA 035
 * 'landing-035': Landing PPA 035 (RTM)
 * 'landing-036': Landing PPA 036
 * 'landing-036': Landing PPA 036 (RTM)
 * 'landing-037': Landing PPA 037 (RTM)
===============================================
We ran this by the launchpad team and based on the analysis that cjwatson carried out, he had this to say:

'The bit where it only lists a subset of PPAs is a bug in add-apt-repository. It's talking to the Launchpad API by hand to avoid needing to introduce additional dependencies, but unfortunately it doesn't implement the collection API properly: it needs to follow links to subsequent batches until it's found everything. (Notice that it's shown you 75 entries, which is exactly our current batch size on production.) I'd suggest raising that with the Foundations team, who can come to us if they need help fixing it.

I'm not completely sure why it couldn't add the PPA in the first place, though, and I suspect the underlying error has been obscured by add-apt-repository somewhere along the line. I compared with appserver logs for the relevant timestamps (which are just about still available); there are records of requests from a bit later in that hour, but add-apt-repository's request didn't hit any appserver as far as I can tell. I therefore suspect that it was denied by haproxy due to temporary high load or some transient network problem, and indeed, add-apt-repository ought to have some back-off-and-retry logic. Again, that's something the Foundations team would be best-placed to fix.'

We could add that this issue is only visible for us on krillin devices and that this is an intermittent issue with somewhat higher frequency. The mako runs with the same command hasn't failed at all this way.

[Para Siva (psivaa)]

There is another type of failure, seemingly due to the same cause:

+ sudo add-apt-repository --yes ppa:ci-train-ppa-service/stable-phone-overlay
Traceback (most recent call last):
  File "/usr/lib/python3.4/urllib/request.py", line 1182, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.4/http/client.py", line 1088, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1126, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.4/http/client.py", line 1084, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.4/http/client.py", line 922, in _send_output
    self.send(msg)
  File "/usr/lib/python3.4/http/client.py", line 857, in send
    self.connect()
  File "/usr/lib/python3.4/http/client.py", line 1223, in connect
    super().connect()
  File "/usr/lib/python3.4/http/client.py", line 834, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python3.4/socket.py", line 494, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.4/socket.py", line 533, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 101, in _get_https_content_py3
    lp_page = urllib.request.urlopen(request, cafile=LAUNCHPAD_PPA_CERT)
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 463, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 481, in _open
    '_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1225, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.4/urllib/request.py", line 1184, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 321, in get_ppa_info
    ret = get_ppa_info_from_lp(user, ppa)
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 91, in get_ppa_info_from_lp
    return get_info_from_lp(lp_url)
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 87, in get_info_from_lp
    return _get_https_content_py3(lp_url)
  File "/usr/lib/python3/dist-packages/softwareproperties/ppa.py", line 107, in _get_https_content_py3
    raise PPAException("Error reading %s: %s" % (lp_url, reason), e)
softwareproperties.ppa.PPAException: 'Error reading https://launchpad.net/api/1.0/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay: [Errno -3] Temporary failure in name resolution'

Du...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.