Bug #1745211 “launchpad outages hang the importer and scripts ca...” : Bugs : git-ubuntu

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2018-01-25:

#1

If libcurl is used at some point, it has a parameter to kill the connection if it the transfer speed is below a certain threshold.

We use these in landscape-client, fwiw:

https://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html
https://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html

Robie Basak (racb) on 2018-01-25

Changed in usd-importer:
milestone:	none → lp-beta

Revision history for this message

Bryce Harrington (bryce) wrote on 2019-11-01:

#2

I suspect this is probably adequately addressed now by commit 5d104bce612db56f3274c667ee7b008da5e68c18 (LP: #1838954), which adds a watchdog that will kill and restart the process if it gets hung. There's some delay in the restart for giving LP a chance to restore itself.

However, we've not yet experienced a launchpad outage so I'm not 100% certain about the behavior, so am leaving the bug open for now.

Changed in usd-importer:
status:	New → In Progress
assignee:	nobody → Bryce Harrington (bryce)

Revision history for this message

Bryce Harrington (bryce) wrote on 2019-11-04:

#3

Download full text (7.0 KiB)

Bingo. We experienced a brief LP outage over the weekend, which the new watchdog detected and correctly resolved by restarting the service. So the new code is successfully handling the error condition now.

Nov 04 00:57:44 git-ubuntu git-ubuntu.import-source-packages[2219]: gpgv: using RSA key 3B4FE6ACC0B21F32
Nov 04 00:57:44 git-ubuntu git-ubuntu.import-source-packages[2219]: gpgv: Signature made Mon 04 Nov 2019 12:04:50 AM UTC
Nov 04 00:57:41 git-ubuntu systemd[1577]: Started Git Ubuntu USD Importer service.
Nov 04 00:57:40 git-ubuntu systemd[1577]: Starting Git Ubuntu USD Importer service...
Nov 04 00:57:40 git-ubuntu systemd[1577]: Stopped Git Ubuntu USD Importer service.
Nov 04 00:57:40 git-ubuntu systemd[1577]: usd-importer.service: Service hold-off time over, scheduling restart.
Nov 04 00:56:40 git-ubuntu systemd[1577]: usd-importer.service: Failed with result 'exit-code'.
Nov 04 00:56:40 git-ubuntu systemd[1577]: usd-importer.service: Triggering OnFailure= dependencies.
Nov 04 00:56:40 git-ubuntu systemd[1577]: usd-importer.service: Unit entered failed state.
Nov 04 00:56:40 git-ubuntu systemd[1577]: usd-importer.service: Main process exited, code=exited, status=1/FAILURE
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: ---
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: div>\n \n \n\n </div>\n\n <div class="colophon">\n © 2004-2019\n <a href="http://canonical.com/">Canonical Ltd.</a>\n  • \n <a href="https://launchpad.net
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: n \n \n <li class="answers"><a href="https://answers.launchpad.net/debian">Answers</a></li>\n \n \n </ul>\n\n </div>\n\n \n <div id="maincontent" class="yui-main">\n
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: type=\' + node.type + \'>): \' + e);\n });\n }\n break;\n }\n }\n }\n\n function selectWidget(widget_name, even
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: pe="text/javascript">\n //<![CDATA[\n LPJS.use(\'base\', \'node\', \'console\', \'event\',\n \'oop\', \'lp\', \'lp.app.foldables\',\'lp.app.sorttable\',\n \'lp.app.inlinehelp\',
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: b'<!DOCTYPE html>\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">\n <head>\n <meta charset="UTF-8" />\n <title>Error: Timeout</title>\n <link rel="shortcut icon" href="/@@/
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: ---
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: Response body:
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: ---
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: x-powered-by: Zope (www.zope.org), Python (www.python.org)
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: x-lazr-oopsid: OOPS-b583e6dcc7a31f6f66fbdadc8...

Bingo.  We experienced a brief LP outage over the weekend, which the new watchdog detected and correctly resolved by restarting the service.  So the new code is successfully handling the error condition now.

Nov 04 00:57:44 git-ubuntu git-ubuntu.import-source-packages[2219]: gpgv:                using RSA key 3B4FE6ACC0B21F32
Nov 04 00:57:44 git-ubuntu git-ubuntu.import-source-packages[2219]: gpgv: Signature made Mon 04 Nov 2019 12:04:50 AM UTC
Nov 04 00:57:41 git-ubuntu systemd[1577]: Started Git Ubuntu USD Importer service.
Nov 04 00:57:40 git-ubuntu systemd[1577]: Starting Git Ubuntu USD Importer service...
Nov 04 00:57:40 git-ubuntu systemd[1577]: Stopped Git Ubuntu USD Importer service.
Nov 04 00:57:40 git-ubuntu systemd[1577]: usd-importer.service: Service hold-off time over, scheduling restart.
Nov 04 00:56:40 git-ubuntu systemd[1577]: usd-importer.service: Failed with result 'exit-code'.
Nov 04 00:56:40 git-ubuntu systemd[1577]: usd-importer.service: Triggering OnFailure= dependencies.
Nov 04 00:56:40 git-ubuntu systemd[1577]: usd-importer.service: Unit entered failed state.
Nov 04 00:56:40 git-ubuntu systemd[1577]: usd-importer.service: Main process exited, code=exited, status=1/FAILURE
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: ---
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: div>\n        \n  \n\n    </div>\n\n    <div class="colophon">\n      &copy; 2004-2019\n      <a href="http://canonical.com/">Canonical&nbsp;Ltd.</a>\n      &nbsp;&bull;&nbsp;\n      <a href="https://launchpad.net
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: n    \n      \n      <li class="answers"><a href="https://answers.launchpad.net/debian">Answers</a></li>\n      \n    \n  </ul>\n\n      </div>\n\n      \n        <div id="maincontent" class="yui-main">\n         
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:  type=\' + node.type + \'>): \' + e);\n                        });\n                    }\n                    break;\n                }\n            }\n        }\n\n        function selectWidget(widget_name, even
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: pe="text/javascript">\n        //<![CDATA[\n        LPJS.use(\'base\', \'node\', \'console\', \'event\',\n            \'oop\', \'lp\', \'lp.app.foldables\',\'lp.app.sorttable\',\n            \'lp.app.inlinehelp\',
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: b'<!DOCTYPE html>\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">\n  <head>\n    <meta charset="UTF-8" />\n    <title>Error: Timeout</title>\n    <link rel="shortcut icon" href="/@@/
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: ---
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: Response body:
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: ---
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: x-powered-by: Zope (www.zope.org), Python (www.python.org)
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: x-lazr-oopsid: OOPS-b583e6dcc7a31f6f66fbdadc8364c517
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: vary: Accept-Encoding
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: status: 503
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: server: zope.server.http (HTTP)
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: retry-after: 900
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: date: Mon, 04 Nov 2019 00:56:35 GMT
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: content-type: text/html;charset=utf-8
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: content-length: 9350
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: connection: close
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: -content-encoding: gzip
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: ---
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: Response headers:
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: lazr.restfulclient.errors.ServerError: HTTP Error 503: Service Unavailable
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     raise error
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/lib/python3.6/site-packages/lazr/restfulclient/_browser.py", line 438, in _request
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     response, content = self._request(url, extra_headers=headers)
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/lib/python3.6/site-packages/lazr/restfulclient/_browser.py", line 448, in get
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     representation = self._root._browser.get(self._wadl_resource)
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/lib/python3.6/site-packages/lazr/restfulclient/resource.py", line 379, in _ensure_representation
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     self._ensure_representation()
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/lib/python3.6/site-packages/lazr/restfulclient/resource.py", line 414, in value
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     return total_size.value
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/lib/python3.6/site-packages/lazr/restfulclient/resource.py", line 813, in __len__
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     if not spph:
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/bin/import-source-packages.py", line 97, in request_new_imports
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     request_new_imports(pkgnames_to_import, num_days_ago, service_state)
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/bin/import-source-packages.py", line 251, in main
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     only_request_new_imports_once=args.only_request_new_imports_once,
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/bin/import-source-packages.py", line 367, in cli_main
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:     cli_main()
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]:   File "/snap/git-ubuntu/460/bin/import-source-packages.py", line 371, in <module>
Nov 04 00:56:40 git-ubuntu git-ubuntu.import-source-packages[30185]: Traceback (most recent call last):
Nov 04 00:55:29 git-ubuntu git-ubuntu.import-source-packages[30185]: Examining publishes in debian since 2019-11-03 22:43:29

Changed in usd-importer:
importance:	Undecided → High
status:	In Progress → Fix Released

git-ubuntu

launchpad outages hang the importer and scripts calling into launchpadlib

Bug Description

Other bug subscribers

Remote bug watches