| 2014-12-16 16:42:29 |
Larry Michel |
bug |
|
|
added bug |
| 2014-12-16 16:43:13 |
Larry Michel |
description |
We run into transient network issues where index files fail to download. The deployment ends up being marked as failed. Then subsequent deployment fails. Curtin should be able to retry when such error happens.
Here's console output:
========================================================================
Get:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]
Get:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]
Get:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]
Fetched 13.8 MB in 5s (2426 kB/s)
W: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
Unexpected error while running command.
Command: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']
Exit code: 100
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: "Ign http://archive.ubuntu.com trusty InRelease\nIgn http://archive.ubuntu.com trusty-updates InRelease\nIgn http://archive.ubuntu.com trusty-security InRelease\nGet:1 http://archive.ubuntu.com trusty Release.gpg [933 B]\nGet:2 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]\nGet:3 http://archive.ubuntu.com trusty-security Release.gpg [933 B]\nGet:4 http://archive.ubuntu.com trusty Release [58.5 kB]\nGet:5 http://archive.ubuntu.com trusty-updates Release [62.0 kB]\nGet:6 http://archive.ubuntu.com trusty-security Release [62.0 kB]\nGet:7 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]\nGet:8 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]\nGet:9 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]\nGet:10 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]\nGet:11 http://archive.ubuntu.com trusty/main Translation-en [762 kB]\nGet:12 http://archive.ubuntu.com trusty/multiverse Translation-en [102 kB]\nGet:13 http://archive.ubuntu.com trusty/restricted Translation-en [3457 B]\nGet:14 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]\nGet:15 http://archive.ubuntu.com trusty-updates/main amd64 Packages [384 kB]\nGet:16 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [8861 B]\nGet:17 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [228 kB]\nGet:18 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [9356 B]\nGet:19 http://archive.ubuntu.com trusty-updates/main Translation-en [179 kB]\nGet:20 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [4719 B]\nGet:21 http://archive.ubuntu.com trusty-updates/restricted Translation-en [2266 B]\nGet:22 http://archive.ubuntu.com trusty-updates/universe Translation-en [117 kB]\nGet:23 http://archive.ubuntu.com trusty-security/main amd64 Packages [181 kB]\nGet:24 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [8861 B]\nGet:25 http://archive.ubuntu.com trusty-security/universe amd64 Packages [76.0 kB]\nGet:26 http://archive.ubuntu.com trusty-security/multiverse amd64 Packages [1143 B]\nGet:27 http://archive.ubuntu.com trusty-security/main Translation-en [90.8 kB]\nGet:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]\nGet:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]\nGet:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]\nFetched 13.8 MB in 5s (2426 kB/s)\nW: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch\n\nE: Some index files failed to download. They have been ignored, or old ones used instead.\nUnexpected error while running command.\nCommand: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']\nExit code: 100\nReason: -\nStdout: ''\nStderr: ''\n"
Stderr: ''
Success
ci-info: +++++++Authorized keys
======================================================================== |
We run into transient network issues where index files fail to download. The deployment ends up being marked as failed. Then subsequent deployment succeeds but test has already failed. Curtin should be able to retry when such error happens.
Here's console output:
========================================================================
Get:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]
Get:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]
Get:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]
Fetched 13.8 MB in 5s (2426 kB/s)
W: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
Unexpected error while running command.
Command: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']
Exit code: 100
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: "Ign http://archive.ubuntu.com trusty InRelease\nIgn http://archive.ubuntu.com trusty-updates InRelease\nIgn http://archive.ubuntu.com trusty-security InRelease\nGet:1 http://archive.ubuntu.com trusty Release.gpg [933 B]\nGet:2 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]\nGet:3 http://archive.ubuntu.com trusty-security Release.gpg [933 B]\nGet:4 http://archive.ubuntu.com trusty Release [58.5 kB]\nGet:5 http://archive.ubuntu.com trusty-updates Release [62.0 kB]\nGet:6 http://archive.ubuntu.com trusty-security Release [62.0 kB]\nGet:7 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]\nGet:8 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]\nGet:9 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]\nGet:10 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]\nGet:11 http://archive.ubuntu.com trusty/main Translation-en [762 kB]\nGet:12 http://archive.ubuntu.com trusty/multiverse Translation-en [102 kB]\nGet:13 http://archive.ubuntu.com trusty/restricted Translation-en [3457 B]\nGet:14 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]\nGet:15 http://archive.ubuntu.com trusty-updates/main amd64 Packages [384 kB]\nGet:16 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [8861 B]\nGet:17 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [228 kB]\nGet:18 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [9356 B]\nGet:19 http://archive.ubuntu.com trusty-updates/main Translation-en [179 kB]\nGet:20 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [4719 B]\nGet:21 http://archive.ubuntu.com trusty-updates/restricted Translation-en [2266 B]\nGet:22 http://archive.ubuntu.com trusty-updates/universe Translation-en [117 kB]\nGet:23 http://archive.ubuntu.com trusty-security/main amd64 Packages [181 kB]\nGet:24 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [8861 B]\nGet:25 http://archive.ubuntu.com trusty-security/universe amd64 Packages [76.0 kB]\nGet:26 http://archive.ubuntu.com trusty-security/multiverse amd64 Packages [1143 B]\nGet:27 http://archive.ubuntu.com trusty-security/main Translation-en [90.8 kB]\nGet:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]\nGet:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]\nGet:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]\nFetched 13.8 MB in 5s (2426 kB/s)\nW: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch\n\nE: Some index files failed to download. They have been ignored, or old ones used instead.\nUnexpected error while running command.\nCommand: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']\nExit code: 100\nReason: -\nStdout: ''\nStderr: ''\n"
Stderr: ''
Success
ci-info: +++++++Authorized keys
======================================================================== |
|
| 2014-12-16 16:43:22 |
Larry Michel |
tags |
|
oil |
|
| 2014-12-17 15:30:18 |
Blake Rouse |
curtin: status |
New |
Triaged |
|
| 2014-12-17 15:30:20 |
Blake Rouse |
curtin: importance |
Undecided |
High |
|
| 2015-06-05 17:23:21 |
Blake Rouse |
branch linked |
|
lp:~blake-rouse/curtin/fix-1453869 |
|
| 2015-06-05 17:23:31 |
Blake Rouse |
curtin: status |
Triaged |
In Progress |
|
| 2015-06-05 17:23:33 |
Blake Rouse |
curtin: assignee |
|
Blake Rouse (blake-rouse) |
|
| 2015-06-05 17:26:06 |
Launchpad Janitor |
branch linked |
|
lp:curtin |
|
| 2015-06-05 17:26:16 |
Blake Rouse |
curtin: status |
In Progress |
Fix Committed |
|
| 2015-06-05 18:53:09 |
Scott Moser |
bug task added |
|
curtin (Ubuntu) |
|
| 2015-06-05 18:53:29 |
Scott Moser |
curtin (Ubuntu): importance |
Undecided |
Medium |
|
| 2015-06-05 18:53:29 |
Scott Moser |
curtin (Ubuntu): status |
New |
In Progress |
|
| 2015-06-05 19:52:20 |
Launchpad Janitor |
curtin (Ubuntu): status |
In Progress |
Fix Released |
|
| 2015-06-21 12:43:41 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/curtin |
|
| 2015-06-22 14:39:56 |
Launchpad Janitor |
branch linked |
|
lp:~smoser/ubuntu/trusty/curtin/trusty-sru |
|
| 2015-06-22 14:42:47 |
Scott Moser |
nominated for series |
|
Ubuntu Vivid |
|
| 2015-06-22 14:42:47 |
Scott Moser |
bug task added |
|
curtin (Ubuntu Vivid) |
|
| 2015-06-22 14:42:47 |
Scott Moser |
nominated for series |
|
Ubuntu Trusty |
|
| 2015-06-22 14:42:47 |
Scott Moser |
bug task added |
|
curtin (Ubuntu Trusty) |
|
| 2015-06-22 14:43:15 |
Scott Moser |
curtin (Ubuntu Trusty): status |
New |
Confirmed |
|
| 2015-06-22 14:43:17 |
Scott Moser |
curtin (Ubuntu Vivid): status |
New |
Confirmed |
|
| 2015-06-22 14:43:20 |
Scott Moser |
curtin (Ubuntu Trusty): importance |
Undecided |
Medium |
|
| 2015-06-22 14:43:23 |
Scott Moser |
curtin (Ubuntu Vivid): importance |
Undecided |
Medium |
|
| 2015-06-22 15:44:01 |
Scott Moser |
description |
We run into transient network issues where index files fail to download. The deployment ends up being marked as failed. Then subsequent deployment succeeds but test has already failed. Curtin should be able to retry when such error happens.
Here's console output:
========================================================================
Get:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]
Get:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]
Get:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]
Fetched 13.8 MB in 5s (2426 kB/s)
W: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
Unexpected error while running command.
Command: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']
Exit code: 100
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: "Ign http://archive.ubuntu.com trusty InRelease\nIgn http://archive.ubuntu.com trusty-updates InRelease\nIgn http://archive.ubuntu.com trusty-security InRelease\nGet:1 http://archive.ubuntu.com trusty Release.gpg [933 B]\nGet:2 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]\nGet:3 http://archive.ubuntu.com trusty-security Release.gpg [933 B]\nGet:4 http://archive.ubuntu.com trusty Release [58.5 kB]\nGet:5 http://archive.ubuntu.com trusty-updates Release [62.0 kB]\nGet:6 http://archive.ubuntu.com trusty-security Release [62.0 kB]\nGet:7 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]\nGet:8 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]\nGet:9 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]\nGet:10 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]\nGet:11 http://archive.ubuntu.com trusty/main Translation-en [762 kB]\nGet:12 http://archive.ubuntu.com trusty/multiverse Translation-en [102 kB]\nGet:13 http://archive.ubuntu.com trusty/restricted Translation-en [3457 B]\nGet:14 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]\nGet:15 http://archive.ubuntu.com trusty-updates/main amd64 Packages [384 kB]\nGet:16 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [8861 B]\nGet:17 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [228 kB]\nGet:18 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [9356 B]\nGet:19 http://archive.ubuntu.com trusty-updates/main Translation-en [179 kB]\nGet:20 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [4719 B]\nGet:21 http://archive.ubuntu.com trusty-updates/restricted Translation-en [2266 B]\nGet:22 http://archive.ubuntu.com trusty-updates/universe Translation-en [117 kB]\nGet:23 http://archive.ubuntu.com trusty-security/main amd64 Packages [181 kB]\nGet:24 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [8861 B]\nGet:25 http://archive.ubuntu.com trusty-security/universe amd64 Packages [76.0 kB]\nGet:26 http://archive.ubuntu.com trusty-security/multiverse amd64 Packages [1143 B]\nGet:27 http://archive.ubuntu.com trusty-security/main Translation-en [90.8 kB]\nGet:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]\nGet:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]\nGet:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]\nFetched 13.8 MB in 5s (2426 kB/s)\nW: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch\n\nE: Some index files failed to download. They have been ignored, or old ones used instead.\nUnexpected error while running command.\nCommand: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']\nExit code: 100\nReason: -\nStdout: ''\nStderr: ''\n"
Stderr: ''
Success
ci-info: +++++++Authorized keys
======================================================================== |
=== Begin SRU Template ===
[Description]
During installation, curtin will run 'apt-get update' on in the target root. That is done as a requirement to installing new packages in the target.
'apt-get update' is widely known to fail as a result of transient network failures. This is commonly worked around by simply sleeping and re-trying the operation.
The solution implemented is to improve the 'subp' (subprocess) helper in curtin/util to take a 'retries' argument.
If provided that is a iterator that contains a time to sleep before trying again. If no retries is provided, then only one try is done.
Then, the curtin/util.py helper apt_update invokes subp with retries=(1, 2, 3).
[Impact]
Installation fails when a simple retry of 'apt-get update' would have succeeded.
[Test Case]
As this is a transient failure, it is hard to catch and hard to test for.
Installation should be more reliable now, with any 'apt-get update' operation that returned non-zero being retried 3 times.
[Regression Potential]
The only really likely regression path here would be retrying 'apt update' on its successful return. That seems fairly unlikely as the code in subp to check exit status has not changed.
[Other]
Related bugs:
* bug 972077: apt repository disk format has race conditions
=== End SRU Template ===
We run into transient network issues where index files fail to download. The deployment ends up being marked as failed. Then subsequent deployment succeeds but test has already failed. Curtin should be able to retry when such error happens.
Here's console output:
========================================================================
Get:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]
Get:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]
Get:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]
Fetched 13.8 MB in 5s (2426 kB/s)
W: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
Unexpected error while running command.
Command: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']
Exit code: 100
Reason: -
Stdout: ''
Stderr: ''
Installation failed with exception: Unexpected error while running command.
Command: ['curtin', 'curthooks']
Exit code: 3
Reason: -
Stdout: "Ign http://archive.ubuntu.com trusty InRelease\nIgn http://archive.ubuntu.com trusty-updates InRelease\nIgn http://archive.ubuntu.com trusty-security InRelease\nGet:1 http://archive.ubuntu.com trusty Release.gpg [933 B]\nGet:2 http://archive.ubuntu.com trusty-updates Release.gpg [933 B]\nGet:3 http://archive.ubuntu.com trusty-security Release.gpg [933 B]\nGet:4 http://archive.ubuntu.com trusty Release [58.5 kB]\nGet:5 http://archive.ubuntu.com trusty-updates Release [62.0 kB]\nGet:6 http://archive.ubuntu.com trusty-security Release [62.0 kB]\nGet:7 http://archive.ubuntu.com trusty/main amd64 Packages [1350 kB]\nGet:8 http://archive.ubuntu.com trusty/restricted amd64 Packages [13.0 kB]\nGet:9 http://archive.ubuntu.com trusty/universe amd64 Packages [5859 kB]\nGet:10 http://archive.ubuntu.com trusty/multiverse amd64 Packages [132 kB]\nGet:11 http://archive.ubuntu.com trusty/main Translation-en [762 kB]\nGet:12 http://archive.ubuntu.com trusty/multiverse Translation-en [102 kB]\nGet:13 http://archive.ubuntu.com trusty/restricted Translation-en [3457 B]\nGet:14 http://archive.ubuntu.com trusty/universe Translation-en [4089 kB]\nGet:15 http://archive.ubuntu.com trusty-updates/main amd64 Packages [384 kB]\nGet:16 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [8861 B]\nGet:17 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [228 kB]\nGet:18 http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages [9356 B]\nGet:19 http://archive.ubuntu.com trusty-updates/main Translation-en [179 kB]\nGet:20 http://archive.ubuntu.com trusty-updates/multiverse Translation-en [4719 B]\nGet:21 http://archive.ubuntu.com trusty-updates/restricted Translation-en [2266 B]\nGet:22 http://archive.ubuntu.com trusty-updates/universe Translation-en [117 kB]\nGet:23 http://archive.ubuntu.com trusty-security/main amd64 Packages [181 kB]\nGet:24 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [8861 B]\nGet:25 http://archive.ubuntu.com trusty-security/universe amd64 Packages [76.0 kB]\nGet:26 http://archive.ubuntu.com trusty-security/multiverse amd64 Packages [1143 B]\nGet:27 http://archive.ubuntu.com trusty-security/main Translation-en [90.8 kB]\nGet:28 http://archive.ubuntu.com trusty-security/multiverse Translation-en [587 B]\nGet:29 http://archive.ubuntu.com trusty-security/restricted Translation-en [2266 B]\nGet:30 http://archive.ubuntu.com trusty-security/universe Translation-en [41.5 kB]\nFetched 13.8 MB in 5s (2426 kB/s)\nW: Failed to fetch http://archive.ubuntu.com//ubuntu/dists/trusty-updates/universe/i18n/Translation-en Hash Sum mismatch\n\nE: Some index files failed to download. They have been ignored, or old ones used instead.\nUnexpected error while running command.\nCommand: ['chroot', '/tmp/tmp8mxme7/target', 'apt-get', 'update', '--quiet']\nExit code: 100\nReason: -\nStdout: ''\nStderr: ''\n"
Stderr: ''
Success
ci-info: +++++++Authorized keys
======================================================================== |
|
| 2015-06-24 18:37:04 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/wily-proposed/curtin |
|
| 2015-06-24 20:18:56 |
Chris J Arges |
curtin (Ubuntu Trusty): status |
Confirmed |
Fix Committed |
|
| 2015-06-24 20:18:58 |
Chris J Arges |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2015-06-24 20:19:00 |
Chris J Arges |
bug |
|
|
added subscriber SRU Verification |
| 2015-06-24 20:19:03 |
Chris J Arges |
tags |
oil |
oil verification-needed |
|
| 2015-06-24 20:22:59 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-proposed/curtin |
|
| 2015-06-24 20:27:09 |
Chris J Arges |
curtin (Ubuntu Vivid): status |
Confirmed |
Fix Committed |
|
| 2015-06-24 20:32:55 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/vivid-proposed/curtin |
|
| 2015-07-09 20:57:01 |
Scott Moser |
tags |
oil verification-needed |
oil verification-done |
|
| 2015-07-09 21:03:19 |
Scott Moser |
attachment added |
|
curtin retrying of apt-get update failure https://bugs.launchpad.net/curtin/+bug/1403133/+attachment/4427189/+files/my.log |
|
| 2015-07-15 18:47:45 |
Launchpad Janitor |
curtin (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2015-07-15 18:48:10 |
Chris J Arges |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2015-07-15 18:48:34 |
Launchpad Janitor |
curtin (Ubuntu Vivid): status |
Fix Committed |
Fix Released |
|
| 2017-12-15 22:09:10 |
Scott Moser |
curtin: status |
Fix Committed |
Fix Released |
|
