| 2016-01-13 13:45:05 |
Alexandr Kostrikov |
bug |
|
|
added bug |
| 2016-01-13 14:00:22 |
Alexandr Kostrikov |
description |
At a job https://product-ci.infra.mirantis.net/job/8.0.ubuntu.bvt_2/383/console we have an error
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Failed to execute hook 'shell' Failed to run command cd / && apt-get update
---
priority: 2200
type: shell
id:
parameters:
retries: 3
cmd: apt-get update
cwd: /
timeout: 1800
interval: 1
uids:
- '1'
- '3'
- '2'
- '5'
- '4'
- '6'
.
Inspect Astute logs for the details'
During debugging I found that at a small amount of time there is `apt-get update` errors.
At most times apt-get has passed successfully, but there were small period of time when such errors appear:
First:
'Fetched 864 kB in 2min 0s (7,162 B/s)
W: GPG error: http://mirror.fuel-infra.org mos8.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-proposed/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.'
Exit code: 100
Second:
'
Fetched 820 kB in 1s (651 kB/s)
W: GPG error: http://mirror.fuel-infra.org mos8.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/main/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/main/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/multiverse/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/multiverse/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/universe/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/universe/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-proposed/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
'
Exit code: 100
As the only available access method for me on the mirrors is http access, I found that
mirror snapshots have not been changed during that time(at least for web server):
http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/
Index of /pkgs/ubuntu-2016-01-12-170104/
../
dists/ 08-Mar-2015 19:35 -
indices/ 12-Jan-2016 15:29 -
pool/ 27-Feb-2010 06:30 -
project/ 28-Jun-2013 11:52 -
ubuntu/ 12-Jan-2016 17:17 -
To be able to debug that error further I need full output of 'apt-get update command' available. |
At a job https://product-ci.infra.mirantis.net/job/8.0.ubuntu.bvt_2/383/console we have an error
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Failed to execute hook 'shell' Failed to run command cd / && apt-get update
---
priority: 2200
type: shell
id:
parameters:
retries: 3
cmd: apt-get update
cwd: /
timeout: 1800
interval: 1
uids:
- '1'
- '3'
- '2'
- '5'
- '4'
- '6'
.
Inspect Astute logs for the details'
During debugging I found that at a small amount of time there is `apt-get update` errors.
At most times apt-get has passed successfully, but there were small period of time when such errors appear:
First:
'Fetched 864 kB in 2min 0s (7,162 B/s)
W: GPG error: http://mirror.fuel-infra.org mos8.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-proposed/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.'
Exit code: 100
Second:
'
Fetched 820 kB in 1s (651 kB/s)
W: GPG error: http://mirror.fuel-infra.org mos8.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/main/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/main/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/multiverse/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/multiverse/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/universe/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/universe/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-proposed/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
'
Exit code: 100
As the only available access method for me on the mirrors is http access, I found that
mirror snapshots have not been changed during that time(at least for web server):
http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/
Index of /pkgs/ubuntu-2016-01-12-170104/
../
dists/ 08-Mar-2015 19:35 -
indices/ 12-Jan-2016 15:29 -
pool/ 27-Feb-2010 06:30 -
project/ 28-Jun-2013 11:52 -
ubuntu/ 12-Jan-2016 17:17 -
To be able to debug that error further I need full output of 'apt-get update command' available. |
|
| 2016-01-13 14:00:33 |
Alexandr Kostrikov |
fuel: status |
New |
Invalid |
|
| 2016-01-13 14:12:29 |
Alexandr Kostrikov |
fuel: status |
Invalid |
New |
|
| 2016-01-13 14:12:37 |
Alexandr Kostrikov |
fuel: assignee |
Fuel Python Team (fuel-python) |
Fuel build team (fuel-build) |
|
| 2016-01-13 14:13:35 |
Alexandr Kostrikov |
description |
At a job https://product-ci.infra.mirantis.net/job/8.0.ubuntu.bvt_2/383/console we have an error
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Failed to execute hook 'shell' Failed to run command cd / && apt-get update
---
priority: 2200
type: shell
id:
parameters:
retries: 3
cmd: apt-get update
cwd: /
timeout: 1800
interval: 1
uids:
- '1'
- '3'
- '2'
- '5'
- '4'
- '6'
.
Inspect Astute logs for the details'
During debugging I found that at a small amount of time there is `apt-get update` errors.
At most times apt-get has passed successfully, but there were small period of time when such errors appear:
First:
'Fetched 864 kB in 2min 0s (7,162 B/s)
W: GPG error: http://mirror.fuel-infra.org mos8.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-proposed/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.'
Exit code: 100
Second:
'
Fetched 820 kB in 1s (651 kB/s)
W: GPG error: http://mirror.fuel-infra.org mos8.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/main/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/main/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/multiverse/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/multiverse/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/universe/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/universe/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-proposed/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
'
Exit code: 100
As the only available access method for me on the mirrors is http access, I found that
mirror snapshots have not been changed during that time(at least for web server):
http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/
Index of /pkgs/ubuntu-2016-01-12-170104/
../
dists/ 08-Mar-2015 19:35 -
indices/ 12-Jan-2016 15:29 -
pool/ 27-Feb-2010 06:30 -
project/ 28-Jun-2013 11:52 -
ubuntu/ 12-Jan-2016 17:17 -
To be able to debug that error further I need full output of 'apt-get update command' available. |
At a job https://product-ci.infra.mirantis.net/job/8.0.ubuntu.bvt_2/383/console we have an error
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. Method granular_deploy. Failed to execute hook 'shell' Failed to run command cd / && apt-get update
---
priority: 2200
type: shell
id:
parameters:
retries: 3
cmd: apt-get update
cwd: /
timeout: 1800
interval: 1
uids:
- '1'
- '3'
- '2'
- '5'
- '4'
- '6'
.
Inspect Astute logs for the details'
During debugging I found that at a small amount of time there is `apt-get update` errors.
At most times apt-get has passed successfully, but there were small period of time when such errors appear:
First:
'Fetched 864 kB in 2min 0s (7,162 B/s)
W: GPG error: http://mirror.fuel-infra.org mos8.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-proposed/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.'
Exit code: 100
Second:
'
Fetched 820 kB in 1s (651 kB/s)
W: GPG error: http://mirror.fuel-infra.org mos8.0-holdback Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: GPG error: http://mirror.fuel-infra.org mos8.0-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BCE5CC461FA22B08
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/main/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/main/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/multiverse/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/multiverse/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/universe/i18n/Translation-en_US Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty/universe/i18n/Translation-en Bad header line
W: Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-proposed/universe/i18n/Translation-en Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.
'
Exit code: 100
As the only available access method for me on the mirrors is http access, I found that
mirror snapshots have not been changed during that time(at least for web server):
http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/
Index of /pkgs/ubuntu-2016-01-12-170104/
../
dists/ 08-Mar-2015 19:35 -
indices/ 12-Jan-2016 15:29 -
pool/ 27-Feb-2010 06:30 -
project/ 28-Jun-2013 11:52 -
ubuntu/ 12-Jan-2016 17:17 -
The errors at the Job are 'Failed to fetch http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-updates/universe/binary-amd64/Packages 404 Not Found'
But there is no need for them to present at http://mirror.seed-cz1.fuel-infra.org/pkgs/ubuntu-2016-01-12-170104/dists/trusty-updates/universe/binary-amd64/ because usually apt-get fetches gz or .bz2 files. And when there is an error with them it fallbacks to
uncompressed vesion 'Packages'.
There are timings and access to mirror state at Build team, so I reassign that bug to them.
Problematic logs are at:
http://paste.openstack.org/show/483770/ |
|
| 2016-01-13 14:16:05 |
Alexandr Kostrikov |
tags |
area-python |
area-build |
|
| 2016-01-13 15:25:41 |
Sergey Kulanov |
fuel: status |
New |
Confirmed |
|
| 2016-01-14 09:04:44 |
Max Rasskazov |
fuel: assignee |
Fuel build team (fuel-build) |
Max Rasskazov (mrasskazov) |
|
| 2016-01-14 09:04:52 |
Max Rasskazov |
fuel: status |
Confirmed |
In Progress |
|
| 2016-01-15 05:29:33 |
Nastya Urlapova |
fuel: importance |
High |
Critical |
|
| 2016-01-15 09:27:40 |
Nastya Urlapova |
tags |
area-build |
area-build swarm-blocker |
|
| 2016-01-15 10:50:42 |
Alexandr Kostrikov |
summary |
Output information is needed during `apt-get update` Astute task. |
Mirror snapshots are broken when `apt-get update` Astute task is running. |
|
| 2016-01-16 17:28:10 |
Tatyanka |
bug |
|
|
added subscriber Tatyanka |
| 2016-01-18 10:26:48 |
Sergii Golovatiuk |
fuel: assignee |
Max Rasskazov (mrasskazov) |
Sergii Golovatiuk (sgolovatiuk) |
|
| 2016-01-18 10:26:58 |
Sergii Golovatiuk |
nominated for series |
|
fuel/mitaka |
|
| 2016-01-18 10:26:58 |
Sergii Golovatiuk |
bug task added |
|
fuel/mitaka |
|
| 2016-01-18 11:05:17 |
Igor Degtiarov |
bug |
|
|
added subscriber Igor Degtiarov |
| 2016-01-18 13:31:52 |
Nastya Urlapova |
nominated for series |
|
fuel/8.0.x |
|
| 2016-01-18 13:31:52 |
Nastya Urlapova |
bug task added |
|
fuel/8.0.x |
|
| 2016-01-18 13:32:51 |
Nastya Urlapova |
fuel/8.0.x: milestone |
|
8.0 |
|
| 2016-01-18 13:33:07 |
Nastya Urlapova |
fuel/mitaka: milestone |
8.0 |
9.0 |
|
| 2016-01-18 13:34:17 |
Nastya Urlapova |
fuel/8.0.x: assignee |
|
Sergii Golovatiuk (sgolovatiuk) |
|
| 2016-01-18 13:34:24 |
Nastya Urlapova |
fuel/8.0.x: importance |
Undecided |
Critical |
|
| 2016-01-18 13:34:34 |
Nastya Urlapova |
fuel/8.0.x: status |
New |
In Progress |
|
| 2016-01-18 17:11:38 |
Sergii Golovatiuk |
fuel/mitaka: status |
In Progress |
Fix Committed |
|
| 2016-01-18 17:11:40 |
Sergii Golovatiuk |
fuel/8.0.x: status |
In Progress |
Fix Committed |
|
| 2016-01-20 10:31:23 |
Sergii Golovatiuk |
fuel/mitaka: status |
Fix Committed |
In Progress |
|
| 2016-01-20 10:31:31 |
Sergii Golovatiuk |
fuel/mitaka: status |
In Progress |
Fix Committed |
|
| 2016-01-22 07:32:26 |
Tatyanka |
fuel/8.0.x: status |
Fix Committed |
Fix Released |
|
| 2016-03-18 09:00:25 |
Aleksei Stepanov |
fuel/mitaka: status |
Fix Committed |
Fix Released |
|
