OpenStack-Ansible

openstack-ansible setup-hosts.yml fails in task: lxc_hosts Ensure image has been pre-staged

Bug #1810584 reported by Andreas Florath
38
This bug affects 8 people
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Low
lei zhang

Bug Description

The first call to

$ openstack-ansible -v -v setup-hosts.yml

fails always when running the command for the first time.
Running the same command a second time, the command succeeds.

I have three controller nodes.

TASK [lxc_hosts : Ensure image has been pre-staged] ****************************
task path: /etc/ansible/roles/lxc_hosts/tasks/lxc_cache_preparation_systemd_new.yml:26
<controller2> The "physical_host" variable of "controller2" has been found to have a corresponding host entry in inventory.
<controller2> The "physical_host" variable of "controller2" terminates at "192.168.72.20" using the host variable "ansible_host".
<controller1> The "physical_host" variable of "controller1" has been found to have a corresponding host entry in inventory.
<controller1> The "physical_host" variable of "controller1" terminates at "192.168.72.19" using the host variable "ansible_host".
<controller0> The "physical_host" variable of "controller0" has been found to have a corresponding host entry in inventory.
<controller0> The "physical_host" variable of "controller0" terminates at "192.168.72.18" using the host variable "ansible_host".
changed: [controller2] => {"ansible_job_id": "652801574311.37901", "attempts": 1, "changed": true, "cmd": "aria2c --max-connection-per-server=4 --allow-overwrite=true --dir=/tmp --out=ubuntu-base-18.04.1-base-amd64.tar.gz --check-certificate=true http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.1-base-amd64.tar.gz > /var/log/aria2c-image-prestage.log 2>&1", "delta": "0:00:00.569013", "end": "2019-01-05 09:05:12.029765", "finished": 1, "rc": 0, "start": "2019-01-05 09:05:11.460752", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
FAILED - RETRYING: Ensure image has been pre-staged (60 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (60 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (59 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (59 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (58 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (58 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (57 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (57 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (56 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (56 retries left).
...
FAILED - RETRYING: Ensure image has been pre-staged (1 retries left).
FAILED - RETRYING: Ensure image has been pre-staged (1 retries left).
fatal: [controller1]: FAILED! => {"ansible_job_id": "992874646118.37860", "attempts": 60, "changed": false, "finished": 0, "started": 1}
fatal: [controller0]: FAILED! => {"ansible_job_id": "226474098475.37911", "attempts": 60, "changed": false, "finished": 0, "started": 1}

/var/log/aria2c-image-prestage.log on controller 1 (which fails):

[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
[#0424ac 0B/0B CN:1 DL:0B]
 *** Download Progress Summary as of Sat Jan 5 09:09:11 2019 ***
===============================================================================
[#0424ac 0B/0B CN:1 DL:0B]
FILE: /tmp/ubuntu-base-18.04.1-base-amd64.tar.gz
-------------------------------------------------------------------------------

/var/log/aria2c-image-prestage.log on controller 2 (which succeeds):

01/05 09:05:11 [NOTICE] Downloading 1 item(s)

01/05 09:05:12 [NOTICE] Download complete: /tmp/ubuntu-base-18.04.1-base-amd64.tar.gz

Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
e1c289|OK | 64MiB/s|/tmp/ubuntu-base-18.04.1-base-amd64.tar.gz

Status Legend:
(OK):download completed.

After the second run, the files on all controllers look very similar to the controller 2 one.

Revision history for this message
JohnMa (zud0ku) wrote :

I had the same problem, I thought it was my connection but It seems to be a real issue.

Revision history for this message
bel (varr) wrote :

Also i had same issue.

Revision history for this message
Mohammed Naser (mnaser) wrote :

This seems to be a third-party failure (cdimage.ubuntu.com) having problems, we consider this a low bug because we should have some retries.

Changed in openstack-ansible:
status: New → Confirmed
importance: Undecided → Low
lei zhang (zhang-lei)
Changed in openstack-ansible:
assignee: nobody → lei zhang (zhang-lei)
status: Confirmed → In Progress
Revision history for this message
Patrick Petit (patrick-michel-petit) wrote :

I have the same problem. Execution of the setup-hosts.yml fails systematically with not attempt to retry.

TASK [lxc_hosts : Ensure image has been pre-staged] ************************************************************************************************
fatal: [Opennext09.opennext.local]: FAILED! => {"ansible_job_id": "806206899545.14933", "attempts": 1, "changed": true, "cmd": "aria2c --max-connection-per-server=4 --allow-overwrite=true --dir=/tmp --out=ubuntu-base-16.04.4-base-amd64.tar.gz --check-certificate=true http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.4-base-amd64.tar.gz > /var/log/aria2c-image-prestage.log 2>&1", "delta": "0:00:00.013248", "end": "2019-02-06 17:06:37.000083", "finished": 1, "msg": "non-zero return code", "rc": 3, "start": "2019-02-06 17:06:36.986835", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

Funnily enough it works fine when run manually on the target node...

The error logged on the target node is the following.

02/06 17:06:37 [^[[1;31mERROR^[[0m] CUID#6 - Download aborted. URI=http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.4-base-amd64.tar.gz
Exception: [AbstractCommand.cc:350] errorCode=3 URI=http://cdimage.ubuntu.com/ubuntu-base/releases/16.04/release/ubuntu-base-16.04.4-base-amd64.tar.gz
  -> [HttpSkipResponseCommand.cc:215] errorCode=3 Resource not found

02/06 17:06:37 [^[[1;32mNOTICE^[[0m] Download GID#e086bcf32ee35ace not complete: /tmp/ubuntu-base-16.04.4-base-amd64.tar.gz

Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
e086bc|ERR | 0B/s|/tmp/ubuntu-base-16.04.4-base-amd64.tar.gz

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

Revision history for this message
Patrick Petit (patrick-michel-petit) wrote :

Please disregard my above bug filing. My problem is different although with similar a symptom.

Revision history for this message
XiaoRuiguo (705437650-6) wrote :

Also i had same issue.

Revision history for this message
Keith Plant (kplant) wrote :

I have a similar issue on CentOS:

CUID#7 - Download aborted. URI=https://github.com/CentOS/sig-cloud-instance-images/raw/CentOS-7/docker/centos-7-docker.tar.xz
Exception: [AbstractCommand.cc:351] errorCode=1 URI=https://github.com/CentOS/sig-cloud-instance-images/raw/CentOS-7/docker/centos-7-docker.tar.xz
  -> [SocketCore.cc:932] errorCode=1 SSL initialization failed: The request is invalid.

Changing from aria2c to wget alleviates the issue.

Revision history for this message
Pride Njukia (cnjukia) wrote :

Having the same issue on centos 7, please share how this was resolved.

[root@aio1 playbooks]# cat /var/log/aria2c-image-prestage.log

05/09 09:14:38 [NOTICE] Downloading 1 item(s)

05/09 09:14:39 [ERROR] CUID#7 - Download aborted. URI=https://github.com/CentOS/sig-cloud-instance-images/raw/CentOS-7/docker/centos-7-docker.tar.xz
Exception: [AbstractCommand.cc:351] errorCode=1 URI=https://github.com/CentOS/sig-cloud-instance-images/raw/CentOS-7/docker/centos-7-docker.tar.xz
  -> [SocketCore.cc:932] errorCode=1 SSL initialization failed: The request is invalid.

05/09 09:14:39 [NOTICE] Download GID#eba28bd0550804d7 not complete: /tmp/centos-7-docker.tar.xz

Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
eba28b|ERR | 0B/s|/tmp/centos-7-docker.tar.xz

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

Arun Badri (arun-badri)
Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
Alberto Corona (albcoron) wrote :

What release was a fix for this released? This exact same task fails for me on 19.1.0, with the _exact_ same aria2c log output as initially reported...

Revision history for this message
Alberto Corona (albcoron) wrote :

When creating the aria2c command that the role lxc_hosts runs to pull the image the download fails as well. It could very well be sporadic errors with aria2c's ability to pull the image as I can very easily download the image with wget. I don't quite understand why something like aria2c was used here instead of just pulling this with curl or wget...

Revision history for this message
Mahmoud abdelhamid (mhamid) wrote :

I am facing the same bug now, but this time the ubuntu image doesn't exist, I even can't download it using wget.

TASK [lxc_hosts : Ensure image has been pre-staged] ***************************************************************************************************************************
fatal: [compute4]: FAILED! => {"ansible_job_id": "413027832344.4796", "attempts": 1, "changed": true, "cmd": "aria2c --max-connection-per-server=4 --allow-overwrite=true --dir=/tmp --out=ubuntu-base-18.04.3-base-amd64.tar.gz --check-certificate=true --follow-metalink=mem --on-download-complete=/usr/local/bin/aria2c-download-helper.sh http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.3-base-amd64.tar.gz > /var/log/aria2c-image-prestage.log 2>&1\n", "delta": "0:00:00.188820", "end": "2020-11-10 12:09:38.792790", "finished": 1, "msg": "non-zero return code", "rc": 3, "start": "2020-11-10 12:09:38.603970", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

oot@compute4:~# cat /var/log/aria2c-image-prestage.log

11/10 12:09:38 [NOTICE] Downloading 1 item(s)

11/10 12:09:38 [ERROR] CUID#7 - Download aborted. URI=http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.3-base-amd64.tar.gz
Exception: [AbstractCommand.cc:351] errorCode=3 URI=http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.3-base-amd64.tar.gz
  -> [HttpSkipResponseCommand.cc:219] errorCode=3 Resource not found

11/10 12:09:38 [NOTICE] Download GID#8f714631ead1ec85 not complete: /tmp/ubuntu-base-18.04.3-base-amd64.tar.gz

Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
8f7146|ERR | 0B/s|/tmp/ubuntu-base-18.04.3-base-amd64.tar.gz

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
root@compute4:~# wget http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.3-base-amd64.tar.gz
--2020-11-10 12:34:52-- http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.3-base-amd64.tar.gz
Resolving cdimage.ubuntu.com (cdimage.ubuntu.com)... 91.189.91.124, 91.189.88.248, 91.189.88.247, ...
Connecting to cdimage.ubuntu.com (cdimage.ubuntu.com)|91.189.91.124|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-11-10 12:34:53 ERROR 404: Not Found.

Why aria2 trying to download statically ubuntu-base-18.04.3 though a newer version exist ubuntu-base-18.04.5?

Revision history for this message
Mahmoud abdelhamid (mhamid) wrote :

The reported symptom in the previous comment is related to another bug, not related to this bug:
https://bugs.launchpad.net/openstack-ansible/+bug/1900975

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.