tripleo

[gate] Error pulling image: invalid character '<' parsing 404 response

Bug #1819632 reported by Rafael Folco on 2019-03-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Medium	Emilien Macchi	tripleo ussuri-3 "tripleo ussuri-3"

Bug Description

gate
tripleo-ci-centos-7-scenario004-standalone

http://logs.openstack.org/90/583890/5/gate/tripleo-ci-centos-7-scenario004-standalone/fac9d52/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-03-11_23_51_47

stderr: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/tripleomaster/centos-binary-nova-libvirt/manifests/current-tripleo-updated-20190311224352 was not found on this server.</p>\\n</body></html>\\n\""

----

2019-03-11 23:51:47 | TASK [Start containers for step 3] *********************************************
2019-03-11 23:51:52 | ok: [standalone]
2019-03-11 23:51:52 |
2019-03-11 23:51:52 | TASK [Debug output for task: Start containers for step 3] **********************
2019-03-11 23:51:52 | fatal: [standalone]: FAILED! => {
2019-03-11 23:51:52 | "failed_when_result": true,
2019-03-11 23:51:52 | "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
2019-03-11 23:51:52 | "docker runtime is deprecated in Stein and will be removed in Train.",
2019-03-11 23:51:52 | "image_exist isn't supported by docker",
2019-03-11 23:51:52 | "Error pulling 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt:current-tripleo-updated-20190311224352. [1]",
2019-03-11 23:51:52 | "",
2019-03-11 23:51:52 | "stdout: Trying to pull repository 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt ... ",
2019-03-11 23:51:52 | "stderr: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/tripleomaster/centos-binary-nova-libvirt/manifests/current-tripleo-updated-20190311224352 was not found on this server.</p>\\n</body></html>\\n\""
2019-03-11 23:51:52 | ]
2019-03-11 23:51:52 | }
2019-03-11 23:51:52 |
2019-03-11 23:51:52 | NO MORE HOSTS LEFT *************************************************************
2019-03-11 23:51:52 |
2019-03-11 23:51:52 | PLAY RECAP *********************************************************************
2019-03-11 23:51:52 | standalone : ok=229 changed=136 unreachable=0 failed=1
2019-03-11 23:51:52 | undercloud : ok=41 changed=23 unreachable=0 failed=0
2019-03-11 23:51:52 |
2019-03-11 23:51:52 | Exception: Deployment failed
2019-03-11 23:51:52 | Traceback (most recent call last):
2019-03-11 23:51:52 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1256, in _standalone_deploy
2019-03-11 23:51:52 | raise exceptions.DeploymentError('Deployment failed')
2019-03-11 23:51:52 | DeploymentError: Deployment failed
2019-03-11 23:51:52 | None
2019-03-11 23:51:54 | ** Found ansible errors for standalone deployment! **
2019-03-11 23:51:54 | [
2019-03-11 23:51:54 | [
2019-03-11 23:51:54 | "Debug output for task: Start containers for step 3",
2019-03-11 23:51:54 | {
2019-03-11 23:51:54 | "changed": false,
2019-03-11 23:51:54 | "failed_when_result": true,
2019-03-11 23:51:54 | "_ansible_no_log": false,
2019-03-11 23:51:54 | "_ansible_verbose_always": true,
2019-03-11 23:51:54 | "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
2019-03-11 23:51:54 | "docker runtime is deprecated in Stein and will be removed in Train.",
2019-03-11 23:51:54 | "image_exist isn't supported by docker",
2019-03-11 23:51:54 | "Error pulling 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt:current-tripleo-updated-20190311224352. [1]",
2019-03-11 23:51:54 | "",
2019-03-11 23:51:54 | "stdout: Trying to pull repository 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt ... ",
2019-03-11 23:51:54 | "stderr: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/tripleomaster/centos-binary-nova-libvirt/manifests/current-tripleo-updated-20190311224352 was not found on this server.</p>\\n</body></html>\\n\""
2019-03-11 23:51:54 | ]
2019-03-11 23:51:54 | }
2019-03-11 23:51:54 | ]
2019-03-11 23:51:54 | ]
2019-03-11 23:51:54 | Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
2019-03-11 23:51:54 | Not cleaning ansible directory /home/zuul/undercloud-ansible-lkvFDg
2019-03-11 23:51:54 | Install artifact is located at /home/zuul/undercloud-install-20190311235152.tar.bzip2
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | Deployment Failed!
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-dsn3Hu
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | Deployment failed.

Revision history for this message

Emilien Macchi (emilienm) wrote on 2019-03-12:

It looks like the nova-libvirt container was pushed with some errors:
http://logs.openstack.org/90/583890/5/gate/tripleo-ci-centos-7-scenario004-standalone/fac9d52/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz?level=ERROR#_2019-03-11_23_29_12_505

Revision history for this message

Emilien Macchi (emilienm) wrote on 2019-03-12:

The real error is here:

http://logs.openstack.org/90/583890/5/gate/tripleo-ci-centos-7-scenario004-standalone/fac9d52/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz?#_2019-03-11_23_29_33_320

2019-03-11 23:29:33.320 20725 DEBUG urllib3.connectionpool [ ] http://mirror.bhs1.ovh.openstack.org:8081 "GET /registry-1.docker/v2/tripleomaster/centos-binary-nova-libvirt/blobs/sha256:299c9c8e48d91e5b3ce1ade9d3dcf3f88cd1a21679ca94778d588c2d364960c4 HTTP/1.1" 401 183
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file tripleo-container-image-prepare, line 136
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting

Also, note that the error wasn't logged as ERROR, we also need to fix that.

Revision history for this message

Emilien Macchi (emilienm) wrote on 2019-03-12:

So 2 things:

- We need to find out why the OVH mirror returns 401
- We need to patch tripleo-common to fix logging and return ERROR when this trace occurs.

Changed in tripleo:
assignee:	Gabriele Cerami (gcerami) → Emilien Macchi (emilienm)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-12: Related fix proposed to tripleo-common (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/642741

Emilien Macchi (emilienm) on 2019-03-12

Changed in tripleo:
status:	Triaged → In Progress

wes hayutin (weshayutin) on 2019-03-12

tags:

added: alert

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-12: Related fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/642741
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d6abcac8dca1e57f1cede2a16359bcd9db857dca
Submitter: Zuul
Branch: master

commit d6abcac8dca1e57f1cede2a16359bcd9db857dca
Author: Emilien Macchi <email address hidden>
Date: Tue Mar 12 08:58:40 2019 -0400

tripleo-container-image-prepare/logging: fix exception formating

    It happens that the remote registry returns 401 and we want to log an
    exception but the formating wasn't converted to strings.
    So we had this error:

TypeError: not all arguments converted during string formatting

With this patch, we use {} code and the .format method to make sure that
the exception will be composed of strings only.

Note: it doesn't fix the root cause of the 401 error reported in bug 1819632
but it should help to figure it out with proper logging.

Change-Id: Idad301b6da30143827da1cfc8083f460073442fe
Related-Bug: #1819632

Emilien Macchi (emilienm) on 2019-03-14

Changed in tripleo:
importance:	Critical → Medium
tags:	removed: alert ci

Alex Schultz (alex-schultz) on 2019-03-14

Changed in tripleo:
milestone:	stein-3 → train-1

wes hayutin (weshayutin) on 2019-04-10

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

Rafael Folco (rafaelfolco) wrote on 2019-04-11:

Reopening as the commit says it was not fixed. Also, seeing similar (if not the same) error when we switched to buildah...
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario001-standalone-stein/4bf87d6/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-04-11_15_38_24

Changed in tripleo:
status:	Fix Released → Won't Fix
status:	Won't Fix → Triaged

Revision history for this message

Emilien Macchi (emilienm) wrote on 2019-04-11:

Do you have https://review.openstack.org/#/c/650985/ ? If no, then it's not good. You need this patch with latest buildah/podman.

Revision history for this message

Emilien Macchi (emilienm) wrote on 2019-04-11:

in fact my theory on this one is that we are missing https://review.rdoproject.org/r/#/c/20152/ and you already have https://review.openstack.org/#/c/650985/ in stein.

Alex Schultz (alex-schultz) on 2019-06-07

Changed in tripleo:
milestone:	train-1 → train-2

Alex Schultz (alex-schultz) on 2019-07-29

Changed in tripleo:
milestone:	train-2 → train-3

Alex Schultz (alex-schultz) on 2019-09-11

Changed in tripleo:
milestone:	train-3 → ussuri-1

Emilien Macchi (emilienm) on 2019-12-19

Changed in tripleo:
milestone:	ussuri-1 → ussuri-2

wes hayutin (weshayutin) on 2020-02-10

Changed in tripleo:
milestone:	ussuri-2 → ussuri-3

Emilien Macchi (emilienm) on 2020-03-30

Changed in tripleo:
status:	Triaged → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1819631

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.