[gate] Error pulling image: invalid character '<' parsing 404 response

Bug #1819632 reported by Rafael Folco on 2019-03-12
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Medium
Emilien Macchi

Bug Description

gate
tripleo-ci-centos-7-scenario004-standalone

http://logs.openstack.org/90/583890/5/gate/tripleo-ci-centos-7-scenario004-standalone/fac9d52/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-03-11_23_51_47

stderr: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/tripleomaster/centos-binary-nova-libvirt/manifests/current-tripleo-updated-20190311224352 was not found on this server.</p>\\n</body></html>\\n\""

----

2019-03-11 23:51:47 | TASK [Start containers for step 3] *********************************************
2019-03-11 23:51:52 | ok: [standalone]
2019-03-11 23:51:52 |
2019-03-11 23:51:52 | TASK [Debug output for task: Start containers for step 3] **********************
2019-03-11 23:51:52 | fatal: [standalone]: FAILED! => {
2019-03-11 23:51:52 | "failed_when_result": true,
2019-03-11 23:51:52 | "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
2019-03-11 23:51:52 | "docker runtime is deprecated in Stein and will be removed in Train.",
2019-03-11 23:51:52 | "image_exist isn't supported by docker",
2019-03-11 23:51:52 | "Error pulling 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt:current-tripleo-updated-20190311224352. [1]",
2019-03-11 23:51:52 | "",
2019-03-11 23:51:52 | "stdout: Trying to pull repository 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt ... ",
2019-03-11 23:51:52 | "stderr: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/tripleomaster/centos-binary-nova-libvirt/manifests/current-tripleo-updated-20190311224352 was not found on this server.</p>\\n</body></html>\\n\""
2019-03-11 23:51:52 | ]
2019-03-11 23:51:52 | }
2019-03-11 23:51:52 |
2019-03-11 23:51:52 | NO MORE HOSTS LEFT *************************************************************
2019-03-11 23:51:52 |
2019-03-11 23:51:52 | PLAY RECAP *********************************************************************
2019-03-11 23:51:52 | standalone : ok=229 changed=136 unreachable=0 failed=1
2019-03-11 23:51:52 | undercloud : ok=41 changed=23 unreachable=0 failed=0
2019-03-11 23:51:52 |
2019-03-11 23:51:52 | Exception: Deployment failed
2019-03-11 23:51:52 | Traceback (most recent call last):
2019-03-11 23:51:52 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1256, in _standalone_deploy
2019-03-11 23:51:52 | raise exceptions.DeploymentError('Deployment failed')
2019-03-11 23:51:52 | DeploymentError: Deployment failed
2019-03-11 23:51:52 | None
2019-03-11 23:51:54 | ** Found ansible errors for standalone deployment! **
2019-03-11 23:51:54 | [
2019-03-11 23:51:54 | [
2019-03-11 23:51:54 | "Debug output for task: Start containers for step 3",
2019-03-11 23:51:54 | {
2019-03-11 23:51:54 | "changed": false,
2019-03-11 23:51:54 | "failed_when_result": true,
2019-03-11 23:51:54 | "_ansible_no_log": false,
2019-03-11 23:51:54 | "_ansible_verbose_always": true,
2019-03-11 23:51:54 | "outputs.stdout_lines | default([]) | union(outputs.stderr_lines | default([]))": [
2019-03-11 23:51:54 | "docker runtime is deprecated in Stein and will be removed in Train.",
2019-03-11 23:51:54 | "image_exist isn't supported by docker",
2019-03-11 23:51:54 | "Error pulling 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt:current-tripleo-updated-20190311224352. [1]",
2019-03-11 23:51:54 | "",
2019-03-11 23:51:54 | "stdout: Trying to pull repository 192.168.24.1:8787/tripleomaster/centos-binary-nova-libvirt ... ",
2019-03-11 23:51:54 | "stderr: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: \"<!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>404 Not Found</title>\\n</head><body>\\n<h1>Not Found</h1>\\n<p>The requested URL /v2/tripleomaster/centos-binary-nova-libvirt/manifests/current-tripleo-updated-20190311224352 was not found on this server.</p>\\n</body></html>\\n\""
2019-03-11 23:51:54 | ]
2019-03-11 23:51:54 | }
2019-03-11 23:51:54 | ]
2019-03-11 23:51:54 | ]
2019-03-11 23:51:54 | Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
2019-03-11 23:51:54 | Not cleaning ansible directory /home/zuul/undercloud-ansible-lkvFDg
2019-03-11 23:51:54 | Install artifact is located at /home/zuul/undercloud-install-20190311235152.tar.bzip2
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | Deployment Failed!
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-dsn3Hu
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-03-11 23:51:54 |
2019-03-11 23:51:54 | Deployment failed.

Emilien Macchi (emilienm) wrote :

The real error is here:

http://logs.openstack.org/90/583890/5/gate/tripleo-ci-centos-7-scenario004-standalone/fac9d52/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz?#_2019-03-11_23_29_33_320

2019-03-11 23:29:33.320 20725 DEBUG urllib3.connectionpool [ ] http://mirror.bhs1.ovh.openstack.org:8081 "GET /registry-1.docker/v2/tripleomaster/centos-binary-nova-libvirt/blobs/sha256:299c9c8e48d91e5b3ce1ade9d3dcf3f88cd1a21679ca94778d588c2d364960c4 HTTP/1.1" 401 183
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file tripleo-container-image-prepare, line 136
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib64/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting

Also, note that the error wasn't logged as ERROR, we also need to fix that.

Emilien Macchi (emilienm) wrote :

So 2 things:

- We need to find out why the OVH mirror returns 401
- We need to patch tripleo-common to fix logging and return ERROR when this trace occurs.

Changed in tripleo:
assignee: Gabriele Cerami (gcerami) → Emilien Macchi (emilienm)
Changed in tripleo:
status: Triaged → In Progress
wes hayutin (weshayutin) on 2019-03-12
tags: added: alert

Reviewed: https://review.openstack.org/642741
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d6abcac8dca1e57f1cede2a16359bcd9db857dca
Submitter: Zuul
Branch: master

commit d6abcac8dca1e57f1cede2a16359bcd9db857dca
Author: Emilien Macchi <email address hidden>
Date: Tue Mar 12 08:58:40 2019 -0400

    tripleo-container-image-prepare/logging: fix exception formating

    It happens that the remote registry returns 401 and we want to log an
    exception but the formating wasn't converted to strings.
    So we had this error:

      TypeError: not all arguments converted during string formatting

    With this patch, we use {} code and the .format method to make sure that
    the exception will be composed of strings only.

    Note: it doesn't fix the root cause of the 401 error reported in bug 1819632
    but it should help to figure it out with proper logging.

    Change-Id: Idad301b6da30143827da1cfc8083f460073442fe
    Related-Bug: #1819632

Changed in tripleo:
importance: Critical → Medium
tags: removed: alert ci
Changed in tripleo:
milestone: stein-3 → train-1
wes hayutin (weshayutin) on 2019-04-10
Changed in tripleo:
status: In Progress → Fix Released
Rafael Folco (rafaelfolco) wrote :
Changed in tripleo:
status: Fix Released → Won't Fix
status: Won't Fix → Triaged
Emilien Macchi (emilienm) wrote :

Do you have https://review.openstack.org/#/c/650985/ ? If no, then it's not good. You need this patch with latest buildah/podman.

Emilien Macchi (emilienm) wrote :

in fact my theory on this one is that we are missing https://review.rdoproject.org/r/#/c/20152/ and you already have https://review.openstack.org/#/c/650985/ in stein.

Changed in tripleo:
milestone: train-1 → train-2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers