DLRN promoter failed to create manifest for neutron-metadata-agent-ovn container in c7 train

Bug #1887660 reported by Sandeep Yadav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Critical
Unassigned

Bug Description

DLRN promoter failed to create manifest for neutron-metadata-agent-ovn container in c7 train

Url:-

http://<promoter ip>/centos7_train.log

~~~
failed: [localhost] (item=neutron-metadata-agent-ovn) => {"ansible_index_var": "index", "ansible_loop_var": "item", "changed": true, "cmd": "docker manifest create docker.io/tripleotrain/centos-binary-neutron-metadata-agent-ovn:7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112_manifest docker.io/tripleotrain/centos-binary-neutron-metadata-agent-ovn:7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112_x86_64\ndocker manifest annotate --arch amd64 docker.io/tripleotrain/centos-binary-neutron-metadata-agent-ovn:7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112_manifest docker.io/tripleotrain/centos-binary-neutron-metadata-agent-ovn:7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112_x86_64\n", "delta": "0:00:00.126053", "end": "2020-07-15 10:23:11.218634", "index": 60, "item": "neutron-metadata-agent-ovn", "msg": "non-zero return code", "rc": 1, "start": "2020-07-15 10:23:11.092581", "stderr": "unexpected end of JSON input\nunexpected end of JSON input", "stderr_lines": ["unexpected end of JSON input", "unexpected end of JSON input"], "stdout": "", "stdout_lines": []}
~~~

~~~
020-07-15 10:33:18,311 23906 ERROR promoter Command '[u'env', u'ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20200715-092451.log', u'RELEASE=train', u'COMMIT_HASH=7d0406b1a2bb054f42b198e9494ddc54372e7285', u'DISTRO_HASH=6e7a0112dd009a51232410123e16527f2e4e1218', u'FULL_HASH=7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112', u'PROMOTE_NAME=current-tripleo', u'SCRIPT_ROOT=/home/centos/ci-config/', u'DISTRO_NAME=centos', u'DISTRO_VERSION=7', u'ansible-playbook', u'-e', u'manifest_push=True', u'-e', u'target_registries_push=True', u'/home/centos/ci-config/ci-scripts/container-push/container-push.yml']' returned non-zero exit status 2
Traceback (most recent call last):
  File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/legacy_promoter.py", line 213, in tag_containers
    stderr=subprocess.STDOUT).split("\n")
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '[u'env', u'ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20200715-092451.log', u'RELEASE=train', u'COMMIT_HASH=7d0406b1a2bb054f42b198e9494ddc54372e7285', u'DISTRO_HASH=6e7a0112dd009a51232410123e16527f2e4e1218', u'FULL_HASH=7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112', u'PROMOTE_NAME=current-tripleo', u'SCRIPT_ROOT=/home/centos/ci-config/', u'DISTRO_NAME=centos', u'DISTRO_VERSION=7', u'ansible-playbook', u'-e', u'manifest_push=True', u'-e', u'target_registries_push=True', u'/home/centos/ci-config/ci-scripts/container-push/container-push.yml']' returned non-zero exit status 2
2020-07-15 10:33:18,312 23906 ERROR promoter END OF CONTAINER IMAGE UPLOAD FAILURE
2020-07-15 10:33:18,312 23906 INFO promoter FAILED promoting centos7-train tripleo-ci-testing as current-tripleo (commit: 7d0406b1a2bb054f42b198e9494ddc54372e7285, distro: 6e7a0112dd009a51232410123e16527f2e4e1218, timestamp=1594720874)
2020-07-15 10:33:18,312 23906 INFO promoter ('%s\n%s/api/civotes_detail.html?commit_hash=%s&distro_hash=%s', 'DETAILED FAILED STATUS: ', u'https://trunk.rdoproject.org/api-centos-train', '7d0406b1a2bb054f42b198e9494ddc54372e7285', '6e7a0112dd009a51232410123e16527f2e4e1218')
2020-07-15 10:33:18,312 23906 ERROR promoter Command '[u'env', u'ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20200715-092451.log', u'RELEASE=train', u'COMMIT_HASH=7d0406b1a2bb054f42b198e9494ddc54372e7285', u'DISTRO_HASH=6e7a0112dd009a51232410123e16527f2e4e1218', u'FULL_HASH=7d0406b1a2bb054f42b198e9494ddc54372e7285_6e7a0112', u'PROMOTE_NAME=current-tripleo', u'SCRIPT_ROOT=/home/centos/ci-config/', u'DISTRO_NAME=centos', u'DISTRO_VERSION=7', u'ansible-playbook', u'-e', u'manifest_push=True', u'-e', u'target_registries_push=True', u'/home/centos/ci-config/ci-scripts/container-push/container-push.yml']' returned non-zero exit status 2
Traceback (most recent call last):
  File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py", line 78, in start_process
    logic.promote_all_links()
  File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/logic.py", line 209, in promote_all_links
    self.promote_label_to_label(candidate_label, target_label)
  File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/logic.py", line 158, in promote_label_to_label
    self.promote(selected_candidate, target_label)
  File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/logic.py", line 124, in promote
    target_label)
  File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/registry.py", line 35, in promote_containers
    self.config.target_registries_push)
  File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/legacy_promoter.py", line 213, in tag_containers
    stderr=subprocess.STDOUT).split("\n")
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
~~~

Tags: alert ci
Revision history for this message
Sandeep Yadav (sandeepyadav93) wrote :

On discussion with senior collegues:-

If there is a problem pushing manifests for now we might just turn that off, manifests are only needed for ppc64le containers tagged but they aren't available so its safe to have them off.

* Came to know that pushing manifests is already off in master/usurri branch

https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-8/master.ini#L11
https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-8/ussuri.ini#L11

Proposed: https://review.rdoproject.org/r/#/c/28540/

Changed in tripleo:
milestone: victoria-1 → victoria-3
Changed in tripleo:
milestone: victoria-3 → wallaby-1
Changed in tripleo:
milestone: wallaby-1 → wallaby-2
Changed in tripleo:
milestone: wallaby-2 → wallaby-3
Changed in tripleo:
milestone: wallaby-3 → wallaby-rc1
Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Changed in tripleo:
milestone: xena-1 → xena-2
Changed in tripleo:
milestone: xena-2 → xena-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.