[master][queens][pike][ocata] Base periodic jobs are broken after zuul v3 migration

Bug #1780726 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Sagi (Sergey) Shnaidman

Bug Description

Some bits are missed in zuulv3 migration related to secrets, so base periodic jobs(promote-consistent-to-tripleo-ci-testing, containers-build[detected from testing]) for all releases are broken.

Logs:
http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-queens-promote-consistent-to-tripleo-ci-testing/f20f5c9/job-output.txt.gz
and
http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing/707cd31/job-output.txt.gz

2018-07-09 04:52:02.739278 | PLAY [primary]
2018-07-09 04:52:02.772148 |
2018-07-09 04:52:02.772309 | TASK [shell]
2018-07-09 04:52:03.289977 | primary | Output suppressed because no_log was given

http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-pike-promote-consistent-to-tripleo-ci-testing/a620d80/job-output.txt.gz
and
http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-centos-7-ocata-promote-consistent-to-tripleo-ci-testing/66a36fa/job-output.txt.gz

2018-07-09 04:07:04.705642 | primary | + dlrnapi --url https://trunk.rdoproject.org/api-centos-pike --username review_rdoproject_org repo-promote --commit-hash df13125683ba1168cc3883479ca2ab849d53ba07 --distro-hash 40c446c1ccbf8ae479ae6e0e17099d707d93118e --promote-name tripleo-ci-testing
2018-07-09 04:07:04.772943 | primary | Traceback (most recent call last):
2018-07-09 04:07:04.773104 | primary | File "/home/zuul/workspace/dlrnapi_venv/bin/dlrnapi", line 11, in <module>
2018-07-09 04:07:04.773187 | primary | sys.exit(main())
2018-07-09 04:07:04.773363 | primary | File "/home/zuul/workspace/dlrnapi_venv/lib/python2.7/site-packages/dlrnapi_client/shell.py", line 342, in main
2018-07-09 04:07:04.773493 | primary | if e.status == 404:
2018-07-09 04:07:04.773664 | primary | AttributeError: 'exceptions.TypeError' object has no attribute 'status'
2018-07-09 04:07:04.780082 | primary | + deactivate_dlrnapi_venv
2018-07-09 04:07:04.780252 | primary | + [[ /home/zuul/workspace/dlrnapi_venv = /home/zuul/workspace/dlrnapi_venv ]]

^^ fixed by https://review.rdoproject.org/r/14694 and https://review.rdoproject.org/r/#/c/14701/

While testing zuulv3 detected in testing project:
http://logs.rdoproject.org/43/13943/9/check/legacy-periodic-tripleo-centos-7-master-containers-build/a22de66/job-output.txt.gz

:Attempt number: 2 to run task: PushTask(base) \nINFO:kolla.common.utils.base:Trying to push the image\nINFO:kolla.common.utils.iscsid:Determining fastest mirrors\nERROR:kolla.common.utils.base:unauthorized: authentication required\nINFO:kolla.common.utils:Attempt number: 3 to run task: PushTask(base) \nINFO:kolla.common.utils.base:Trying to push the image\nERROR:kolla.common.utils.base:unauthorized: authentication required\nINFO:kolla.common.utils:Attempt number: 4 to run task: PushTask(base) \nINFO:kolla.common.utils.base:Trying to push the image\nINFO:kolla.common.utils.openstack-base:Loaded plugins: fastestmirror, ovl, priorities\nINFO:kolla.common.utils.etcd: ---> 1b5743b31bc3\nINFO:kolla.common.utils.openvswitch-base:Loaded plugins: fastestmirror, ovl, priorities\nINFO:kolla.common.utils.etcd:Removing intermediate container

Attempting by fixing here: https://review.rdoproject.org/r/14699

We are hitting subsequent failures after zuulv3 migration so moving forward by fixing while we encounter these issues.

Note: This bug will be used to track all the zuulv3 issues we encountered from now on.

tags: added: alert
description: updated
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

RDO registry password issue was fixed in https://review.rdoproject.org/r/#/c/14732/

Changed in tripleo:
assignee: nobody → Sagi (Sergey) Shnaidman (sshnaidm)
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

OVB jobs issues with hosts is fixed here: https://review.rdoproject.org/r/#/c/14736/

Revision history for this message
wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

Hosts issue for reporting task for ovb jobs is fixed here: https://review.rdoproject.org/r/#/c/14737/

Revision history for this message
chandan kumar (chkumar246) wrote :
Revision history for this message
yatin (yatinkarel) wrote :
Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :
Revision history for this message
yatin (yatinkarel) wrote :
Revision history for this message
Alan Pevec (apevec) wrote :

one more followup fix for images upload: https://review.rdoproject.org/r/14786

Revision history for this message
chandan kumar (chkumar246) wrote :

permission denied issue in promotion logs
+ PROMOTED_HASH=f84ce61bb1e7f9cd62c73cc8c14f01644989c279_1346e217
+ LINK_NAME=current-tripleo
+ sftp_command 'rm /var/www/html/images/queens/rdo_trunk/previous-current-tripleo'
+ echo 'rm /var/www/html/images/queens/rdo_trunk/previous-current-tripleo'
+ sftp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null <email address hidden>
Warning: Permanently added 'images.rdoproject.org,38.145.33.168' (ECDSA) to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
Couldn't read packet: Connection reset by peer

2018-07-12 04:34:10,859 8737 ERROR promoter Command '['bash', '/home/centos/ci-config/ci-scripts/promote-images.sh', u'queens', 'f84ce61bb1e7f9cd62c73cc8c14f01644989c279_1346e217', u'current-tripleo']' returned non-zero exit status 255
Traceback (most recent call last):
  File "/home/centos/ci-config/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py", line 168, in tag_qcow_images
    stderr=subprocess.STDOUT
  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)

in http://38.145.34.55/queens.log and http://38.145.34.55/master.log

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

@chandan, it's not a bug. I didn't update the key intentionally not to touch images server until we fix everything. Now it has a right key.

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

Last issue with post yaml playbook was addressed: https://review.rdoproject.org/r/#/c/14807/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.