all master jobs RED, dlrnapi down after partial promotion

[wes hayutin]

ERROR: Got error 404, probably endpoint https://trunk.rdoproject.org/api-centos8-master-uc is not available

https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5 IS WRONG.. we need to promote https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=ee4aecfe06de7e8ca63aed041b3e42a8

Cinder - Validations needs the following promoted to current-tripleo to bring the md5sum to ee4aecfe06de7e8ca63aed041b3e42a8

[delorean-component-baremetal]
name=delorean-openstack-ironic-python-agent-builder-609531aa681185b1ac59d3c12106b4c69181bc4c
baseurl=https://trunk.rdoproject.org/centos8/component/baremetal/60/95/609531aa681185b1ac59d3c12106b4c69181bc4c_a36069e8
enabled=1
gpgcheck=0
priority=1

[delorean-component-cinder]
name=delorean-python-cinder-tests-tempest-51b819b912591b395c0e1c844cbcc5de8bbb75e9
baseurl=https://trunk.rdoproject.org/centos8/component/cinder/51/b8/51b819b912591b395c0e1c844cbcc5de8bbb75e9_49ca67e9
enabled=1
gpgcheck=0
priority=1

[delorean-component-clients]
name=delorean-python-neutronclient-ae397565be6b734484d9e96c66f13de97762303c
baseurl=https://trunk.rdoproject.org/centos8/component/clients/ae/39/ae397565be6b734484d9e96c66f13de97762303c_817f66ee
enabled=1
gpgcheck=0
priority=1

[delorean-component-cloudops]
name=delorean-openstack-ceilometer-dd9218d932af199cad33e3480b8a4c4e6b5a1d57
baseurl=https://trunk.rdoproject.org/centos8/component/cloudops/dd/92/dd9218d932af199cad33e3480b8a4c4e6b5a1d57_bbfe9c00
enabled=1
gpgcheck=0
priority=1

[delorean-component-common]
name=delorean-openstack-kuryr-kubernetes-862deaa455abfce55a8e50993cbfd8b1e108b720
baseurl=https://trunk.rdoproject.org/centos8/component/common/86/2d/862deaa455abfce55a8e50993cbfd8b1e108b720_92f50ace
enabled=1
gpgcheck=0
priority=1

[delorean-component-compute]
name=delorean-openstack-nova-052cf963583ab7c6bbe4fcbf7bfe69f8f6733bdb
baseurl=https://trunk.rdoproject.org/centos8/component/compute/05/2c/052cf963583ab7c6bbe4fcbf7bfe69f8f6733bdb_03cf95c6
enabled=1
gpgcheck=0
priority=1

[delorean-component-glance]
name=delorean-openstack-glance-4f20e5007f3cef4b2406051744356ee56c1d3c25
baseurl=https://trunk.rdoproject.org/centos8/component/glance/4f/20/4f20e5007f3cef4b2406051744356ee56c1d3c25_875668e8
enabled=1
gpgcheck=0
priority=1

[delorean-component-manila]
name=delorean-python-manila-tests-tempest-ac53004eddfa5be7b95b2264cb181dbe2d97ec8c
baseurl=https://trunk.rdoproject.org/centos8/component/manila/ac/53/ac53004eddfa5be7b95b2264cb181dbe2d97ec8c_c3045b18
enabled=1
gpgcheck=0
priority=1

[delorean-component-network]
name=delorean-openstack-neutron-7933dae46df8bd2569261f4f6adeec26627526dd
baseurl=https://trunk.rdoproject.org/centos8/component/network/79/33/7933dae46df8bd2569261f4f6adeec26627526dd_181dc17f
enabled=1
gpgcheck=0
priority=1

[delorean-component-octavia]
name=delorean-python-octavia-tests-tempest-cac3eefc44afca458692e9dadb539c0cb17dc9e4
baseurl=https://trunk.rdoproject.org/centos8/component/octavia/ca/c3/cac3eefc44afca458692e9dadb539c0cb17dc9e4_4c153372
enabled=1
gpgcheck=0
priority=1

[delorean-component-security]
name=delorean-openstack-keystone-252733dce7ae7d335b982d36e96b64e7a29d4ef2
baseurl=https://trunk.rdoproject.org/centos8/component/security/25/27/252733dce7ae7d335b982d36e96b64e7a29d4ef2_641e26f1
enabled=1
gpgcheck=0
priority=1

[delorean-component-swift]
name=delorean-openstack-swift-e17ae735a6986ee30a00f23c2e798c5a56987b6b
baseurl=https://trunk.rdoproject.org/centos8/component/swift/e1/7a/e17ae735a6986ee30a00f23c2e798c5a56987b6b_39a3c6ac
enabled=1
gpgcheck=0
priority=1

[delorean-component-tempest]
name=delorean-openstack-tempest-aafb5900e7450316e44541655e0ecd5575a69d73
baseurl=https://trunk.rdoproject.org/centos8/component/tempest/aa/fb/aafb5900e7450316e44541655e0ecd5575a69d73_9e03ba94
enabled=1
gpgcheck=0
priority=1

[delorean-component-tripleo]
name=delorean-tripleo-operator-ansible-c1527ca9fcd5d2a50af896e2335b0af992d7a94b
baseurl=https://trunk.rdoproject.org/centos8/component/tripleo/c1/52/c1527ca9fcd5d2a50af896e2335b0af992d7a94b_2ecfbcb5
enabled=1
gpgcheck=0
priority=1

[delorean-component-ui]
name=delorean-python-django-horizon-647c2b7530f81c4a534ef1d403aa3d90ae608442
baseurl=https://trunk.rdoproject.org/centos8/component/ui/64/7c/647c2b7530f81c4a534ef1d403aa3d90ae608442_ccc43b1e
enabled=1
gpgcheck=0
priority=1

[delorean-component-validation]
name=delorean-openstack-tripleo-validations-c1277a5e5cc6af20cebe432be34d5b82624a3e6c
baseurl=https://trunk.rdoproject.org/centos8/component/validation/c1/27/c1277a5e5cc6af20cebe432be34d5b82624a3e6c_63448dc7
enabled=1
gpgcheck=0
priority=1

[wes hayutin (weshayutin)]

issues w/ the repo-promote api in dlrn

 dlrnapi --url https://trunk.rdoproject.org/api-centos8-master-uc --username ciuser repo-promote --commit-hash 51b819b912591b395c0e1c844cbcc5de8bbb75e9 --distro-hash 49ca67e913fc007a64cd7fd09f1083a7a213804b --extended-hash None --promote-name current-tripleo
ERROR: Got error 404, probably endpoint https://trunk.rdoproject.org/api-centos8-master-uc is not available

Even though this is available... https://trunk.rdoproject.org/api-centos8-master-uc/api/repo_status?commit_hash=51b819b912591b395c0e1c844cbcc5de8bbb75e9&distro_hash=49ca67e913fc007a64cd7fd09f1083a7a213804b

Same w/
dlrnapi --url https://trunk.rdoproject.org/api-centos8-master-uc/api/ --username ciuser repo-promote --commit-hash 51b819b912591b395c0e1c844cbcc5de8bbb75e9 --distro-hash 49ca67e913fc007a64cd7fd09f1083a7a213804b --extended-hash None --promote-name current-tripleo
ERROR: Got error 404, probably endpoint https://trunk.rdoproject.org/api-centos8-master-uc/api/ is not available

[Marios Andreou (marios-b)]

Is this bug about tracking a problem with the promoter which caused issues when trying to promote the one we were chasing on friday? i.e. that:

Candidate hash 'aggregate: 882777ccfc662a70d1c3ada62992ffb4, commit: 35f34ffdabe5869a612c2d266bdb01222fba428e, distro: b430c44187b5eee1c712aa5f2076c085a2094c14

OR,

is it about tracking an issue with the dlrn url @ "https://trunk.rdoproject.org/api-centos8-master-uc/api/ is not available"

Incidentally, are you seeing this error in the promoter logs, i.e. is that why the promotion failed? There are no logs at the master promoter log since Friday so I can't check (last is [1])

With respect to the hash you mention in the commit message, agree if we have lost the previous candidate from Friday, then this is indeed a good one to promote today, the buildset looks like:

        * https://review.rdoproject.org/zuul/buildset/4ae70f72549f488297d645dc6865e4ae
         * 2021-06-13 17:28:37.511523 | primary | "container_build_id": "ee4aecfe06de7e8ca63aed041b3e42a8"

Only fs2 missing (but we waived that on friday)

[1] http://10.0.148.74/promoter_logs/centos8_master_2021-06-11T13:30.log-20210612

[Marios Andreou (marios-b)]

Forgot to add - on the issue of dlrn being down, I saw another possibly related issue in today's master periodic integration there

"" * https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing/76e54b3/job-output.txt

        * 2021-06-14 00:12:04.771383 | TASK [get_hash : get md5 repo file]
2021-06-14 00:12:15.501756 | primary | ERROR
2021-06-14 00:12:15.502270 | primary | {
2021-06-14 00:12:15.502384 | primary | "dest": "/home/zuul/workspace/delorean.repo",
2021-06-14 00:12:15.502455 | primary | "elapsed": 10,
2021-06-14 00:12:15.502520 | primary | "msg": "Request failed: <urlopen error _ssl.c:880: The handshake operation timed out>",
2021-06-14 00:12:15.502584 | primary | "url": "https://trunk.rdoproject.org/centos8-master/promoted-components/delorean.repo"
2021-06-14 00:12:15.502648 | primary | }
2021-06-14 00:12:15.529066 |
2021-06-14 00:12:15.529334 | PLAY RECAP
2021-06-14 00:12:15.529457 | primary | ok: 9 changed: 4 unreachable: 0 failed: 1

[Javier Peña (jpena-c)]

Last week the DLRN code was updated in all API endpoints (excluding Wallaby, which was already up-to-date), and it included proper support for the extended_hash parameter in the API calls (commits [1][2]). This was a follow-up to a question by Sandeep regarding a behavior difference between the API endpoints.

By specifying "--extended-hash None" in the dlrnapi-client call, we are passing "None" as a string to the API call, and it is trying to find a commit with "None" as extended_hash (as opposed to None as a null value).

For the cases where extended_hash needs to be None (upstream), we should just not include it in the dlrnapi-client command-line. I can also add a workaround in the client to avoid this confusion in the future.

[1] - https://github.com/softwarefactory-project/DLRN/commit/74dcc54ff591bf473b5c3818823f2b48043eebcc
[2] - https://github.com/softwarefactory-project/DLRN/commit/da5cb584f1081762d64a49a6156f70faecd58594

[Javier Peña (jpena-c)]

So 404 errors must be related to my previous comment. I see there was some network outage during the weekend, and that might have introduced extra errors in some cases (seen as network connection failures). Let's ignore that and focus on the "--extended-hash None" part.

[wes hayutin (weshayutin)]

whayutin> marios|ruck, anbanerj|rover rlandy jpena https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5 :)
[08:13:23] <whayutin> we're now at the right hash
[08:13:30] <whayutin> thanks for all your time folks
[08:13:46] <anbanerj|rover> ++
[08:13:50] <whayutin> https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=ee4aecfe06de7e8ca63aed041b3e42a8
[08:14:14] <whayutin> http://images.rdoproject.org/centos8/master/rdo_trunk/ee4aecfe06de7e8ca63aed041b3e42a8/
[08:14:35] <whayutin> https://hub.docker.com/r/tripleomaster/openstack-base/tags?page=1&ordering=last_updated
[08:14:45] <whayutin> TAG
[08:14:45] <whayutin> ee4aecfe06de7e8ca63aed041b3e42a8_manifest
[08:14:45] <whayutin> docker pull tripleomaster/openstack-base:ee4aecfe06de7e8ca63aed041b3e42a8_manifest
[08:14:45] <whayutin> Last pushed16 hours agobyrdotripleomirror
[08:14:45] <whayutin> DIGEST
[08:14:45] <whayutin> OS/ARCH
[08:14:45] <whayutin> COMPRESSED SIZE
[08:14:45] <whayutin> 4041f5fa79ea
[08:14:45] <whayutin> linux/amd64
[08:14:45] <whayutin> 197.73 MB
[08:14:45] <whayutin> 6a6f59b227f5
[08:14:45] <whayutin> linux/ppc64le

[Marios Andreou (marios-b)]

we can move this back to triaged so we can track the fix on our side ie. removing the extended hash from the promoter server calls to dlrn