tripleo

urllib3.exceptions.LocationParseError: Failed to parse: https://trunk.rdoproject.org/api-centos8-master-uc/api/report_result

Bug #1882534 reported by Pooja Jadhav on 2020-06-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Critical	Unassigned	tripleo wallaby-3

Bug Description

Below CentOS 8 master jobs failing with error logs and skipping others:

2020-06-08 08:32:38.312645 | TASK [Report to DLRN]
2020-06-08 08:32:39.254164 | primary | + source /home/zuul/workspace/hash_info.sh
2020-06-08 08:32:39.254241 | primary | ++ export DLRNAPI_URL=https://trunk.rdoproject.org/api-centos8-master-uc
2020-06-08 08:32:39.254258 | primary | ++ DLRNAPI_URL=https://trunk.rdoproject.org/api-centos8-master-uc
2020-06-08 08:32:39.254266 | primary | ++ export RELEASE=master
2020-06-08 08:32:39.254272 | primary | ++ RELEASE=master
2020-06-08 08:32:39.254278 | primary | ++ export FULL_HASH=4971182127e82705f0ec113c2e2493ab
2020-06-08 08:32:39.254286 | primary | ++ FULL_HASH=4971182127e82705f0ec113c2e2493ab
2020-06-08 08:32:39.254293 | primary | ++ export COMMIT_HASH=
2020-06-08 08:32:39.254301 | primary | ++ COMMIT_HASH=
2020-06-08 08:32:39.254307 | primary | ++ export DISTRO_HASH=
2020-06-08 08:32:39.254313 | primary | ++ DISTRO_HASH=
2020-06-08 08:32:39.254318 | primary | ++ export COMPONENT_NAME=
2020-06-08 08:32:39.254324 | primary | ++ COMPONENT_NAME=
2020-06-08 08:32:39.254330 | primary | ++ export UPLOAD_EXIT_VALUE=0
2020-06-08 08:32:39.254335 | primary | ++ UPLOAD_EXIT_VALUE=0
2020-06-08 08:32:39.254343 | primary | + '[' True = true ']'
2020-06-08 08:32:39.254352 | primary | + '[' True = True ']'
2020-06-08 08:32:39.254360 | primary | + echo 'REPORTING SUCCESS TO DLRN API'
2020-06-08 08:32:39.254369 | primary | REPORTING SUCCESS TO DLRN API
2020-06-08 08:32:39.254435 | primary | + [[ 4971182127e82705f0ec113c2e2493ab == *\_* ]]
2020-06-08 08:32:39.254453 | primary | + HASH_ARGS='--agg-hash 4971182127e82705f0ec113c2e2493ab'
2020-06-08 08:32:39.254947 | primary | ++ command -v pip
2020-06-08 08:32:39.255047 | primary | ++ command -v pip3
2020-06-08 08:32:39.255553 | primary | + pip_cmd=/usr/bin/pip3
2020-06-08 08:32:39.255583 | primary | + /usr/bin/pip3 install --user dlrnapi-client shyaml
2020-06-08 08:32:40.000656 | primary | Collecting dlrnapi-client
2020-06-08 08:32:40.136256 | primary | Downloading http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/pypifiles/packages/c8/32/4250993e2b897728eae74b391e8137169b0a6a34b9c96b24c45e68e034c4/dlrnapi_client-0.9.1-py2.py3-none-any.whl (49kB)
2020-06-08 08:32:40.304016 | primary | Collecting shyaml
2020-06-08 08:32:40.317657 | primary | Downloading http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/pypifiles/packages/29/84/30051cc954696388d63bac118278e5c5a02d1765eed257b41b63a8b9c333/shyaml-0.6.1-py2.py3-none-any.whl
2020-06-08 08:32:40.400287 | primary | Requirement already satisfied: setuptools>=21.0.0 in /usr/lib/python3.6/site-packages (from dlrnapi-client)
2020-06-08 08:32:40.480154 | primary | Collecting certifi>=14.05.14 (from dlrnapi-client)
2020-06-08 08:32:40.507979 | primary | Downloading http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org/pypifiles/packages/98/99/def511020aa8f663d4a2cfaa38467539e864799289ff354569e339e375b1/certifi-2020.4.5.2-py2.py3-none-any.whl (157kB)
2020-06-08 08:32:40.596350 | primary | Requirement already satisfied: python-dateutil>=2.5.3 in /usr/lib/python3.6/site-packages (from dlrnapi-client)
2020-06-08 08:32:40.599231 | primary | Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.6/site-packages (from dlrnapi-client)
2020-06-08 08:32:40.601308 | primary | Requirement already satisfied: urllib3>=1.15.1 in /usr/lib/python3.6/site-packages (from dlrnapi-client)
2020-06-08 08:32:40.608860 | primary | Requirement already satisfied: pyyaml in /usr/lib64/python3.6/site-packages (from shyaml)
2020-06-08 08:32:40.609304 | primary | Installing collected packages: certifi, dlrnapi-client, shyaml
2020-06-08 08:32:40.674653 | primary | Successfully installed certifi-2020.4.5.2 dlrnapi-client-0.9.1 shyaml-0.6.1
2020-06-08 08:32:40.890681 | primary | + PATH=/home/zuul/.local/bin:/home/zuul/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/zuul/.local/bin
2020-06-08 08:32:40.891067 | primary | ++ date +%s
2020-06-08 08:32:40.892220 | primary | + dlrnapi --url https://trunk.rdoproject.org/api-centos8-master-uc report-result --agg-hash 4971182127e82705f0ec113c2e2493ab --job-id periodic-tripleo-centos-8-buildimage-ironic-python-agent-master --info-url https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-master/f0cf193 --timestamp 1591605160 --success True
2020-06-08 08:32:41.005566 | primary | Traceback (most recent call last):
2020-06-08 08:32:41.005633 | primary | File "/home/zuul/.local/bin/dlrnapi", line 11, in <module>
2020-06-08 08:32:41.005649 | primary | sys.exit(main())
2020-06-08 08:32:41.005671 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/shell.py", line 448, in main
2020-06-08 08:32:41.005679 | primary | raise e
2020-06-08 08:32:41.005686 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/shell.py", line 436, in main
2020-06-08 08:32:41.005692 | primary | api_response = command_funcs[options.command](api_instance, options)
2020-06-08 08:32:41.005700 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/shell.py", line 177, in report_result
2020-06-08 08:32:41.005707 | primary | api_response = api_instance.api_report_result_post(params)
2020-06-08 08:32:41.005712 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/apis/default_api.py", line 916, in api_report_result_post
2020-06-08 08:32:41.005718 | primary | **kwargs)
2020-06-08 08:32:41.005724 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/apis/default_api.py", line 1001, in api_report_result_post_with_http_info
2020-06-08 08:32:41.005732 | primary | collection_formats=collection_formats)
2020-06-08 08:32:41.005739 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/api_client.py", line 320, in call_api
2020-06-08 08:32:41.005747 | primary | _preload_content, _request_timeout)
2020-06-08 08:32:41.005753 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/api_client.py", line 142, in __call_api
2020-06-08 08:32:41.005761 | primary | _request_timeout=_request_timeout)
2020-06-08 08:32:41.005769 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/api_client.py", line 368, in request
2020-06-08 08:32:41.005775 | primary | body=body)
2020-06-08 08:32:41.005781 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/rest.py", line 276, in POST
2020-06-08 08:32:41.005788 | primary | body=body)
2020-06-08 08:32:41.005794 | primary | File "/home/zuul/.local/lib/python3.6/site-packages/dlrnapi_client/rest.py", line 155, in request
2020-06-08 08:32:41.005801 | primary | headers=headers)
2020-06-08 08:32:41.005807 | primary | File "/usr/lib/python3.6/site-packages/urllib3/request.py", line 80, in request
2020-06-08 08:32:41.005815 | primary | method, url, fields=fields, headers=headers, **urlopen_kw
2020-06-08 08:32:41.005821 | primary | File "/usr/lib/python3.6/site-packages/urllib3/request.py", line 171, in request_encode_body
2020-06-08 08:32:41.005826 | primary | return self.urlopen(method, url, **extra_kw)
2020-06-08 08:32:41.005832 | primary | File "/usr/lib/python3.6/site-packages/urllib3/poolmanager.py", line 318, in urlopen
2020-06-08 08:32:41.005838 | primary | u = parse_url(url)
2020-06-08 08:32:41.005844 | primary | File "/usr/lib/python3.6/site-packages/urllib3/util/url.py", line 398, in parse_url
2020-06-08 08:32:41.005850 | primary | return six.raise_from(LocationParseError(source_url), None)
2020-06-08 08:32:41.005857 | primary | File "<string>", line 3, in raise_from
2020-06-08 08:32:41.005863 | primary | urllib3.exceptions.LocationParseError: Failed to parse: https://trunk.rdoproject.org/api-centos8-master-uc/api/report_result
2020-06-08 08:32:41.502510 | primary | ERROR
2020-06-08 08:32:41.502846 | primary | {
2020-06-08 08:32:41.502933 | primary | "delta": "0:00:01.766545",
2020-06-08 08:32:41.502976 | primary | "end": "2020-06-08 08:32:41.015096",
2020-06-08 08:32:41.503044 | primary | "msg": "non-zero return code",
2020-06-08 08:32:41.503084 | primary | "rc": 1,
2020-06-08 08:32:41.503120 | primary | "start": "2020-06-08 08:32:39.248551"
2020-06-08 08:32:41.503156 | primary | }
2020-06-08 08:32:41.530243 |
2020-06-08 08:32:41.530393 | PLAY RECAP
2020-06-08 08:32:41.530472 | primary | ok: 1 changed: 0 unreachable: 0 failed: 1 skipped: 5 rescued: 0 ignored: 0

Affacted Jobs :

1. periodic-tripleo-centos-8-buildimage-ironic-python-agent-master
2. periodic-tripleo-centos-8-buildimage-overcloud-full-master
3. periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-master
4. periodic-tripleo-centos-8-master-containers-build-push

Reference Links :

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-master/f0cf193/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/ea5c167/job-output.txt

Tags:

Revision history for this message

wes hayutin (weshayutin) wrote on 2020-06-08:

fixed in https://review.rdoproject.org/r/#/c/27977/

Changed in tripleo:
status:	In Progress → Fix Released

Pooja Jadhav (poojajadhav22) on 2020-06-09

Changed in tripleo:
status:	Fix Released → In Progress

Revision history for this message

yatin (yatinkarel) wrote on 2020-06-09:

Still some issue, jobs still failing after the fix https://review.rdoproject.org/r/#/c/27977/:- https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&result=POST_FAILURE

For original issue:-
The issue all started after tripleo-ansible was added to bindep.txt in https://review.opendev.org/#/c/724147/. Before this patch urllib3 was not installed from rpm and installed via pip as a dep for dlrnapi-client, it came as indirect dependency (tripleo-ansibe <- python3-tripleo-common <- python3-keystoneaut1 <- python3-urllib3).

For why https://review.opendev.org/#/c/724147/ not working in affected jobs:-
Since now both python3-urllib3-1.25.7-2.el8(Needs python3-six) and python3-urllib3-1.25.7-2.el8(Needs python3-six >= 1.12.0) available in repo. When tripleo-ansible is being installed it avoids upgrade of already installed python3-six and chooses python3-urllib3-1.25.7-2.el8. Don't know it's a bug in dnf or it's a feature.

We should update already installed packages after setting up repos like same way it's done in tripleo deployment jobs via quickstart else we can hit issues like this.

Currently the issue is worked around by updating python3-urllib3 explicitly in post playbook via https://review.rdoproject.org/r/#/c/27997/.

Revision history for this message

yatin (yatinkarel) wrote on 2020-06-09:

<<< We should update already installed packages after setting up repos like same way it's done in tripleo deployment jobs via quickstart else we can hit issues like this.

Proposed https://review.opendev.org/#/c/734605/

Revision history for this message

Rafael Folco (rafaelfolco) wrote on 2020-06-10:

still seeing this after the fix merged
https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-master/24ebc9a/job-output.txt

Revision history for this message

Marios Andreou (marios-b) wrote on 2020-06-10:

fyi I was seeing the same issue in d/stream rhel jobs - tried the workaround today of installing a newer urllib3 and it worked for me

e.g. error there:
https://sf.hosted.upshift.rdu2.redhat.com/logs/43/201543/11/check/periodic-tripleo-ci-rhel-8-standalone-upgrade-rhos-17/bedb8f2/logs/emit_releases_file.log
(for those that can't access the logs my apologies, copy/pasting the trace below [1] )

and then today I added an explicit pip install (cos the current d/stream repos only gave me not new enough python3-urllib3-1.25.7-2.el8ost.noarch.rpm ) there https://review.opendev.org/#/c/732170/12/roles/run-test/templates/toci_gate_test.sh.j2

and no longer have the LocationParseError.

[1] Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 379, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "/usr/lib/python3.6/site-packages/urllib3/util/url.py", line 398, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: http://[redacted]/rdo/rhel8-osp17/current/delorean.repo.md5

Revision history for this message

yatin (yatinkarel) wrote on 2020-06-10:

<< still seeing this after the fix merged
https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-full-master/24ebc9a/job-output.txt

These logs are older, before workaround https://review.rdoproject.org/r/#/c/27997/ or the fix https://review.opendev.org/#/c/734605/ merged.

The fix https://review.opendev.org/#/c/734605/ is merged now and we also reverted the workaround with https://review.rdoproject.org/r/#/c/28037/. If u still see issue please let me know.

<<< and then today I added an explicit pip install (cos the current d/stream repos only gave me not new enough python3-urllib3-1.25.7-2.el8ost.noarch.rpm )
Hmm in downstream urllib3 needs to be updated seperately.

Emilien Macchi (emilienm) on 2020-07-28

Changed in tripleo:
milestone:	victoria-1 → victoria-3

Marios Andreou (marios-b) on 2020-11-03

Changed in tripleo:
milestone:	victoria-3 → wallaby-1

Marios Andreou (marios-b) on 2020-12-08

Changed in tripleo:
milestone:	wallaby-1 → wallaby-2

Marios Andreou (marios-b) on 2021-01-29

Changed in tripleo:
milestone:	wallaby-2 → wallaby-3

Revision history for this message

yatin (yatinkarel) wrote on 2021-01-29:

Fixed with https://review.opendev.org/#/c/734605/

Changed in tripleo:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.