invalid dlrn_hash created for tripleo current-tripleo

Bug #1812261 reported by wes hayutin
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned

Bug Description

master promoted again in http://38.145.34.55/master.log

2019-01-17 12:21:01,543 3938 INFO promoter SUCCESS promoting tripleo-ci-testing as current-tripleo ({'timestamp': 1547700379, 'distro_hash': 'ee467b4016b25b7cb517ebdf0b7227ad15271109', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org', 'repo_url': 'https://trunk.rdoproject.org/centos7/1e/da/1edad15e6d36eec93911f18935e077d6606e091e_ee467b40', 'full_hash': '1edad15e6d36eec93911f18935e077d6606e091e_ee467b40', 'repo_hash': '1edad15e6d36eec93911f18935e077d6606e091e_ee467b40', 'commit_hash': '1edad15e6d36eec93911f18935e077d6606e091e'})

the current-tripleo link was not created...

probably due to
https://trunk.rdoproject.org/centos7/1e/da/1edad15e6d36eec93911f18935e077d6606e091e_ee467b40/commit.yaml

is pointing at stable/rocky

commits:
- commit_branch: stable/rocky
  commit_hash: 1edad15e6d36eec93911f18935e077d6606e091e
  distgit_dir: /home/centos-master-uc/data/openstack-tripleo-heat-templates-compat_distro/
  distro_hash: ee467b4016b25b7cb517ebdf0b7227ad15271109
  dt_build: '1547696807'
  dt_commit: '1547696700.0'
  dt_distro: '1547530369'
  dt_extended: '0'
  extended_hash: None
  flags: '0'
  id: '66009'
  notes: OK
  project_name: openstack-tripleo-heat-templates-compat
  repo_dir: /home/centos-master-uc/data/openstack-tripleo-heat-templates-compat
  rpms: repos/1e/da/1edad15e6d36eec93911f18935e077d6606e091e_ee467b40/openstack-tripleo-heat-templates-compat-9.2.1-0.20190117034756.1edad15.el7.src.rpm,repos/1e/da/1edad15e6d36eec93911f18935e077d6606e091e_ee467b40/openstack-tripleo-heat-templates-compat-9.2.1-0.20190117034756.1edad15.el7.noarch.rpm
  status: SUCCESS

working w/ ops to resolve..
hopefully setting back to previous-current-tripleo

Revision history for this message
wes hayutin (weshayutin) wrote :

Hopefully pointing back to

https://trunk.rdoproject.org/centos7-master/previous-current-tripleo/commit.yaml
commits:
- commit_branch: master
  commit_hash: 296949631440d0f5b6dc0566516f916fe91a58fe
  distgit_dir: /home/centos-master-uc/data/python-networking-ansible_distro/
  distro_hash: b1bca2afd92c3532422ad8dd59a2684f7d7f9b12
  dt_build: '1547571130'
  dt_commit: '1547566558.0'
  dt_distro: '1538150083'
  dt_extended: '0'
  extended_hash: None
  flags: '0'
  id: '65857'
  notes: OK
  project_name: python-networking-ansible
  repo_dir: /home/centos-master-uc/data/python-networking-ansible
  rpms: repos/29/69/296949631440d0f5b6dc0566516f916fe91a58fe_b1bca2af/python-networking-ansible-1.1.0-0.20190115165332.2969496.el7.src.rpm,repos/29/69/296949631440d0f5b6dc0566516f916fe91a58fe_b1bca2af/ansible-role-openstack-ml2-1.1.0-0.20190115165332.2969496.el7.noarch.rpm,repos/29/69/296949631440d0f5b6dc0566516f916fe91a58fe_b1bca2af/python2-networking-ansible-doc-1.1.0-0.20190115165332.2969496.el7.noarch.rpm,repos/29/69/296949631440d0f5b6dc0566516f916fe91a58fe_b1bca2af/python2-networking-ansible-tests-1.1.0-0.20190115165332.2969496.el7.noarch.rpm,repos/29/69/296949631440d0f5b6dc0566516f916fe91a58fe_b1bca2af/python2-networking-ansible-1.1.0-0.20190115165332.2969496.el7.noarch.rpm
  status: SUCCESS

Revision history for this message
wes hayutin (weshayutin) wrote :

https://trunk.rdoproject.org/centos7-master/current-tripleo/delorean.repo is back set to the previous-current-tripleo.

Leaving the bug in alert in case there is fall out.

Revision history for this message
wes hayutin (weshayutin) wrote :

Possible root cause.. stein is marked as stable/rocky

https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L3021-L3030

- project: tripleo-heat-templates-compatMike Burns, 2 years ago: • add tripleo-heat-templates-compat
  conf: rpmfactory-core
  upstream: git://git.openstack.org/openstack/tripleo-heat-templatesJavier Peña, 2 years ago: • Revert "TEMP: replace git.openstack.org usage…
  tags:Mike Burns, 2 years ago: • add tripleo-heat-templates-compat
    stein-py3-uc:yatin, 29 days ago: • Add some more python3 enabled packages to ste…
      source-branch: stable/rocky
    stein-uc:Alfredo Moralejo, 5 months ago: • Switch to stein tags
      source-branch: stable/rockyyatin, 29 days ago: • Add some more python3 enabled packages to ste…
    stein:Alfredo Moralejo, 5 months ago: • Switch to stein tags
      source-branch: stable/rockyyatin, 29 days ago: • Add some more python3 enabled packages to ste…

Revision history for this message
Alan Pevec (apevec) wrote :

@Wes this is tripleo-heat-templates-compat which has N-1 templates, so stable/rocky for stein is correct

Revision history for this message
wes hayutin (weshayutin) wrote :

@Alan, ah makes sense. Thanks for pointing that out. Perhaps the link getting created was just a fluke. I'll turn the promoter back on and we'll see how it goes.

I was unable to resolve the problem myself.. possibly doing something incorrectly as seen in https://bugs.launchpad.net/tripleo/+bug/1812264

It's critical as w/o a current-tripleo link all the upstream jobs go red and fail.

wes hayutin (weshayutin)
tags: removed: alert
Revision history for this message
Javier Peña (jpena-c) wrote :
Download full text (3.3 KiB)

I have checked the logs. I see the following promotion attempts on Jan 18:

38.145.34.55 - - [17/Jan/2019:12:20:48 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 404 "-" "dlrnapi-client/1.0.0/python"
38.145.34.55 - - [17/Jan/2019:12:20:55 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 395 "-" "dlrnapi-client/1.0.0/python"
38.145.33.147 - - [17/Jan/2019:14:46:13 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 413 "-" "dlrnapi-client/1.0.0/python"
88.97.26.43 - - [17/Jan/2019:20:21:40 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 395 "-" "dlrnapi-client/1.0.0/python"
88.97.26.43 - - [17/Jan/2019:20:22:03 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 395 "-" "dlrnapi-client/1.0.0/python"
38.145.34.55 - - [17/Jan/2019:21:32:33 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 401 19 "-" "dlrnapi-client/1.0.0/python"
38.145.34.55 - - [17/Jan/2019:21:33:20 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 401 19 "-" "dlrnapi-client/1.0.0/python"
38.145.34.55 - - [17/Jan/2019:21:33:46 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 401 19 "-" "dlrnapi-client/1.0.0/python"
38.145.34.55 - - [17/Jan/2019:21:36:26 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 401 19 "-" "dlrnapi-client/1.0.0/python"
38.145.33.201 - - [17/Jan/2019:22:46:42 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 413 "-" "dlrnapi-client/1.0.0/python"

The one at 12:20 succeeded (HTTP code 201). Looking at the DLRN DB, we find the following entry

+------+-----------+-----------------+------------+
| id | commit_id | promotion_name | timestamp |
+------+-----------+-----------------+------------+
| 1874 | 66009 | current-tripleo | 1547727660 |

That timestamp is GMT: Thursday, January 17, 2019 12:21:00 PM

MariaDB [dlrn_centos_master_uc]> select project_name, commit_hash, distro_hash from commits where id=66009;
+-----------------------------------------+------------------------------------------+------------------------------------------+
| project_name | commit_hash | distro_hash |
+-----------------------------------------+------------------------------------------+------------------------------------------+
| openstack-tripleo-heat-templates-compat | 1edad15e6d36eec93911f18935e077d6606e091e | ee467b4016b25b7cb517ebdf0b7227ad15271109 |
+-----------------------------------------+------------------------------------------+------------------------------------------+

This is the same hash that apparently failed, so the entry in the DB was created.

According to the DLRN API promotion code at https://github.com/softwarefactory-project/DLRN/blob/master/dlrn/api/dlrn_api.py#L524-L551, if the symlink creation fails, the entry will not be created in the DB.

What I find interesting is that we have two promotion calls at almost the same time:

38.145.34.55 - - [17/Jan/2019:12:20:48 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 404 "-" "dlrnapi-client/1.0.0/python"
38.145.34.55 - - [17/Jan/2019:12:20:55 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 395...

Read more...

Revision history for this message
Javier Peña (jpena-c) wrote :

This is the first time the current-tripleo symlink access failed, according to the httpd logs:

[Thu Jan 17 20:21:51.856975 2019] [core:error] [pid 5454] [client 23.253.166.120:37732] AH00037: Symbolic link not allowed or link target not accessible: /var/www/html/centos7-master/current-tripleo

Revision history for this message
Javier Peña (jpena-c) wrote :

Looking at the DB, the timestamp for that error matches the following promotion:

88.97.26.43 - - [17/Jan/2019:20:21:40 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 395 "-" "dlrnapi-client/1.0.0/python"
88.97.26.43 - - [17/Jan/2019:20:22:03 +0000] "POST /api-centos-master-uc/api/promote HTTP/1.1" 201 395 "-" "dlrnapi-client/1.0.0/python"

That promoted a commit for tripleo-heat-templates with commit_hash a09f481909fd75d8be4d42ed9f678bf6d1387792 and distro_hash 99947e05167cef613af384dd667869352d490145. That is a very old commit, that had already been purged from the DB.

The IP running the promotion does not look familiar, it does not come from RDO Cloud. Who tried to promote that commit, and why?

Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
wes hayutin (weshayutin) wrote :

the was caused by user error.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.