Stein periodic featureset020 job is timing out on overcloud deployment - 1.5 hour gap in a single task

Bug #1850934 reported by Ronelle Landy on 2019-11-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Ronelle Landy

Bug Description

Stein periodic featureset020 jobs have been timing out in overcloud deployment. The logs show that there are tasks where 1.5 hours are spent:

See:

2019-11-01 06:37:56 | TASK [create persistent directories] *******************************************
2019-11-01 06:37:56 | changed: [overcloud-novacompute-1] => (item={u'path': u'/var/log/containers/neutron', u'setype': u'svirt_sandbox_file_t'}) => {
2019-11-01 06:37:56 | "ansible_loop_var": "item",
2019-11-01 06:37:56 | "changed": true,
2019-11-01 06:37:56 | "gid": 0,
2019-11-01 06:37:56 | "group": "root",
2019-11-01 06:37:56 | "item": {
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "setype": "svirt_sandbox_file_t"
2019-11-01 06:37:56 | },
2019-11-01 06:37:56 | "mode": "0755",
2019-11-01 06:37:56 | "owner": "root",
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "secontext": "unconfined_u:object_r:container_file_t:s0",
2019-11-01 06:37:56 | "size": 6,
2019-11-01 06:37:56 | "state": "directory",
2019-11-01 06:37:56 | "uid": 0
2019-11-01 06:37:56 | }
2019-11-01 06:37:56 | changed: [overcloud-novacompute-0] => (item={u'path': u'/var/log/containers/neutron', u'setype': u'svirt_sandbox_file_t'}) => {
2019-11-01 06:37:56 | "ansible_loop_var": "item",
2019-11-01 06:37:56 | "changed": true,
2019-11-01 06:37:56 | "gid": 0,
2019-11-01 06:37:56 | "group": "root",
2019-11-01 06:37:56 | "item": {
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "setype": "svirt_sandbox_file_t"
2019-11-01 06:37:56 | },
2019-11-01 06:37:56 | "mode": "0755",
2019-11-01 06:37:56 | "owner": "root",
2019-11-01 08:07:53 | "path": "/var/log/containers/neutron",

Full log:

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/1bfe724/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

And ...

2019-10-30 06:41:40 | RUNNING HANDLER [chrony : Restart chronyd] *************************************
2019-10-30 06:41:40 | changed: [overcloud-novacompute-1] => {
2019-10-30 06:41:40 | "changed": true,
2019-10-30 06:41:40 | "name": "chronyd",
2019-10-30 06:41:40 | "state": "started",
2019-10-30 06:41:40 | "status": {

<snip>
2019-10-30 06:41:40 | }
2019-10-30 06:41:40 | changed: [overcloud-novacompute-0] => {
2019-10-30 06:41:40 | "changed": true,
2019-10-30 06:41:40 | "name": "chronyd",
2019-10-30 06:41:40 | "state": "started",

<snip>

2019-10-30 06:41:40 | "IgnoreOnSnapshot": "no",
2019-10-30 06:41:40 | "IgnoreSIGPIPE": "yes",
2019-10-30 06:41:40 | "InactiveEnterTimestamp": "Wed 2019-10-30 06:41:29 UTC",
2019-10-30 06:41:40 | "InactiveEnterTimestampMonotonic": "460780200",
2019-10-30 06:41:40 | "InactiveExitTimestamp": "Wed 2019-10-30 06:41:29 UTC",
2019-10-30 08:10:34 | "InactiveExitTimestampMonotonic": "460785678",

Full log:

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/9d2b9bb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

The OVB deploy timeout is already set to 2 hours.

Ronelle Landy (rlandy) on 2019-11-01
tags: added: ci promotion-blocker
Changed in tripleo:
milestone: none → ussuri-1
importance: Undecided → Critical
status: New → Triaged
assignee: nobody → Ronelle Landy (rlandy)
Ronelle Landy (rlandy) wrote :

Passed on rerun:

periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein testproject master check 21850,9 16352 2019-11-01T13:45:54 SUCCESS

Reviewed: https://review.opendev.org/692557
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=61c40a7800a4d1b469832b7dfabf00b3072442b6
Submitter: Zuul
Branch: stable/stein

commit 61c40a7800a4d1b469832b7dfabf00b3072442b6
Author: Emilien Macchi <email address hidden>
Date: Mon Sep 23 19:14:41 2019 -0400

    Adapt ContainerImagePrepareDebug to the string pattern

    Like other *Debug parameters, make it so we first look for
    ContainerImagePrepareDebug to be set, otherwise we fallback to Debug;
    like we already do in all other OpenStack services.

    Related-Bug: #1850934
    Change-Id: I0f18b475c69a8ba71b06f517e87caf0d5c209fbb
    (cherry picked from commit 1d11972b10b8154d2e4a34c233a7c7c66048f0a9)

tags: added: in-stable-stein

Reviewed: https://review.opendev.org/692559
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fb494f110d6e02a729bf5b67c363845c0012981d
Submitter: Zuul
Branch: stable/stein

commit fb494f110d6e02a729bf5b67c363845c0012981d
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934
    (cherry picked from commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae)

Reviewed: https://review.opendev.org/692558
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f70ba4bfa467d5622dc4d781a4d3b108064de6ae
Submitter: Zuul
Branch: master

commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934

Reviewed: https://review.opendev.org/692598
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=eb6fd5354916f279ac980514ecedfd3ac9ca5618
Submitter: Zuul
Branch: stable/train

commit eb6fd5354916f279ac980514ecedfd3ac9ca5618
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934
    (cherry picked from commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae)

tags: added: in-stable-train
Ronelle Landy (rlandy) wrote :

Going to close this out as stein is doing much better lately and debug has been added

Changed in tripleo:
status: Triaged → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers