Stein periodic featureset020 job is timing out on overcloud deployment - 1.5 hour gap in a single task

[Ronelle Landy]

Stein periodic featureset020 jobs have been timing out in overcloud deployment. The logs show that there are tasks where 1.5 hours are spent:

See:

2019-11-01 06:37:56 | TASK [create persistent directories] *******************************************
2019-11-01 06:37:56 | changed: [overcloud-novacompute-1] => (item={u'path': u'/var/log/containers/neutron', u'setype': u'svirt_sandbox_file_t'}) => {
2019-11-01 06:37:56 | "ansible_loop_var": "item",
2019-11-01 06:37:56 | "changed": true,
2019-11-01 06:37:56 | "gid": 0,
2019-11-01 06:37:56 | "group": "root",
2019-11-01 06:37:56 | "item": {
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "setype": "svirt_sandbox_file_t"
2019-11-01 06:37:56 | },
2019-11-01 06:37:56 | "mode": "0755",
2019-11-01 06:37:56 | "owner": "root",
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "secontext": "unconfined_u:object_r:container_file_t:s0",
2019-11-01 06:37:56 | "size": 6,
2019-11-01 06:37:56 | "state": "directory",
2019-11-01 06:37:56 | "uid": 0
2019-11-01 06:37:56 | }
2019-11-01 06:37:56 | changed: [overcloud-novacompute-0] => (item={u'path': u'/var/log/containers/neutron', u'setype': u'svirt_sandbox_file_t'}) => {
2019-11-01 06:37:56 | "ansible_loop_var": "item",
2019-11-01 06:37:56 | "changed": true,
2019-11-01 06:37:56 | "gid": 0,
2019-11-01 06:37:56 | "group": "root",
2019-11-01 06:37:56 | "item": {
2019-11-01 06:37:56 | "path": "/var/log/containers/neutron",
2019-11-01 06:37:56 | "setype": "svirt_sandbox_file_t"
2019-11-01 06:37:56 | },
2019-11-01 06:37:56 | "mode": "0755",
2019-11-01 06:37:56 | "owner": "root",
2019-11-01 08:07:53 | "path": "/var/log/containers/neutron",

Full log:

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/1bfe724/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

And ...

2019-10-30 06:41:40 | RUNNING HANDLER [chrony : Restart chronyd] *************************************
2019-10-30 06:41:40 | changed: [overcloud-novacompute-1] => {
2019-10-30 06:41:40 | "changed": true,
2019-10-30 06:41:40 | "name": "chronyd",
2019-10-30 06:41:40 | "state": "started",
2019-10-30 06:41:40 | "status": {

<snip>
2019-10-30 06:41:40 | }
2019-10-30 06:41:40 | changed: [overcloud-novacompute-0] => {
2019-10-30 06:41:40 | "changed": true,
2019-10-30 06:41:40 | "name": "chronyd",
2019-10-30 06:41:40 | "state": "started",

<snip>

2019-10-30 06:41:40 | "IgnoreOnSnapshot": "no",
2019-10-30 06:41:40 | "IgnoreSIGPIPE": "yes",
2019-10-30 06:41:40 | "InactiveEnterTimestamp": "Wed 2019-10-30 06:41:29 UTC",
2019-10-30 06:41:40 | "InactiveEnterTimestampMonotonic": "460780200",
2019-10-30 06:41:40 | "InactiveExitTimestamp": "Wed 2019-10-30 06:41:29 UTC",
2019-10-30 08:10:34 | "InactiveExitTimestampMonotonic": "460785678",

Full log:

http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/9d2b9bb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

The OVB deploy timeout is already set to 2 hours.

[Ronelle Landy (rlandy)]

Could possibly be fixed by https://review.opendev.org/#/c/691919/3/roles/undercloud-deploy/templates/containers-prepare-parameter.yaml.j2

Waiting on testproject results

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/692557

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/692558

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/692559

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/692598

[Ronelle Landy (rlandy)]

Passed on rerun:

periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein testproject master check 21850,9 16352 2019-11-01T13:45:54 SUCCESS

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692557
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=61c40a7800a4d1b469832b7dfabf00b3072442b6
Submitter: Zuul
Branch: stable/stein

commit 61c40a7800a4d1b469832b7dfabf00b3072442b6
Author: Emilien Macchi <email address hidden>
Date: Mon Sep 23 19:14:41 2019 -0400

    Adapt ContainerImagePrepareDebug to the string pattern

    Like other *Debug parameters, make it so we first look for
    ContainerImagePrepareDebug to be set, otherwise we fallback to Debug;
    like we already do in all other OpenStack services.

    Related-Bug: #1850934
    Change-Id: I0f18b475c69a8ba71b06f517e87caf0d5c209fbb
    (cherry picked from commit 1d11972b10b8154d2e4a34c233a7c7c66048f0a9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692559
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=fb494f110d6e02a729bf5b67c363845c0012981d
Submitter: Zuul
Branch: stable/stein

commit fb494f110d6e02a729bf5b67c363845c0012981d
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934
    (cherry picked from commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692558
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f70ba4bfa467d5622dc4d781a4d3b108064de6ae
Submitter: Zuul
Branch: master

commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934

[Ronelle Landy (rlandy)]

Related bug: https://bugs.launchpad.net/tripleo/+bug/1849101

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692598
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=eb6fd5354916f279ac980514ecedfd3ac9ca5618
Submitter: Zuul
Branch: stable/train

commit eb6fd5354916f279ac980514ecedfd3ac9ca5618
Author: Alex Schultz <email address hidden>
Date: Fri Nov 1 08:51:41 2019 -0600

    Honor Debug for container image prepare

    We were never using Debug if ContainerImagePrepareDebug was not set.
    This change adds the Debug param and updates ContainerImagePrepareDebug
    to be '' by default. This makes it honor the Debug setting if
    ContainerImagePrepareDebug is not configured.

    Change-Id: I09b3b112d7654fed5270c0b1148b57b91b4a3215
    Related-Bug: #1850934
    (cherry picked from commit f70ba4bfa467d5622dc4d781a4d3b108064de6ae)

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein is doing better the last two days

[Ronelle Landy (rlandy)]

Going to close this out as stein is doing much better lately and debug has been added