octavia certs are autogenerated on every update

[Brent Eagles]

The octavia ansible code for autogenerating certificates is running whenever stack update breaking secure comms between the octavia agents and load balancers. After the initial deployment cert management should be handled by something outside of stack update operations or in conjunction with some as-yet-unwritten management code that can be integrated with update/upgrade tasks.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/672995

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672529
Committed: https://git.openstack.org/cgit/openstack/tripleo-ansible/commit/?id=cb692576145b97beadb0116ea50a102a50eb3ef5
Submitter: Zuul
Branch: master

commit cb692576145b97beadb0116ea50a102a50eb3ef5
Author: Brent Eagles <email address hidden>
Date: Wed Jul 24 11:28:10 2019 -0230

    Allow distribution of non-autogenerated certs

    This patch changes the conditional run of the certs generation from
    being dependent on the value of generate_certs to whether there are
    actual certs present.

    Related-Bug: #1838039

    Change-Id: I90bb377c76f51db906de64c134271ec866d11bb5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672995
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b61156785517f767a9ad0ee1613588f6b049fc8c
Submitter: Zuul
Branch: master

commit b61156785517f767a9ad0ee1613588f6b049fc8c
Author: Brent Eagles <email address hidden>
Date: Fri Jul 26 11:50:19 2019 -0230

    Only generate Octavia certs on stack create

    We are regenerating octavia certs whenever an overcloud is updated,
    breaking any deployments using the auto-generated certs. Certificate
    updates after the initial deployment require special handling and
    shouldn't be performed by stack updates/upgrades at this time.

    Depends-On: I90bb377c76f51db906de64c134271ec866d11bb5
    Closes-Bug: #1838039
    Change-Id: I05f69df627e5637fdb254285cb3ad6d3d8328f90

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/678918

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/678919

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/678918
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=16fb6c247bddcb123bc3b1f768bdb38223e0f20b
Submitter: Zuul
Branch: stable/stein

commit 16fb6c247bddcb123bc3b1f768bdb38223e0f20b
Author: Brent Eagles <email address hidden>
Date: Tue Aug 27 15:02:09 2019 -0230

    Allow distribution of non-autogenerated certs

    This patch changes the conditional run of the certs generation from
    being dependent on the value of generate_certs to whether there are
    actual certs present.

    Note: this is a semantic backport of https://review.opendev.org/#/c/672529/

    Change-Id: I8088a0a42094b2d038ba29779535a05195138747
    Related-Bug: #1838039

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/678919
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2f4dd2c927660e1d950e9d6ef49e4cdc628c94df
Submitter: Zuul
Branch: stable/stein

commit 2f4dd2c927660e1d950e9d6ef49e4cdc628c94df
Author: Brent Eagles <email address hidden>
Date: Fri Jul 26 11:50:19 2019 -0230

    Only generate Octavia certs on stack create

    We are regenerating octavia certs whenever an overcloud is updated,
    breaking any deployments using the auto-generated certs. Certificate
    updates after the initial deployment require special handling and
    shouldn't be performed by stack updates/upgrades at this time.

    Note: depends on changed because the dependent patch was a semantic
    backport.

    Depends-On: I8088a0a42094b2d038ba29779535a05195138747
    Closes-Bug: #1838039
    Change-Id: I05f69df627e5637fdb254285cb3ad6d3d8328f90
    (cherry picked from commit b61156785517f767a9ad0ee1613588f6b049fc8c)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/679772

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/679773

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679772
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=0755e93d5baae6caeb11b7c024bdc9da8429ac7f
Submitter: Zuul
Branch: stable/rocky

commit 0755e93d5baae6caeb11b7c024bdc9da8429ac7f
Author: Brent Eagles <email address hidden>
Date: Tue Aug 27 15:02:09 2019 -0230

    Allow distribution of non-autogenerated certs

    This patch changes the conditional run of the certs generation from
    being dependent on the value of generate_certs to whether there are
    actual certs present.

    Conflicts:
     playbooks/roles/octavia-controller-config/tasks/certificate.yml

    Note: this is a semantic backport of https://review.opendev.org/#/c/672529/

    Change-Id: I8088a0a42094b2d038ba29779535a05195138747
    Related-Bug: #1838039

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679773
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=82bfea421e7f8195b7bd6cd3bbb93e605f82163e
Submitter: Zuul
Branch: stable/rocky

commit 82bfea421e7f8195b7bd6cd3bbb93e605f82163e
Author: Brent Eagles <email address hidden>
Date: Fri Jul 26 11:50:19 2019 -0230

    Only generate Octavia certs on stack create

    We are regenerating octavia certs whenever an overcloud is updated,
    breaking any deployments using the auto-generated certs. Certificate
    updates after the initial deployment require special handling and
    shouldn't be performed by stack updates/upgrades at this time.

    Note: depends on changed because the dependent patch was a semantic
    backport.

    Depends-On: I8088a0a42094b2d038ba29779535a05195138747
    Closes-Bug: #1838039
    Change-Id: I05f69df627e5637fdb254285cb3ad6d3d8328f90
    (cherry picked from commit b61156785517f767a9ad0ee1613588f6b049fc8c)
    (cherry picked from commit 2f4dd2c927660e1d950e9d6ef49e4cdc628c94df)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.1 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/682659

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/682660

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/682660
Reason: We are facing gate issue: https://bugs.launchpad.net/tripleo/+bug/1844446

To clear the gate we need to abandon this patch and I will restore once the gate is ready again to land patches in TripleO. Please don't touch this patch, and ask on #tripleo Wes or Emilien for any question. Thanks for your patience.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.2.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682659
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=16232417e1a1719eab5a013f0abe90e327e10387
Submitter: Zuul
Branch: stable/queens

commit 16232417e1a1719eab5a013f0abe90e327e10387
Author: Brent Eagles <email address hidden>
Date: Tue Aug 27 15:02:09 2019 -0230

    Allow distribution of non-autogenerated certs

    This patch changes the conditional run of the certs generation from
    being dependent on the value of generate_certs to whether there are
    actual certs present.

    Conflicts:
     playbooks/roles/octavia-controller-config/tasks/certificate.yml

    Note: this is a semantic backport of https://review.opendev.org/#/c/672529/

    Change-Id: I8088a0a42094b2d038ba29779535a05195138747
    Related-Bug: #1838039
    (cherry picked from commit 0755e93d5baae6caeb11b7c024bdc9da8429ac7f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682660
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=73fa7e2c32b99a6dd819ca31915d3e7e3a1de824
Submitter: Zuul
Branch: stable/queens

commit 73fa7e2c32b99a6dd819ca31915d3e7e3a1de824
Author: Brent Eagles <email address hidden>
Date: Fri Jul 26 11:50:19 2019 -0230

    Only generate Octavia certs on stack create

    We are regenerating octavia certs whenever an overcloud is updated,
    breaking any deployments using the auto-generated certs. Certificate
    updates after the initial deployment require special handling and
    shouldn't be performed by stack updates/upgrades at this time.

    Note: depends on changed because the dependent patch was a semantic
    backport.

    Depends-On: I8088a0a42094b2d038ba29779535a05195138747
    Closes-Bug: #1838039
    Change-Id: I05f69df627e5637fdb254285cb3ad6d3d8328f90
    (cherry picked from commit b61156785517f767a9ad0ee1613588f6b049fc8c)
    (cherry picked from commit 2f4dd2c927660e1d950e9d6ef49e4cdc628c94df)
    (cherry picked from commit 82bfea421e7f8195b7bd6cd3bbb93e605f82163e)

[Tal Liron (emblem-parade)]

Unfortunately, this patch causes a breaking regression for a common use case: adding Octavia to an existing deployment.

Even more unfortunately, there's no other choice but doing a stack update when using RDO. The issue in RDO is that TripleO will not upload the Amphora image to Glance during overcloud deployment, and Glance won't be running until that first deployment succeeds.

The issue is with this addition to octavia-deployment-config.j2.yaml:

      - equals:
        - get_param: StackAction
        - CREATE

I understand the problem you were trying to solve here, but I think we need a different solution. Perhaps there's a way to check on disk that the cert files don't already exist before generated them? And possibly add a way to force (re-)generation in case they do exist?

[Brent Eagles (beagles)]

@Tal, yes we realized that shortly after the patches propogated. We are in the process of implementing a solution that scans all of the nodes for existing CAs and private keys and using those if they do otherwise generating new ones. I realized that there is also a hidden bug with scaleout of controllers and controller replacement along the same lines.

See https://bugs.launchpad.net/tripleo/+bug/1849548 and https://bugs.launchpad.net/tripleo/+bug/1849550

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.