Filtering by VIP in /extraconfig/nova_metadata/krb-service-principals/role.role.j2.yaml breaks deployment

Bug #1854846 reported by Harald Jensås
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Harald Jensås

Bug Description

The following patch:

~~~~
Per-Role krb-service-principal for CompactServices

Filter krb-service-principals for the CompactServices
based on the networks associated with the role.

Filtering for the IndividualServices was added in previous
fix https://review.openstack.org/646005, which did'nt
fully fix the bug.

Closes-Bug: #1821377
Change-Id: Id54477ca5581e1f5fe8a09c3bc60a238d114dbb2
(cherry picked from commit 578bcb2ffa)

tags/10.6.1
~~~~

LINK: https://opendev.org/openstack/tripleo-heat-templates/commit/223ddba9137a3c9129fc33593db086518bf75a78?lang=en-US

Contains additional filtering (as means to workaround nova metadata fields size limit: each field cannot exceed 256 bytes).

However, the filtering might be a little too aggressive:

l63: {%- for network in networks if network.vip|default(false) and network.name in role.networks %}
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^

$ cat network_data.yaml

- name: SomeNetwork
  name_lower: somenet
  vip: false <<<<<<<<<<<<<<<<<
  ip_subnet: xxx.xxx.xxx.0/xx
  allocation_pools: [{'start': 'xxx.xxx.xxx.xxx', 'end': 'xxx.xxx.xxx.xxx'}]
  vlan: xxxxxxxx

< snip >

os-collect-config logs:

Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: "Warning: Could not get certificate: Execution of '/usr/bin/getcert request -I contrail -f /etc/contrail/ssl/certs/server.pem -c IPA -N CN=ctrl1.example.com -K contrail/ctrl1.example.com -D ctl1.ctlplane.example.com -D ctl1.somenet.example.com-D < snip > -D -D -D -D -D -C sudo docker ps -q --filter=name=\"contrail*\" | xargs -i sudo docker restart {} -w -k /etc/contrail/ssl/private/server-privkey.pem' returned 3: New signing request \"contrail\" added.",
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: "Error: /Stage[main]/Tripleo::Certmonger::Contrail/Certmonger_certificate[contrail]: Could not evaluate: Could not get certificate: Server at https://idm.example.com/ipa/xml failed request, will retry: 4001 (RPC failed at server. The service principal for subject alt name ctl1.somenet.example.com in certificate request does not exist).",
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: "Warning: Could not get certificate: Execution of '/usr/bin/getcert request -I httpd-somenet -f /etc/pki/tls/certs/httpd/httpd-somenet.crt -c IPA -N CN=ctl1.somenet.example.com -K HTTP/ctl1.somenet.xxxxxxxxxxxxxxxx -D ctl1.somenetxxxxxxxxxxxxxxxxx -C pkill -USR1 httpd -w -k /etc/pki/tls/private/httpd/httpd-somenet.key' returned 3: New signing request \"httpd-somenet\" added.",
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-somenet]/Certmonger_certificate[httpd-somenet]: Could not evaluate: Could not get certificate: Server at https://idm.example.com/ipa/xml failed request, will retry: 4001 (RPC failed at server. The host 'ctl1.somenet.example.com' does not exist to add a service to.).",
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: "Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Exec[tripleo-ca-crl]: Skipping because of failed dependencies",
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: "Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/File[tripleo-ca-crl-file]: Skipping because of failed dependencies",
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: "Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Exec[tripleo-ca-crl-process-command]: Skipping because of failed dependencies",
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: "Warning: /Stage[main]/Tripleo::Certmonger::Ca::Crl/Cron[tripleo-refresh-crl-file]: Skipping because of failed dependencies"
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: ]
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: }
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx_playbook.retry
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: PLAY RECAP *********************************************************************
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: localhost : ok=26 changed=13 unreachable=0 failed=1
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: [2019-11-22 15:00:22,217] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-ansible/_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxplaybook.yaml. [2]
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: [2019-11-22 15:00:22,223] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/ansible
Nov 22 15:00:22 ctrl1.example.com os-collect-config[12345]: [2019-11-22 15:00:22,224] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.json < /var/lib/heat-config/deployed/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.notify.json

Revision history for this message
Harald Jensås (harald-jensas) wrote :

I have reproduced this issue with just the default managment network enabled:

Dec 05 13:04:57 overcloud-controller-0.lab.example.com os-collect-config[9657]: "Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Httpd[httpd-management]/Certmonger_certificate[httpd-management]: Could not evaluate: Could not get certificat
e: Server at https://idm.lab.example.com/ipa/xml failed request, will retry: 4001 (RPC failed at server. The host 'overcloud-controller-0.management.lab.example.com' does not exist to add a service to.).",

tags: added: rocky-backport-potential stein-backport-potential train-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/697498

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Harald Jensås (harald-jensas) wrote :

without fix:
------------
[root@idm ~]# ipa host-find | grep overcloud | grep "Principal name"
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>

with fix:
---------
[root@idm ~]# ipa host-find | grep overcloud | grep "Principal name"
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>
  Principal name: <email address hidden>

/** The entry for management network is now included with the fix */
  Principal name: <email address hidden>

Deployment completes successfully.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/697498
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=af79ae34ad9a3ed1581c0f6df91fa1e3ef0f8433
Submitter: Zuul
Branch: master

commit af79ae34ad9a3ed1581c0f6df91fa1e3ef0f8433
Author: Harald Jensås <email address hidden>
Date: Mon Dec 2 18:49:45 2019 +0100

    Relax filtering in krb-service-principals jinja

    The filtering added to fix Bug: #1821377 filters any
    network without a VIP address. This filtering is to
    agressive and cause deployment failure when a management
    network without a VIP is used.

    Change-Id: If189eb6fc0b2dc2c78323a7c08f7e303be2b6124
    Resolves: rhbz#1778719
    Closes-Bug: #1854846

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/697597

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/697598

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/697599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/697600

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/697600
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=bc0bd3f1512883affbdacb1285e7f7bfc17c6d0c
Submitter: Zuul
Branch: stable/queens

commit bc0bd3f1512883affbdacb1285e7f7bfc17c6d0c
Author: Harald Jensås <email address hidden>
Date: Mon Dec 2 18:49:45 2019 +0100

    Relax filtering in krb-service-principals jinja

    The filtering added to fix Bug: #1821377 filters any
    network without a VIP address. This filtering is to
    agressive and cause deployment failure when a management
    network without a VIP is used.

    Change-Id: If189eb6fc0b2dc2c78323a7c08f7e303be2b6124
    Resolves: rhbz#1778719
    Closes-Bug: #1854846
    (cherry picked from commit af79ae34ad9a3ed1581c0f6df91fa1e3ef0f8433)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/697598
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=632d80441e5b072cd429a1049f00ce67926ceb73
Submitter: Zuul
Branch: stable/stein

commit 632d80441e5b072cd429a1049f00ce67926ceb73
Author: Harald Jensås <email address hidden>
Date: Mon Dec 2 18:49:45 2019 +0100

    Relax filtering in krb-service-principals jinja

    The filtering added to fix Bug: #1821377 filters any
    network without a VIP address. This filtering is to
    agressive and cause deployment failure when a management
    network without a VIP is used.

    Change-Id: If189eb6fc0b2dc2c78323a7c08f7e303be2b6124
    Resolves: rhbz#1778719
    Closes-Bug: #1854846
    (cherry picked from commit af79ae34ad9a3ed1581c0f6df91fa1e3ef0f8433)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/697597
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cfe728a51bdcb534a99bb3d4ad6759b9c51d247b
Submitter: Zuul
Branch: stable/train

commit cfe728a51bdcb534a99bb3d4ad6759b9c51d247b
Author: Harald Jensås <email address hidden>
Date: Mon Dec 2 18:49:45 2019 +0100

    Relax filtering in krb-service-principals jinja

    The filtering added to fix Bug: #1821377 filters any
    network without a VIP address. This filtering is to
    agressive and cause deployment failure when a management
    network without a VIP is used.

    Change-Id: If189eb6fc0b2dc2c78323a7c08f7e303be2b6124
    Resolves: rhbz#1778719
    Closes-Bug: #1854846
    (cherry picked from commit af79ae34ad9a3ed1581c0f6df91fa1e3ef0f8433)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.3.1

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/697599
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=80c0d697b5be96ec4179560c85c171bfadeff198
Submitter: Zuul
Branch: stable/rocky

commit 80c0d697b5be96ec4179560c85c171bfadeff198
Author: Harald Jensås <email address hidden>
Date: Mon Dec 2 18:49:45 2019 +0100

    Relax filtering in krb-service-principals jinja

    The filtering added to fix Bug: #1821377 filters any
    network without a VIP address. This filtering is to
    agressive and cause deployment failure when a management
    network without a VIP is used.

    Change-Id: If189eb6fc0b2dc2c78323a7c08f7e303be2b6124
    Resolves: rhbz#1778719
    Closes-Bug: #1854846
    (cherry picked from commit af79ae34ad9a3ed1581c0f6df91fa1e3ef0f8433)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 12.1.0

This issue was fixed in the openstack/tripleo-heat-templates 12.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates rocky-eol

This issue was fixed in the openstack/tripleo-heat-templates rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates queens-eol

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates stein-eol

This issue was fixed in the openstack/tripleo-heat-templates stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.