deployment failures with latest puppet-ssh (from current)

[Luca Miccini]

Since Mar 23rd we've been seeing this in our centos9/master (bgp and non-bgp deployments):

Mar 24 23:36:13 puppet-user: Warning: Unknown variable: 'ssh::sshd_config'. (file: /etc/puppet/modules/ssh/manifests/server/match_block.pp, line: 10, column: 34)
Mar 24 23:36:13 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Ssh::Server::Match_block[nova_migration deny]: parameter 'target' expects a Stdlib::Absolutepath = Variant[Stdlib::Windowspath = Pattern[/\A(([a-zA-Z]:[\\\/])|([\\\/][\\\/][^\\\/]+[\\\/][^\\\/]+)|([\\\/][\\\/]\?[\\\/][^\\\/]+)).*\z/], Stdlib::Unixpath = Pattern[/\A\/([^\n\/\0]+\/*)*\z/]] value, got Undef (file: /etc/puppet/modules/tripleo/manifests/profile/base/nova/migration/target.pp, line: 73) on node compute-0.home.arpa
+ rc=1

this is extracted from container-puppet-nova_libvirt logs.

We think the issue has been introduced via https://github.com/saz/puppet-ssh/commit/0513f08eda27aeb3878185313cfd5212878a83db#diff-4f37388630d12973a8f0b41a16d2715e5e1495913b78e6a1e15fb2e968694f27

https://github.com/saz/puppet-ssh/blob/45b81357a036c087fbafdbe92a2ee8516c0aec26/manifests/server/match_block.pp#L10

this should probably be:

Stdlib::Absolutepath $target = $ssh::params::sshd_config

After patching this line we hit another issue with a duplicate declaration of "Ssh::Server" because it is included in:

https://github.com/saz/puppet-ssh/blob/45b81357a036c087fbafdbe92a2ee8516c0aec26/manifests/server/match_block.pp#L12

and:

 https://github.com/openstack/puppet-tripleo/blob/8b0cbfad8483fa2c89044e94870a7e06587682ae/manifests/profile/base/sshd.pp#L111-L112

  if hiera('ssh:server::options', undef) {
    err('ssh:server::options must not be set, use tripleo::profile::base::sshd::options')

here we think we have another issue because it should be "ssh::server::options" (double ":") instead.

[Luca Miccini (lmiccini2)]

[root@compute-1 ~]# rpm -qa |grep puppet-tripleo
puppet-tripleo-16.0.1-0.20220317223409.8b0cbfa.el9.noarch
[root@compute-1 ~]# rpm -qa |grep puppet-ssh
puppet-ssh-8.0.1-0.20220322155843.dc877e2.el9.noarch

[Luca Miccini (lmiccini2)]

first issue has been fixed via https://github.com/saz/puppet-ssh/commit/013ea6528b29f28c3ef22afbbc8aaeaed68e2eab

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/835446

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/835448

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/835454

[Takashi Kajinami (kajinamit)]

Can I have a look about the actual error caused by the second issue (duplicate declaration)?

The line you pointed doesn't include a class.
The line just tries to look up a variable from the ssh::server class, and ends up with undef value and should not affect class inclusion at my first glance.

[Takashi Kajinami (kajinamit)]

ignore it. I understood the problem. will fix it in puppet-sshd

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Bogdan Dobrelya <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/835454

[Bogdan Dobrelya (bogdando)]

TripleO doesn't look affected [0] but only RDO (since it builds puppet-ssh from master)?

[0] https://893b7893e38745f91f12-6eabc6840d85d43aa006c18e755cf4b4.ssl.cf1.rackcdn.com/835448/3/check/tripleo-ci-centos-9-standalone/1a8e06b/logs/undercloud/var/log/extra/rpm-list.txt

(see puppet-ssh-8.0.0-0.20210921175728.b84d4dc.el9.noarch, which is older version that the reported commit causing the subject issue is)

[Bogdan Dobrelya (bogdando)]

According to reporting the affected puppet build comes with current-tripleo tag. Apparently in CI is used a less recent tagging (is it tripleo-ci-testing?)

[Bogdan Dobrelya (bogdando)]

OK, thank you for discovering this promo blocker, @Luca. While it is being fixed, please double-check your deployment method does not fetch non-promoted components (current-tripleo has no 8.0.0 version of puppet-ssh [0])

[0] https://trunk.rdoproject.org/centos9-master/component/tripleo/current-tripleo/

[Bogdan Dobrelya (bogdando)]

correction: current-tripleo has no the reported 8.0.1 version of puppet-ssh and brings no that regression for current-tripleo DLRN tag users

[Sandeep Yadav (sandeepyadav93)]

We are also hitting the same issue in tripleo component pipeline.

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master/639f586/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
<13>Mar 28 21:39:33 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Ssh::Server::Match_block[nova_migration deny]: parameter 'target' expects a Stdlib::Absolutepath = Variant[Stdlib::Windowspath = Pattern[/\A(([a-zA-Z]:[\\\/])|([\\\/][\\\/][^\\\/]+[\\\/][^\\\/]+)|([\\\/][\\\/]\?[\\\/][^\\\/]+)).*\z/], Stdlib::Unixpath = Pattern[/\A\/([^\n\/\0]+\/*)*\z/]] value, got Undef (file: /etc/puppet/modules/tripleo/manifests/profile/base/nova/migration/target.pp, line: 73) on node standalone.localdomain
+ rc=1
~~~

The triple component pipeline is doing its job, it block the issue from reaching the integration line and check.

We don't have 8.0.1 in current tripleo: https://trunk.rdoproject.org/centos9-master/component/tripleo/current-tripleo/

While the issue is correct, I do not expect the current-tripleo tag users to see it.

Luca could you please confirm if you use tripleo-repos to layout repos?

[Luca Miccini (lmiccini2)]

I just double-checked and we use https://trunk.rdoproject.org/centos9-master/current/delorean.repo (we need the latest rpms for bgp)

[Ronelle Landy (rlandy)]

Any update here? The master tripleo line is still blocked on this.:

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master/54c37cb/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[Alan Pevec (apevec)]

> will fix it in puppet-sshd

Is this is the upstream fix and tracking issue?

https://github.com/saz/puppet-ssh/pull/326

https://github.com/saz/puppet-ssh/issues/324

[Marios Andreou (marios-b)]

@Alan:

There is also that one https://review.opendev.org/c/openstack/puppet-tripleo/+/835448 (addressing the https://github.com/saz/puppet-ssh/issues/324) ... not clear if we still need that as well as the upstream fix from https://github.com/saz/puppet-ssh/pull/326 ?

Also that has related bug https://review.opendev.org/c/openstack/puppet-tripleo/+/835446

[Bogdan Dobrelya (bogdando)]

https://review.opendev.org/c/openstack/puppet-tripleo/+/835448 is needed after https://github.com/saz/puppet-ssh/pull/326 to fix a double-declaration issue that comes by it

[Bogdan Dobrelya (bogdando)]

In case if https://github.com/saz/puppet-ssh/pull/326 resolves all issues, then https://review.opendev.org/c/openstack/puppet-tripleo/+/835448 would not be needed though.

[Takashi Kajinami (kajinamit)]

Although my PR in puppet-ssh repo would work as a short term fix, there is an open PR in puppet-ssh repo which would prevent usage of ssh::server[1] .

[1] https://github.com/saz/puppet-ssh/pull/325/

Considering that the proposed change to puppet-tripleo[2] looks more appropriate fix atm.

[2] https://review.opendev.org/c/openstack/puppet-tripleo/+/835448

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/835448
Committed: https://opendev.org/openstack/puppet-tripleo/commit/ba1e593200a9ce3c636dca9d3ed120ab10ccd7af
Submitter: "Zuul (22348)"
Branch: master

commit ba1e593200a9ce3c636dca9d3ed120ab10ccd7af
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Mar 28 14:57:26 2022 +0200

    Do not include the ssh::server class directly

    ... to avoid the duplicate declaration of the ssh::server class we've
    seen since https://github.com/saz/puppet-ssh/pull/318 was merged.

    There is an open PR[1] in puppet-ssh repo, which makes the server
    class private. This change also works as pre-emptive fix for that
    breaking change.

    [1] https://github.com/saz/puppet-ssh/pull/325

    Closes-bug: #1966625
    Change-Id: I9c5b174e0c8377d50b9b306456d2b2147c5f38b8
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/puppet-tripleo/+/835446
Committed: https://opendev.org/openstack/puppet-tripleo/commit/1d7c92df16ad9ac91fefaddabc28448d2f786c89
Submitter: "Zuul (22348)"
Branch: master

commit 1d7c92df16ad9ac91fefaddabc28448d2f786c89
Author: Cédric Jeanneret <email address hidden>
Date: Mon Mar 28 14:54:23 2022 +0200

    Correct typo on hieradata key

    Path separator is "::", not just ":"

    Change-Id: Ifaf09792fd57ab17a12a1bca1bd3bb0a072e91be
    Related-Bug: #1966625

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 17.0.0 release.