tripleo

Error: Could not run: No facts available for target node: agent.local

Bug #1980198 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned
tripleo zed-1

Bug Description

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ansible-centos-stream-molecule-backup_and_restore&project=openstack/tripleo-ansible is failing on the cs8 to cs9 migration patch.

Below is the failure:

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a67/839688/91/check/tripleo-ansible-centos-stream-molecule-backup_and_restore/a67b12d/job-output.txt

```
TASK [backup_and_restore : Getting enabled_services] ***************************
2022-06-29 02:14:54.482869 | centos-9-stream | fatal: [instance]: FAILED! => changed=true
2022-06-29 02:14:54.482881 | centos-9-stream | cmd:
2022-06-29 02:14:54.482892 | centos-9-stream | - puppet
2022-06-29 02:14:54.482903 | centos-9-stream | - lookup
2022-06-29 02:14:54.482914 | centos-9-stream | - --node
2022-06-29 02:14:54.482938 | centos-9-stream | - agent.local
2022-06-29 02:14:54.482949 | centos-9-stream | - --explain
2022-06-29 02:14:54.482960 | centos-9-stream | - enabled_services
2022-06-29 02:14:54.482971 | centos-9-stream | delta: '0:00:02.085366'
2022-06-29 02:14:54.482989 | centos-9-stream | end: '2022-06-29 02:14:54.146486'
2022-06-29 02:14:54.483015 | centos-9-stream | msg: non-zero return code
2022-06-29 02:14:54.483027 | centos-9-stream | rc: 1
2022-06-29 02:14:54.483040 | centos-9-stream | start: '2022-06-29 02:14:52.061120'
2022-06-29 02:14:54.483051 | centos-9-stream | stderr: "\e[1;31mError: Could not run: No facts available for target node: agent.local\e[0m"
2022-06-29 02:14:54.483063 | centos-9-stream | stderr_lines: <omitted>
2022-06-29 02:14:54.483075 | centos-9-stream | stdout: ''
2022-06-29 02:14:54.483087 | centos-9-stream | stdout_lines: <omitted>
2022-06-29 02:14:54.483098 | centos-9-stream |
```

It is failing at this step:
https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/backup_and_restore/tasks/main.yml#L35

I am logging this bug to track the issue and mark this job as a non-voting till we get the patch in.

Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

I suspect an issue due to puppet or facter version.

In order to check this out, I've just pushed a patch that should help gathering environmental information:
https://review.opendev.org/c/openstack/tripleo-ansible/+/848088

Since it's stacked onto Chandan's patch, we should be able to get the data for this specific failure right away.

Revision history for this message
Takashi Kajinami (kajinamit) wrote :

I need to check the actual fact cache generated in the node and maybe debug output of that puppet command to find out the cause but I suspect the cache had expired before lookup was executed.

In general I do not recommend using the fact cache in tripleo-ansible especially if the role is supposed to be executed independently from deployment. Fact cache is not kept forever and expires. If you rely on facts then you might face a problem if the cache has expired.

Can't we use ansible group var generated as part of config-download process ?

Revision history for this message
Takashi Kajinami (kajinamit) wrote :

https://github.com/puppetlabs/puppet/commit/6b1cc3f6ec2d75508395a0803f39fc89c7db980d

It seems the above change made in puppet 7.13.0 is causing the difference.
We recently bumped puppet from 7.10.0 to 7.16.0 to bump facter to 4, for FIPS compliance,
and that change was pulled as part of the version bump.

Revision history for this message
Cédric Jeanneret (cjeanner) wrote :

Note, this is coming from molecule tests, not an actual deploy. So I'm wondering if we can't just fake some facter content for this specific usage?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/848150

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/839688
Committed: https://opendev.org/openstack/tripleo-ansible/commit/4c9233d07c0886871719b46b2801c7b0565f26cc
Submitter: "Zuul (22348)"
Branch: master

commit 4c9233d07c0886871719b46b2801c7b0565f26cc
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Mon May 30 09:45:42 2022 +0530

    Use ubi9-init image as base

    CentOS-8 no longer exists and existing jobs are running
    on centos-stream-8. There will be no openstack master
    rpm packages for centos-stream-8.

    In order to use ubi9-init image in molecule jobs,
    we need to include following workarounds:
    - redhat-release package(RH specific package) is installed in ubi-init
      which does not work with centos-stream package.
      On removing redhat-release, dnf ansible breaks as they did not find
      --releasever during package installation.
      - By adding releasever in the package module fixes the issue.
    - In Molecule job, we can use any hostname either of centos or ubi both
      will work.
    - It changes the nodeset and image to ubi9.
    - Install pip < 20.3.0 to avoid https://pip.pypa.io/en/stable/topics/dependency-resolution/#backtracking
      and upstream requirements file to avoid cyclic dependency
      issue.
    - Set enable_fips: true in collectd jobs as in molecule it is not
      needed otherwise job will enter into retry_limit.

    We cannot use centos:stream9 image as, init support is removed
    from there.

    Below is the list of following jobs marked non-voting
    due to following bugs.
    - tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo-modules
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo_container_manage
      - https://bugs.launchpad.net/tripleo/+bug/1979810
    - tripleo-ansible-centos-stream-molecule-tripleo_redhat_enforce
      - https://bugs.launchpad.net/tripleo/+bug/1979971
    - tripleo-ansible-centos-stream-molecule-tripleo_collectd
      - https://bugs.launchpad.net/tripleo/+bug/1979972
    - tripleo-ansible-centos-stream-molecule-backup_and_restore
      - https://bugs.launchpad.net/tripleo/+bug/1980198

    Related-Bug: #1979807
    Related-Bug: #1979810
    Related-Bug: #1979971
    Related-Bug: #1979972
    Related-Bug: #1980198

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I75a27ca496fb7670ba22a6a22f62f001c8afe6a9

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/848150
Committed: https://opendev.org/openstack/tripleo-ansible/commit/f54b54839abe6e9229aff77cc959a4cd63043ad6
Submitter: "Zuul (22348)"
Branch: master

commit f54b54839abe6e9229aff77cc959a4cd63043ad6
Author: Takashi Kajinami <email address hidden>
Date: Thu Jun 30 00:48:49 2022 +0900

    Remove wrong usage of --node in puppet lookup command

    Puppet does not translate agent.local and regards it as an actual node
    name. Usage of an invalid command results in a failure since puppet
    7.13.0[1] (now we have puppet 7.16.0) because hieradata of that node is
    not properly looked up. If we want to look up data for the local node
    then --node is not needed.

    [1] https://github.com/puppetlabs/puppet/commit/6b1cc3f6ec2d75508395a0803f39fc89c7db980d

    Closes-Bug: #1980198
    Change-Id: I9fcca0551ff673bbfabddfe9daefb6e96699f93d

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-ansible (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/848810

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-ansible/+/848810
Committed: https://opendev.org/openstack/tripleo-ansible/commit/b666c78954fbbe7edfd50b6b949c4921fc6c5518
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b666c78954fbbe7edfd50b6b949c4921fc6c5518
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Mon May 30 09:45:42 2022 +0530

    Use ubi9-init image as base

    CentOS-8 no longer exists and existing jobs are running
    on centos-stream-8. There will be no openstack master
    rpm packages for centos-stream-8.

    In order to use ubi9-init image in molecule jobs,
    we need to include following workarounds:
    - redhat-release package(RH specific package) is installed in ubi-init
      which does not work with centos-stream package.
      On removing redhat-release, dnf ansible breaks as they did not find
      --releasever during package installation.
      - By adding releasever in the package module fixes the issue.
    - In Molecule job, we can use any hostname either of centos or ubi both
      will work.
    - It changes the nodeset and image to ubi9.
    - Install pip < 20.3.0 to avoid https://pip.pypa.io/en/stable/topics/dependency-resolution/#backtracking
      and upstream requirements file to avoid cyclic dependency
      issue.
    - Set enable_fips: true in collectd jobs as in molecule it is not
      needed otherwise job will enter into retry_limit.

    We cannot use centos:stream9 image as, init support is removed
    from there.

    Below is the list of following jobs marked non-voting
    due to following bugs.
    - tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo-modules
      - https://bugs.launchpad.net/tripleo/+bug/1979807
    - tripleo-ansible-centos-stream-molecule-tripleo_container_manage
      - https://bugs.launchpad.net/tripleo/+bug/1979810
    - tripleo-ansible-centos-stream-molecule-tripleo_redhat_enforce
      - https://bugs.launchpad.net/tripleo/+bug/1979971
    - tripleo-ansible-centos-stream-molecule-tripleo_collectd
      - https://bugs.launchpad.net/tripleo/+bug/1979972
    - tripleo-ansible-centos-stream-molecule-backup_and_restore
      - https://bugs.launchpad.net/tripleo/+bug/1980198

    Note: It is not a clean cherry pick.
    - Remove cap from molecule dependency
      to fix No module named 'ansiblelint.prerun'
    - Populate wallaby repos using tripleo-repos tool
    - This includes the commit 0bc2fa808d34235ffe7cf8d36920f6d96f0685b5
      which fixed the issue with the original change in master.
    - Marking tripleo-ansible-centos-stream-molecule-tripleo_packages
      to non-voting due to
      https://bugs.launchpad.net/tripleo/+bug/1983347

    Related-Bug: #1979807
    Related-Bug: #1979810
    Related-Bug: #1979971
    Related-Bug: #1979972
    Related-Bug: #1980198
    Related-Bug: #1983347

    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>
    Change-Id: I75a27ca496fb7670ba22a6a22f62f001c8afe6a9
    (cherry picked from commit 4c9233d07c0886871719b46b2801c7b0565f26cc)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-ansible 5.0.0

This issue was fixed in the openstack/tripleo-ansible 5.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.