tripleo

The error was: error while evaluating conditional ((wait_for_all_osds_up.stdout | from_json)[\"osdmap\"][\"num_osds\"] | int > 0): ''dict object'' has no attribute ''num_osds''"}' failing cpeh integration jobs while standalone deployment sc001 and 004 on ussuri

Bug #1883689 reported by Bhagyashri Shewale on 2020-06-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Fix Released	Medium	Bhagyashri Shewale	tripleo victoria-1 "tripleo victoria"

Bug Description

I am getting below issue while adding ceph integration jobs in third party rdo jobs [1]

Error log:

  - 'TASK [ceph-osd : include_tasks systemd.yml] ************************************'
  - 'Monday 15 June 2020 20:18:05 +0000 (0:00:03.867) 0:03:19.379 *********** '
  - 'included: /usr/share/ceph-ansible/roles/ceph-osd/tasks/systemd.yml for standalone'
  - 'TASK [ceph-osd : generate ceph osd docker run script] **************************'
  - 'Monday 15 June 2020 20:18:05 +0000 (0:00:00.127) 0:03:19.506 *********** '
  - 'changed: [standalone] => {"changed": true, "checksum": "1ab0e66503e12ca6976c479f93d33d2d36d0cb1c", "dest": "/usr/share/ceph-osd-run.sh", "gid": 0, "group": "root", "md5sum": "a01d6630f6013667a99bea7503ef8f8e", "mode": "0744", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 856, "src": "/tmp/ceph_ansible_tmp/ansible-tmp-1592252285.8198211-114812-233138311475739/source", "state": "file", "uid": 0}'
  - 'TASK [ceph-osd : generate systemd unit file] ***********************************'
  - 'Monday 15 June 2020 20:18:06 +0000 (0:00:00.488) 0:03:19.995 *********** '
  - 'changed: [standalone] => {"changed": true, "checksum": "c2d830365e413a62482958c4bf736300bf9e0e41", "dest": "/etc/systemd/system/ceph-osd@.service", "gid": 0, "group": "root", "md5sum": "6b63e28becb4d8b6a4b6dc05cc205675", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:systemd_unit_file_t:s0", "size": 489, "src": "/tmp/ceph_ansible_tmp/ansible-tmp-1592252286.3112404-114830-223539576437186/source", "state": "file", "uid": 0}'
  - 'TASK [ceph-osd : systemd start osd] ********************************************'
  - 'Monday 15 June 2020 20:18:06 +0000 (0:00:00.487) 0:03:20.482 *********** '
  - 'changed: [standalone] => (item=0) => {"ansible_loop_var": "item", "changed": true, "enabled": true, "item": "0", "name": "ceph-osd@0", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "network.target systemd-journald.socket basic.target system-ceph\\x2dosd.slice sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill
    cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Ceph OSD", "DevicePolicy": "auto", "DynamicUser": "no", "EnvironmentFiles": "/etc/environment (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/share/ceph-osd-run.sh ;
    argv[]=/usr/share/ceph-osd-run.sh 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/bin/podman ; argv[]=/usr/bin/podman rm -f ceph-osd-0 ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/bin/podman ; argv[]=/usr/bin/podman stop ceph-osd-0 ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/etc/systemd/system/ceph-osd@.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ceph-osd@0.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE":
    "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "none", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "31734", "LimitNPROCSoft": "31734", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING":
    "31734", "LimitSIGPENDINGSoft": "31734", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ceph-osd@0.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no",
    "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system-ceph\\x2dosd.slice sysinit.target", "Restart": "always", "RestartUSec": "10s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system-ceph\\x2dosd.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction":
    "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "2min", "TimeoutStopUSec": "15s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "disabled", "UtmpMode": "init", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}}'
  - 'ok: [standalone] => (item=0) => {"ansible_loop_var": "item", "changed": false, "enabled": true, "item": "0", "name": "ceph-osd@0", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-06-15 20:18:07 UTC", "ActiveEnterTimestampMonotonic": "2680899818", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target systemd-journald.socket sysinit.target system-ceph\\x2dosd.slice network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-06-15 20:18:07 UTC", "AssertTimestampMonotonic": "2680693531", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart":
    "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-06-15 20:18:07 UTC", "ConditionTimestampMonotonic": "2680693531", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/system-ceph\\x2dosd.slice/ceph-osd@0.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Ceph OSD", "DevicePolicy": "auto", "DynamicUser":
    "no", "EnvironmentFiles": "/etc/environment (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "114923", "ExecMainStartTimestamp": "Mon 2020-06-15 20:18:07 UTC", "ExecMainStartTimestampMonotonic": "2680899754", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/share/ceph-osd-run.sh ; argv[]=/usr/share/ceph-osd-run.sh 0 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPre": "{ path=/usr/bin/podman ; argv[]=/usr/bin/podman rm -f ceph-osd-0 ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/bin/podman ; argv[]=/usr/bin/podman stop ceph-osd-0 ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/etc/systemd/system/ceph-osd@.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass":
    "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ceph-osd@0.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-06-15 20:18:07 UTC", "InactiveExitTimestampMonotonic": "2680694594", "InvocationID": "1888aa5be8cf4d1a827d98495e46df10", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "none", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity",
    "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "31734", "LimitNPROCSoft": "31734", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "31734", "LimitSIGPENDINGSoft": "31734", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "114923", "MemoryAccounting": "yes", "MemoryCurrent": "34041856", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ceph-osd@0.service", "NeedDaemonReload": "no",
    "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system-ceph\\x2dosd.slice sysinit.target", "Restart": "always", "RestartUSec": "10s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system-ceph\\x2dosd.slice", "StandardError": "inherit", "StandardInput":
    "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-06-15 20:18:07 UTC", "StateChangeTimestampMonotonic": "2680899818", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "16", "TasksMax": "26213", "TimeoutStartUSec": "2min", "TimeoutStopUSec": "15s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy":
    "multi-user.target", "WatchdogTimestamp": "Mon 2020-06-15 20:18:07 UTC", "WatchdogTimestampMonotonic": "2680899816", "WatchdogUSec": "0"}}'
  - 'TASK [ceph-osd : ensure systemd service override directory exists] *************'
  - 'Monday 15 June 2020 20:18:08 +0000 (0:00:01.448) 0:03:21.930 *********** '
  - 'TASK [ceph-osd : add ceph-osd systemd service overrides] ***********************'
  - 'Monday 15 June 2020 20:18:08 +0000 (0:00:00.078) 0:03:22.009 *********** '
  - 'TASK [ceph-osd : unset noup flag] **********************************************'
  - 'Monday 15 June 2020 20:18:08 +0000 (0:00:00.131) 0:03:22.140 *********** '
  - 'ok: [standalone -> 192.168.24.1] => {"changed": false, "cmd": ["podman", "exec", "ceph-mon-standalone", "ceph", "--cluster", "ceph", "osd", "unset", "noup"], "delta": "0:00:01.331173", "end": "2020-06-15 20:18:10.010029", "rc": 0, "start": "2020-06-15 20:18:08.678856", "stderr": "noup is unset", "stderr_lines": ["noup is unset"], "stdout": "", "stdout_lines": []}'
  - 'TASK [ceph-osd : wait for all osd to be up] ************************************'
  - 'Monday 15 June 2020 20:18:10 +0000 (0:00:01.699) 0:03:23.839 *********** '
  - 'fatal: [standalone]: FAILED! => {"msg": "The conditional check ''(wait_for_all_osds_up.stdout | from_json)[\"osdmap\"][\"num_osds\"] | int > 0'' failed. The error was: error while evaluating conditional ((wait_for_all_osds_up.stdout | from_json)[\"osdmap\"][\"num_osds\"] | int > 0): ''dict object'' has no attribute ''num_osds''"}'
  - NO MORE HOSTS LEFT *************************************************************
  - PLAY RECAP *********************************************************************
  - 'standalone : ok=251 changed=37 unreachable=0 failed=1 skipped=341 rescued=0 ignored=0 '
  - INSTALLER STATUS ***************************************************************
  - 'Install Ceph Monitor : Complete (0:00:54)'
  - 'Install Ceph Manager : Complete (0:00:36)'
  - 'Install Ceph OSD : In Progress (0:01:01)'
  - "\tThis phase can be restarted by running: roles/ceph-osd/tasks/main.yml"
  - 'Monday 15 June 2020 20:18:11 +0000 (0:00:01.142) 0:03:24.982 *********** '
  - '=============================================================================== '

Note: Here i am using ceph-ansible stable-2.4 from @ceph-ansible-override and ansible.noarch 2.9.9-2.el8 from @delorean-ussuri-testing

Refence link:

[1]: https://review.rdoproject.org/r/#/c/27926/

[2]: https://logserver.rdoproject.org/26/27926/7/check/tripleo-ceph-integration-centos-8-scenario001-standalone-ussuri/e98051b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[3]: https://logserver.rdoproject.org/26/27926/7/check/tripleo-ceph-integration-centos-8-scenario001-standalone-ussuri/e98051b/logs/undercloud/var/log/extra/package-list-installed.txt.gz

See original description

Tags:

chandan kumar (chkumar246) on 2020-06-16

tags:

added: alert ci depcheck

Bhagyashri Shewale (bhagyashri-shewale) on 2020-06-16

tags:	removed: alert ci
description:	updated

Revision history for this message

Giulio Fidente (gfidente) wrote on 2020-06-16:

for ussuri, like for train, we want ceph-ansible from stable-4.0 branch and I think that is what is happening in [1] but, g0175c205 which is the version of ceph-ansible installed is an hash coming from "git describe --tags" in master not in stable-4.0; git describe --tags against stable-4.0 should return "v4.0.23-17-gb30a9439"

1. https://review.rdoproject.org/r/#/c/27926/