Storage host unlock fails - readded journal, update journal location and size

[Wendy Mitchell]

Brief Description
-----------------
Traceback in puppet in attempt to unlock storage host that had journal readded (with location and size update)
(TypeError: cannot concatenate 'str' and 'NoneType' objects)

Severity
--------
standard

Steps to Reproduce
------------------
1. system which has storage hosts configured
storage hosts are both unlocked, enabled, available prior to the test
A journal is configured prior to the test and 4 nvme devices are returned with the host-disk-list (system host-disk-list storage-1)

$ system host-stor-list storage-1
+--------------------------------------+----------+-------+------------+--------------------------------------+-------------------------------------------------+-------------- --+------------------+-----------+
| uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
+--------------------------------------+----------+-------+------------+--------------------------------------+-------------------------------------------------+-------------- --+------------------+-----------+
| 3f9a6686-2b76-4278-b997-b029211d7632 | osd | 1 | configured | eff63bd4-aaa2-486e-bcfc-cffedf07ee3a | /dev/disk/by-path/pci-0000:84:00.0-nvme-1-part2 | /dev/nvme1n1p 2 | 1 | storage |
| d3bdd1ab-30ed-46b6-afd0-ed18a84b2253 | osd | 0 | configured | c5705e5b-fdc5-4dd6-8a74-7878ddee0521 | /dev/disk/by-path/pci-0000:84:00.0-nvme-1-part1 | /dev/nvme1n1p 1 | 1 | storage |
| fded4a8b-4c03-4092-9034-cc8c70bb69bb | journal | None | configured | 97535924-c9df-4409-ac02-439a638f052f | None | None | None | None

2. Update the journal size and location
]$ system host-stor-update --journal-location fded4a8b-4c03-4092-9034-cc8c70bb69bb --journal-size 2 3f9a6686-2b76-4278-b997-b029211d7632
+------------------+-------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------+
| osdid | 1 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | fded4a8b-4c03-4092-9034-cc8c70bb69bb |
| journal_size_gib | 2048 |
| journal_path | /dev/disk/by-path/pci-0000:84:00.0-nvme-1-part2 |
| journal_node | /dev/nvme1n1p2 |
| uuid | 3f9a6686-2b76-4278-b997-b029211d7632 |
| ihost_uuid | b3f5dd04-f8e8-4f3e-9011-791d16d7abe8 |
| idisk_uuid | eff63bd4-aaa2-486e-bcfc-cffedf07ee3a |
| tier_uuid | f4626531-86fd-45fb-af9f-8417af1a46ff |
| tier_name | storage |
| created_at | 2019-11-13T15:49:59.427588+00:00 |
| updated_at | 2019-11-13T15:55:18.884702+00:00 |

Note: Unlock storage-1 --- succeeds.

3. Lock storage-1, delete the journal, readd the journal as follows:

$ system host-stor-add storage-1 journal 97535924-c9df-4409-ac02-439a638f052f
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| osdid | None |
| function | journal |
| state | configured |
| journal_location | None |
| journal_size_gib | None |
| journal_path | None |
| journal_node | None |
| uuid | cbfbf28f-d7b8-4437-9402-611787b1b6f7 |
| ihost_uuid | b3f5dd04-f8e8-4f3e-9011-791d16d7abe8 |
| idisk_uuid | 97535924-c9df-4409-ac02-439a638f052f |
| tier_uuid | None |
| tier_name | None |
| created_at | 2019-11-13T17:45:15.399017+00:00 |
| updated_at | None |
+------------------+--------------------------------------+

$ system host-stor-list storage-1
+--------------------------------------+----------+-------+------------+----------------------------
| uuid | function | osdid | state | idisk_uuid | journal_path | journal_node | journal_size_gib | tier_name |
+--------------------------------------+----------+-------+------------+--------------------------------------+-------------------------------------------------+----------------+------------------+-----------+
| 3f9a6686-2b76-4278-b997-b029211d7632 | osd | 1 | configured | eff63bd4-aaa2-486e-bcfc-cffedf07ee3a | /dev/disk/by-path/pci-0000:86:00.0-nvme-1-part2 | /dev/nvme3n1p2 | 1 | storage |
| cbfbf28f-d7b8-4437-9402-611787b1b6f7 | journal | None | configured | 97535924-c9df-4409-ac02-439a638f052f | None | None | None | None |
| d3bdd1ab-30ed-46b6-afd0-ed18a84b2253 | osd | 0 | configured | c5705e5b-fdc5-4dd6-8a74-7878ddee0521 | /dev/disk/by-path/pci-0000:85:00.0-nvme-1-part2 | /dev/nvme2n1p2 | 1 | storage |
+--------------------------------------+----------+-------+------------+----------------------------

4. Update the journal size and location:
[sysadmin@controller-0 ~(keystone_admin)]$ system host-stor-update --journal-location cbfbf28f-d7b8-4437-9402-611787b1b6f7 --journal-size 10 3f9a6686-2b76-4278-b997-b029211d7632
+------------------+-------------------------------------------------+
| Property | Value |
+------------------+-------------------------------------------------+
| osdid | 1 |
| function | osd |
| state | configuring-on-unlock |
| journal_location | cbfbf28f-d7b8-4437-9402-611787b1b6f7 |
| journal_size_gib | 10240 |
| journal_path | /dev/disk/by-path/pci-0000:84:00.0-nvme-1-part1 |
| journal_node | /dev/nvme1n1p1 |
| uuid | 3f9a6686-2b76-4278-b997-b029211d7632 |
| ihost_uuid | b3f5dd04-f8e8-4f3e-9011-791d16d7abe8 |
| idisk_uuid | eff63bd4-aaa2-486e-bcfc-cffedf07ee3a |
| tier_uuid | f4626531-86fd-45fb-af9f-8417af1a46ff |
| tier_name | storage |
| created_at | 2019-11-13T15:49:59.427588+00:00 |
| updated_at | 2019-11-13T17:39:36.615134+00:00 |
+------------------+-------------------------------------------------+

5. Unlock storage-1 (~ Nov 13 17:52:10 UTC 2019)

Expected Behavior
------------------
1. Unlock of storage-1 in step 5

Actual Behavior
----------------
storage-1 failed to unlock

see storage-1 \var\log\2019-11-13-17-59-23_storage\puppet.log

reports Traceback
2019-11-13T17:59:55.794 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Ceph::Osd[/dev/disk/by-path/pci-0000:85:00.0-nvme-1]/Exec[ceph-osd-activate-/dev/disk/by-path/pci-0000:85:00.0-nvme-1]/returns: executed successfully
2019-11-13T17:59:55.796 Debug: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Ceph::Osd[/dev/disk/by-path/pci-0000:85:00.0-nvme-1]/Exec[ceph-osd-activate-/dev/disk/by-path/pci-0000:85:00.0-nvme-1]: The container Ceph::Osd[/dev/disk/by-path/pci-0000:85:00.0-nvme-1] will propagate my refresh event
2019-11-13T17:59:55.798 Debug: 2019-11-13 17:59:55 +0000 Ceph::Osd[/dev/disk/by-path/pci-0000:85:00.0-nvme-1]: The container Platform_ceph_osd[stor-2] will propagate my refresh event
2019-11-13T17:59:55.800 Debug: 2019-11-13 17:59:55 +0000 Exec[configure journal location stor-2](provider=posix): Executing '/usr/sbin/ceph-manage-journal location '{"osdid": 0, "journal_path": "/dev/disk/by-path/pci-0000:85:00.0-nvme-1-part2", "data_path": "/dev/disk/by-path/pci-0000:85:00.0-nvme-1-part1"}''
2019-11-13T17:59:55.802 Debug: 2019-11-13 17:59:55 +0000 Executing: '/usr/sbin/ceph-manage-journal location '{"osdid": 0, "journal_path": "/dev/disk/by-path/pci-0000:85:00.0-nvme-1-part2", "data_path": "/dev/disk/by-path/pci-0000:85:00.0-nvme-1-part1"}''
2019-11-13T17:59:55.804 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: Fixing journal location for OSD id: 0
2019-11-13T17:59:55.806 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: Traceback (most recent call last):
2019-11-13T17:59:55.808 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: File "/usr/sbin/ceph-manage-journal", line 334, in <module>
2019-11-13T17:59:55.810 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: main(sys.argv[1:])
2019-11-13T17:59:55.812 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: File "/usr/sbin/ceph-manage-journal", line 328, in main
2019-11-13T17:59:55.814 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: location['osdid'])
2019-11-13T17:59:55.816 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: File "/usr/sbin/ceph-manage-journal", line 228, in fix_location
2019-11-13T17:59:55.818 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: new_target = DISK_BY_PARTUUID + journal_uuid
2019-11-13T17:59:55.820 Notice: 2019-11-13 17:59:55 +0000 /Stage[main]/Platform::Ceph::Osds/Platform_ceph_osd[stor-2]/Exec[configure journal location stor-2]/returns: TypeError: cannot concatenate 'str' and 'NoneType' objects
2019-11-13T17:59:55.822 Error: 2019-11-13 17:59:55 +0000 /usr/sbin/ceph-manage-journal location '{"osdid": 0, "journal_path": "/dev/disk/by-path/pci-0000:85:00.0-nvme-1-part2", "data_path": "/dev/disk/by-path/pci-0000:85:00.0-nvme-1-part1"}' returned 1 instead of one of [0]
2019-11-13T17:59:55.824 /usr/share/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
2019-11-13T17:59:55.826 /usr/share/ruby/vendor_ruby/puppet/type/exec.rb:160:in `sync'

Reproducibility
---------------
100%

System Configuration
--------------------
Storage

Branch/Pull Time/Commit
-----------------------
2019-11-11_20-00-00"

[Wendy Mitchell (wmitchellwr)]

hw lab 2controllers + 2storage + workers: yow-cgcs-wildcat-92-98

[Ghada Khalil (gkhalil)]

Low priority / Not gating - this is not a typical procedure.

[Ghada Khalil (gkhalil)]

Assigning to the storage PL

[Austin Sun (sunausti)]

martin:
   would you like check this issue ?

Thanks.
BR
Austin Sun