Misleading asynchronous user message when share services are down

[Goutham Pacha Ravi]

Description
===========
When all of the share manager services are down, or are still initializing, the scheduler has no information to make an informed decision for placement. When this happens, the user message one receives is quite confusing, and should be enhanced.

Steps to reproduce
==================

A chronological list of steps which will help reproduce the issue you hit:
* Configure manila with a share driver that fails to initialize. You can do this by misconfiguring your share backend such that the driver cannot connect to the storage system, and keeps retrying.
* Create a share, the share status will transition from "creating" to "error".
* Check user messages ('manila message-list')

Expected result
===============
The user message should suggest that there is a problem with the environment and isn't ready for provisioning yet

Actual result
=============
The user message is as follows:

$ manila message-list --resource-id b0283ea1-6a40-484a-bc5c-fea25df6f3cd
+--------------------------------------+---------------+--------------------------------------+-----------+--------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| ID | Resource Type | Resource ID | Action ID | User Message | Detail ID | Created At |
+--------------------------------------+---------------+--------------------------------------+-----------+--------------------------------------------------------------------------------------------------------------+-----------+----------------------------+
| ca92cb0f-da53-4c97-88a2-f0e6e6f02337 | SHARE | b0283ea1-6a40-484a-bc5c-fea25df6f3cd | 001 | allocate host: No storage could be allocated for this share request, AvailabilityZone filter didn't succeed. | 007 | 2020-07-07T17:10:11.000000 |
+--------------------------------------+---------------+--------------------------------------+-----------+--------------------------------------------------------------------------------------------------------------+-----------+----------------------------+

$ manila message-show ca92cb0f-da53-4c97-88a2-f0e6e6f02337
+---------------+--------------------------------------------------------------------------------------------------------------+
| Property | Value |
+---------------+--------------------------------------------------------------------------------------------------------------+
| id | ca92cb0f-da53-4c97-88a2-f0e6e6f02337 |
| resource_type | SHARE |
| resource_id | b0283ea1-6a40-484a-bc5c-fea25df6f3cd |
| action_id | 001 |
| user_message | allocate host: No storage could be allocated for this share request, AvailabilityZone filter didn't succeed. |
| message_level | ERROR |
| detail_id | 007 |
| created_at | 2020-07-07T17:10:11.000000 |
| expires_at | 2020-08-06T17:10:11.000000 |
| request_id | req-81c4d5f9-eba3-48bd-a04c-a1d7bfeaa97e |
+---------------+--------------------------------------------------------------------------------------------------------------+

The scheduler log file has this logging:

ubuntu-bionic-rax-iad-0017829885 manila-scheduler[664]: ERROR manila.scheduler.manager [None req-81c4d5f9-eba3-48bd-a04c-a1d7bfeaa97e None None] Failed to schedule create_share: No valid host was found. Failed to find a weighted host, the last executed filter was AvailabilityZoneFilter.: manila.exception.NoValidHost: No valid host was found. Failed to find a weighted host, the last executed filter was AvailabilityZoneFilter.

Environment
===========
1. Exact version of OpenStack Manila you are running. See the following
  list for all releases: trunk

2. Which storage backend did you use?
   Dummy driver, the situation is backend agnostic

3. Which networking type did you use?
   Neutron with OpenVSwitch, the problem occurs agnostic to the DHSS mode

[Goutham Pacha Ravi (gouthamr)]

The user message should suggest something along the lines of "No storage could be allocated for this share request. Share back end services are not ready yet, trying again after a while may succeed. Contact your administrator in case retrying does not help."

[Goutham Pacha Ravi (gouthamr)]

Additional comments: http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-08-13-15.01.log.html#l-143

[Hitesh Bhati (bhitesh)]

assignee:nobody -> Hitesh (bhitesh)

[Hitesh Bhati (bhitesh)]

Changed in manila:
assignee:nobody -> Hitesh (bhitesh)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/899513

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/899513
Committed: https://opendev.org/openstack/manila/commit/ecda09a14d0b31169b275ed67a9088a131fbef12
Submitter: "Zuul (22348)"
Branch: master

commit ecda09a14d0b31169b275ed67a9088a131fbef12
Author: melakualehegn <email address hidden>
Date: Sat Oct 28 02:09:22 2023 +0300

    Change misleading user message when user
    services are down

    enhance the user message when all of the share
    manager services are down or are still
    initializing
    removed a duplicate test: test_create_share_non_admin

    Closes-Bug: #1886690
    Change-Id: I168564a5b054d17762ad668ebbe4f5e7b562197b

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 18.0.0.0rc1 release candidate.