[cinder] current zone fallback behaviour results to volume creation failure

[Serg Lystopad]

Detailed bug description:
Actual cinder zones are zones specified for cinder-volume processes. So to create
additional cinder zone we should spawn one more cinder-volume process with
`storage_availability_zone` value set to new zone name.
By default there is only one cinder zone in MOS - 'nova'. 'nova' is also default value for:
`storage_availability_zone` and `default_availability_zone` options.

If we have cinder set-up with two or more zones and none of them is `nova`,
cinder CLI fails to create volume (cinder create without --availability-zone option).
As result 'Any availability zone' option in volume creation dialog of Horizon UI also fails
to create volume (volume falls into error state).
It is result of current fallback behaviour:

if zone isn't specified in request to API cinder tries to determine zone for volume
https://github.com/openstack/cinder/blob/stable/liberty/cinder/volume/flows/api/create_volume.py#L273

if `default_availability_zone` hasn't been configured (empty string) it falls back to `storage_availability_zone`
which is as well `nova` by default and then reqults to 'unclear' error message 'Availability zone is invalid' (bad UX).
`storage_availability_zone` makes no sence for api process. It is required for volume process but it may
be absent in config for api (assumption that both processes use the same config is wrong because processes may
at all recide on different physical hosts).
Current behaviour works if value of `default_availability_zone` option is in list on 'actual' AZ,
but it results to all volumes are created in 'default' AZ and therefor most probably to overflow of storage backend for default AZ

Another point is related to Horizon which relies on cinder.
'Any availability zone' option may confuse users (bad UX) (user may perceive that it works in the same way as similar option for Nova.
Nova randomy selects AZ for instance in case AZ hasn't been exactly specified).

So the better approach would be:
1) fallback to one of the 'actual' zones
2) return more clear error if `default_availability_zone` is not in the list of 'actual' availability zones

Steps to reproduce:
stop default cinder-volume process on controllers
start two or more cinder-volume in different zones (zone1, zone2)
check that cinder availability-zone-list output doesn't contain zone `nova`
create volume with `cinder create 1` (or create volume using 'Any availability zone' option in Horizon)

Expected results:
volume is created in zone 1 or zone2

Actual result:
volume is in error state

Description of the environment:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[Serg Lystopad (slystopad)]

workaround

[Serg Lystopad (slystopad)]

Workaround is to apply patch https://bugs.launchpad.net/mos/+bug/1625238/comments/1
and set in conf for cinder-api:
[DEAFULT]
storage_availability_zone=''
default_availability_zone=''

[Vitaly Sedelnik (vsedelnik)]

Looks like miscofiguration issues - default_availability_zone param should contain name of existing AZ. It's ok to fail if default_availability_zone doesn't exist. It shouldn't select random availability zone. Current behavior is expected, setting to Invalid.

[Serg Lystopad (slystopad)]

Then 'Any availability zone' should be renamed to 'Default availability zone' in Horizon, because for in nova 'Any availability zone' randomly selects zone for VM, so using same option name for volumes confuse users.

[Serg Lystopad (slystopad)]

And clear error message should be returned instead of 'Invalid availability zone' in case `default_availability_zone` does not exist.