In some cases, there can be a mismatch between group AZ and backend host.
MariaDB [cinder]> select * from groups where deleted=0;
+---------------------+---------------------+------------+---------+--------------------------------------+----------------------------------+----------------------------------+--------------+------------------------------------------------------------------------------+-------------------+------+-------------+--------------------------------------+-----------+-------------------+-----------------+--------------------+
| created_at | updated_at | deleted_at | deleted | id | user_id | project_id | cluster_name | host | availability_zone | name | description | group_type_id | status | group_snapshot_id | source_group_id | replication_status |
+---------------------+---------------------+------------+---------+--------------------------------------+----------------------------------+----------------------------------+--------------+------------------------------------------------------------------------------+-------------------+------+-------------+--------------------------------------+-----------+-------------------+-----------------+--------------------+
| 2018-05-22 19:02:35 | 2018-05-22 19:02:35 | NULL | 0 | 0241ce11-0982-4bb8-b034-f252c4436d1e | 51c74de1996847e88b5051ebfe345ec7 | 5a6e496d4c994209b8ac6e28a3aae6e6 | NULL | controller02@az-2:vxlan-vm-111-76#VMwareVcVmdkDriver | az-1
In this case, the group was created without any AZ. The backend "az-2:vxlan-vm-111-76#VMwareVcVmdkDriver" is in az-2, but the group's AZ is the default AZ 'az-1'. As a result, volume creation in the default AZ with the above group ID fails with error "no valid backend" in the scheduler.
Fix proposed to branch: master
Review: https:/