Comment 8 for bug 2018318

I left a very large comment on Gerrit but I'll add it here for better visibility.

FWIW, I think the problem is legit and needs to be addressed. I'm gonna change the title and the subject to make it clearer but I also think that the solution isn't simple at first and requires some design discussion, hence the Wishlist status.

Now, the comment I wrote explaining my -1 (you can find it here https://review.opendev.org/c/openstack/nova/+/864760/comment/b2b03637_f15d6dd2/ )

=================
> Just because you say so? =)

> Can you provide a more technical explanation on why not? I mean, why would that be wrong? Or, what alternative would be better, and why?

Sorry, that's kind of a non-documented design consensus (or a tribal knowledge if you prefer)
We, as the Nova community, want to keep the RequestSpec.availability_zone record as an immutable object, that is only set when creating the RequestSpec, so then we know whether the user wanted to pin the instance to a specific AZ or not.

> What is your proposal? We see the following two different alternatives so far. [...]

Maybe you haven't seen my proposal before, but I was talking of https://review.opendev.org/c/openstack/nova/+/469675/12/nova/compute/api.py#1173 that was merged.
See again my comment https://review.opendev.org/c/openstack/nova/+/864760/comments/4a302ce3_9805e7c6
TBC, lemme explain the problem and what we need to fix : if an user creates an instance with an image and asking to create a volume on that image, then we need to modify the AZ for the related Request if and only if cross_az_attach=False

Now, let's discuss the implementation :
1/ we know that volumes are created way later in the instance boot by the compute service, but we do pass the information of the instance.az to Cinder to tell it to create a volume within that AZ if cross_az_attach=False :
https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/virt/block_device.py#L427
https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/virt/block_device.py#L53-L78

2/ unfortunately,instance.availability_zone is only trustworthy if the instance is pinned to an AZ

3/ we know that at the API level, we're able to know whether we will create a volume based on an image since we have the BDMs and we do check them :
https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1460
https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1866
https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1960-L1965C43

4/ Accordingly, we are able to follow the same logic than in https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1396-L1397 by checking the BDMs and see whether we gonna create a volume. If so, we SHALL pin the AZ exactly like https://github.com/openstack/nova/blob/b3fdd7ccf01bafb68e37a457f703b79119dbfa86/nova/compute/api.py#L1264

Unfortunately, since the user didn't specify an AZ, Nova doesn' know which AZ to pin the instance to. Consequently, we have multiple options :

1/ we could return an exception to the user if he didn't pinned the instance. That said, I really don't like this UX since the user doesn't know whether cross_az_attach is False or not
2/ we could document the fact that cross_az_attach only works with pre-created volumes.
3/ we could pre-create the volume way earlier at the API level and get its AZ.
4/ we could augment the RequestSpec to have a field saying 'pinned' or something else that the scheduler would honor on a move operation even if RequestSpec.az is None

As you see, all those options need to be correctly discussed, so IMHO I'd prefer you to draft a spec so the nova community could address those points so we could find an approved design solution.

HTH.