vcenter.yaml not being created due to possible vcenter_registration metadata bug

[greg]

I am running MAAS: `3.1.0`

I want to start this out by saying that i'm not 100% sure that this is a bug, but after the troubleshooting i've done, I believe it is one. For completeness, I opened this yesterday https://discourse.maas.io/t/vcenter-registration-variable-question/6222 and was told to open a bug report.

The issue I am having is that my esxi deployments are not being joined to vcenter. They are successfully completing deployments in MAAS, but fail to join vcenter even though I have vcenter hostname and creds specified in the MAAS settings which I believe *should* cause the metadata script `vendor_data.py` to write a vcenter.yaml file to `/altbookbank/maas/vcenter.yaml` which is NOT being created for me so I dug in to find out why.

I added some additional logging to the `vendor_data.py` script and confirmed that the `vcenter_registration` variable is `None` which is why the vcenter.yaml file isn't being created. This bit of code is what i'm referring to: https://github.com/maas/maas/blob/master/src/metadataserver/vendor_data.py#L352

*NOTE:* The additional debug logging statements I added are printing to stdout which seems to print out in the regiond logs. Any statement I added will be prefaced with `GSPTEST` - Here is an example from regiond (these are in the tar.gz logs i'm uploading as well):
```
2022-07-20 15:18:05 stdout: [info] GSPTEST: Starting generate_vcenter_config
2022-07-20 15:18:05 stdout: [info] GSPTEST: confirmed we are admin
2022-07-20 15:18:05 stdout: [info] GSPTEST: vcenter_registration is: None
```

In the code it looks like the `vcenter_registration` variable is something that should be specified on each nodes metadata, the problem is that I can't figure out how to add that via the GUI or CLI to the node. I could not find anything online about needing to specify this metadata manually per node so that is why I believe this to be a bug.

As an additional bit of info, when searching around before opening this bug I discovered this issue which seems to have the same behavior I am experiencing. If vcenter.yaml fails to get created (and creds aren't hard coded into the `vcenter` script built into the image), the `vcenter` script will fail as it has no config file to get certain arguments such as `vc_server`, `vc_username`, `vc_password` and fail completely with `vcenter: error: the following fields are missing: vc_server, vc_username, vc_password` - https://discourse.maas.io/t/esxi-autoregistration-doesnt-work/4600/2

I will continue do some more testing and report back any findings. I don't want to permanently rip out the `vcenter_registration` check but am going to workaround it so i can produce a vcenter.yaml and see if things work as expected. After that I plan to dig through the code to find the source of how this metadata is supposed to get populated (I'm guessing it should be automatic when an operating system of ESXI is chosen). If I find a fix i'll make a PR!

#############

EDIT 1: I edited the code a bit to have the `config` variable (here: https://github.com/maas/maas/blob/master/src/metadataserver/vendor_data.py#L359-L367) get set even if vcenter_registration is None and confirmed that it does get the values from MAAS as expected so I think if we can solve the issue with the metdata for vcenter_registration then all should work. Confirmed that commenting out the vcenter_registration check did allow the host to be automatically joined to vcenter and the vcenter.yaml was created and used.

2022-07-21 14:15:27 stdout: [info] GSPTEST: Starting generate_vcenter_config
2022-07-21 14:15:27 stdout: [info] GSPTEST: confirmed we are admin
2022-07-21 14:15:27 stdout: [info] GSPTEST: vcenter_registration is: None
2022-07-21 14:15:27 stdout: [info] GSPTEST: configs is: {'vcenter_server': 'vcenter.fqdn.here', 'vcenter_username': '<email address hidden>', 'vcenter_password': 'password_was_here', 'vcenter_datacenter': 'TestDatacenter'}

#############

EDIT 2: Doing a bit more digging on this and was able to track down the proper postgres DB table the metadata is stored in and was able to manually create the `vcenter_registration` key with a value of `true` - Based on the python code I saw the value could really be set to anything, as long as it exists then vcenter.yaml will be created.

Example:

```
maasdb=# INSERT INTO maasserver_nodemetadata (created, updated, key, value, node_id)
maasdb-# VALUES ("2022-07-22 16:09:19.048035+00", "2022-07-22 16:09:19.048035+00", "vcenter_registration", "true", 9);
```

Deployments are joining to vcenter via parameters supplied through vcenter.yaml as expected now!

#############

EDIT 3: This will likely be my final edit to this post. I have discovered that the `vcenter_registration` metadata can ONLY be supplied via the CLI / API. You cannot do it via the GUI currently which I found by digging through the code that builds the MAAS UI. You can see that code here: https://github.com/canonical-web-and-design/maas-ui/blob/487123a55609b3fac7a5dc7483ab12616d0f99b9/src/app/store/machine/slice.ts#L943-L949

You can see various pieces of metadata that are passable here, via the GUI. But you will not find vcenter_regsitration. This is an option you WILL see for the `deploy` command via the CLI, which in turn operates via the MAAS API and allows metadata to be passed and vcenter information to be provided from MAAS at deployment time.

As an interesting bit if information I found that if you deploy a MAAS node using the CLI and passing the `vcenter_registration=true` option, it gets PERMANENTLY written to the database for that node. This means that future deployments, - even from the GUI!! - cause vcenter.yaml to be created and vcenter joins to occur on that specific node. You will either need to pass `vcenter_registration=false` at runtime or remove the node from MAAS completely and re-add it which will cause the node to generate a new UID and not have the metadata relationship for `vcenter_registration`

Due to these findings I don't think this is a bug but rather a poorly documented feature. It was not clear to me that this could not be done via the GUI and only after reverse engineering the code a bit was I able to test and confirm this.

[Jerzy Husakowski (jhusakowski)]

It looks like the GUI does not offer any means of passing the `vcenter_registration=true` option during machine deployment. It is also likely that MAAS does not remove that option when releasing the machine.