cinder-manage commands crash during db sync

[Salvatore]

I'm stuck in deploying Cinder using Openshift-Ansible because I get this error

Traceback (most recent call last):
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/types.py", line 843, in __call__
    value = self.ip_address(value)
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/types.py", line 728, in __call__
    self.version_checker(value)
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/types.py", line 748, in _check_both_versions
    raise ValueError("%s is not IPv4 or IPv6 address" % address)
ValueError: infra1_cinder_api_container-789627db is not IPv4 or IPv6 address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/types.py", line 846, in __call__
    value = self.hostname(value)
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/types.py", line 799, in __call__
    raise ValueError("%s is an invalid hostname" % value)
ValueError: infra1_cinder_api_container-789627db is an invalid hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/cfg.py", line 611, in _check_default
    self.type(self.default)
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/types.py", line 849, in __call__
    "%s is not a valid host address" % (value,))
ValueError: infra1_cinder_api_container-789627db is not a valid host address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/openstack/venvs/cinder-23.0.1/bin/cinder-manage", line 5, in <module>
    from cinder.cmd.manage import main
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/cinder/cmd/manage.py", line 67, in <module>
    from cinder.backup import rpcapi as backup_rpcapi
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/cinder/backup/__init__.py", line 21, in <module>
    from cinder.common import config
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/cinder/common/config.py", line 90, in <module>
    help='Name of this node. This can be an opaque '
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/cfg.py", line 1175, in __init__
    **kwargs)
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/cfg.py", line 595, in __init__
    self._check_default()
  File "/openstack/venvs/cinder-23.0.1/lib/python3.6/site-packages/oslo_config/cfg.py", line 616, in _check_default
    'opt': self.type})
oslo_config.cfg.DefaultValueError: Error processing default value infra1_cinder_api_container-789627db for Opt type of HostAddress.

This happens during the run of the command cinder-manager db sync.

I've tried naturally to troubleshoot it but I could find any way issues.

I've looked at the etc/hosts file and what I have there is

[root@infra1_cinder_api_container-789627db bin]# cat /etc/hosts
127.0.0.1 localhost
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
### BEGIN OPENSTACK-ANSIBLE MANAGED BLOCK ###
10.10.2.168 infra1_cinder_api_container-789627db.openstack.local infra1-cinder-api-container-789627db infra1_cinder_api_container-789627db infra1_cinder_api_container-789627db

[Dmitriy Rabotyagov (noonedeadpunk)]

I believe you recieve that because you have in /etc/cinder/cinder.conf my_ip set to the infra1_cinder_api_container-789627db

By default, this value is managed with variable cinder_management_address that is set to {{ ansible_host }}: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/cinder_all.yml#L23

OpenStack-Ansible dynamic inventory sets ansible_host to the IP address of the container and not container name.

Anyway, you can override this variable to any sutiable value (like container_address or main ipv4 address) in your user_variables.yml

[Salvatore (apocalipse89)]

I've looked at the cinder.conf file and my IP is an IPV4 address. so it must be something else

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/805989

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/805991

[Dmitriy Rabotyagov (noonedeadpunk)]

Ok, so it's erroring on https://opendev.org/openstack/cinder/src/branch/master/cinder/common/config.py#L87-L92

Help message explicitly says that it can be anything and not required to be an IP or FQDN while seems like oslo thinks differently.

So maybe I was wrong with marking it as invalid for cinder.

[Sofia Enriquez (lsofia-enriquez)]

Greetings,
As far as I understood, this is a configuration problem. The reporter is using IP as an IPV4 address that should be fine. Let me know if I'm missing something.
Regards,
Sofia

[Eric Harney (eharney)]

For Cinder: Evaluate "host" from HostAddressOpt to a StrOpt.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/806010

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/806011

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Gorka Eguileor <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/806011
Reason: In favor of https://review.opendev.org/c/openstack/cinder/+/806010 which came in first.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/cinder/+/807898

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/806010
Committed: https://opendev.org/openstack/cinder/commit/c995438ca2b0cfb078bfd29e585d49fa8f6a693a
Submitter: "Zuul (22348)"
Branch: master

commit c995438ca2b0cfb078bfd29e585d49fa8f6a693a
Author: Eric Harney <email address hidden>
Date: Wed Aug 25 11:09:05 2021 -0400

    Change 'host' option from HostAddressOpt to StrOpt

    HostAddressOpt is overly restrictive for this field --
    it should actually just be a StrOpt.

    Co-Author: Gorka Eguileor <email address hidden>
    Closes-Bug: #1941068
    Change-Id: I7699433b283ba9aa1bd11da4f6ff4de071f1b51d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/cinder/+/808473

[Dmitriy Rabotyagov (noonedeadpunk)]

After patch being merged, there's another issue with migrations:

```
root@aio1-cinder-api-container-c0d724ad:/# /openstack/venvs/cinder-23.1.0.dev44/bin/cinder-manage db online_data_migrations
/openstack/venvs/cinder-23.1.0.dev44/lib/python3.8/site-packages/cinder/db/sqlalchemy/models.py:136: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery.
  last_heartbeat = column_property(
/openstack/venvs/cinder-23.1.0.dev44/lib/python3.8/site-packages/cinder/db/sqlalchemy/models.py:142: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery.
  num_hosts = column_property(
/openstack/venvs/cinder-23.1.0.dev44/lib/python3.8/site-packages/cinder/db/sqlalchemy/models.py:149: SAWarning: implicitly coercing SELECT object to scalar subquery; please use the .scalar_subquery() method to produce a scalar subquery.
  num_down_hosts = column_property(
Running batches of 50 until complete.
Error attempting to run volume_use_quota_online_data_migration
Error attempting to run snapshot_use_quota_online_data_migration
+------------------------------------------+----------------+-------------+
| Migration | Total Needed | Completed |
|------------------------------------------+----------------+-------------|
| snapshot_use_quota_online_data_migration | 0 | 0 |
| volume_use_quota_online_data_migration | 0 | 0 |
+------------------------------------------+----------------+-------------+
Some migrations failed unexpectedly. Check log for details.
root@aio1-cinder-api-container-c0d724ad:/# echo $?
2
root@aio1-cinder-api-container-c0d724ad:/#
```

[Dmitriy Rabotyagov (noonedeadpunk)]

This is resolved with https://review.opendev.org/c/openstack/cinder/+/807083

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/808473
Committed: https://opendev.org/openstack/cinder/commit/8e48b4ded983956a7426e63435c1d7a91bccbe47
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 8e48b4ded983956a7426e63435c1d7a91bccbe47
Author: Eric Harney <email address hidden>
Date: Wed Aug 25 11:09:05 2021 -0400

    Change 'host' option from HostAddressOpt to StrOpt

    HostAddressOpt is overly restrictive for this field --
    it should actually just be a StrOpt.

    Co-Author: Gorka Eguileor <email address hidden>
    Closes-Bug: #1941068
    Change-Id: I7699433b283ba9aa1bd11da4f6ff4de071f1b51d
    (cherry picked from commit c995438ca2b0cfb078bfd29e585d49fa8f6a693a)
    (cherry picked from commit 498608a840b18a038811a3572ef7b5616aab810d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/807898
Committed: https://opendev.org/openstack/cinder/commit/498608a840b18a038811a3572ef7b5616aab810d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 498608a840b18a038811a3572ef7b5616aab810d
Author: Eric Harney <email address hidden>
Date: Wed Aug 25 11:09:05 2021 -0400

    Change 'host' option from HostAddressOpt to StrOpt

    HostAddressOpt is overly restrictive for this field --
    it should actually just be a StrOpt.

    Co-Author: Gorka Eguileor <email address hidden>
    Closes-Bug: #1941068
    Change-Id: I7699433b283ba9aa1bd11da4f6ff4de071f1b51d
    (cherry picked from commit c995438ca2b0cfb078bfd29e585d49fa8f6a693a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/805989
Committed: https://opendev.org/openstack/openstack-ansible-os_cinder/commit/2cf00fc1349711b6b3a0d16e7cdd26c7378852d4
Submitter: "Zuul (22348)"
Branch: master

commit 2cf00fc1349711b6b3a0d16e7cdd26c7378852d4
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Aug 25 16:02:40 2021 +0300

    Use management_address by default

    management_address is defined in group vars and by default set to
    container_address which is valid default

    Change-Id: Ib9373ba7d09845ad0fc5c8578db18ffd87e48b20
    Related-Bug: #1941068

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/805991
Committed: https://opendev.org/openstack/openstack-ansible/commit/5505d4a1465dbc19eeecef39a8947c2c746486a9
Submitter: "Zuul (22348)"
Branch: master

commit 5505d4a1465dbc19eeecef39a8947c2c746486a9
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Aug 25 16:04:42 2021 +0300

    Use cinder defaults for cinder_management_address

    ansible_host might be not always safe default and instead we would use
    management_address that is set as a default in cinder role.

    Depends-On: https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/805989
    Related-Bug: #1941068
    Change-Id: Icbfe86d984774dbb412f643cecc62b6a5f683918

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 17.3.0 release.