tempest complains 'Share network xxx could not be found.'

[Sam Wan]

tempest complains 'Share network xxx could not be found.' how ever manila share-network-show can see the share network.

================================
{1} setUpClass (manila_tempest_tests.tests.api.test_access_rules_metadata.AccessRulesMetadataTest) [0.000000s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/opt/stack/new/tempest/tempest/test.py", line 188, in setUpClass
        six.reraise(etype, value, trace)
      File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
        raise value
      File "/opt/stack/new/tempest/tempest/test.py", line 181, in setUpClass
        cls.resource_setup()
      File "/opt/stack/new/manila-tempest-plugin/manila_tempest_tests/tests/api/test_access_rules_metadata.py", line 81, in resource_setup
        cls.share = cls.create_share(share_type_id=cls.share_type_id)
      File "/opt/stack/new/manila-tempest-plugin/manila_tempest_tests/tests/api/base.py", line 506, in create_share
        result = cls.create_shares([{"args": args, "kwargs": kwargs}])
      File "/opt/stack/new/manila-tempest-plugin/manila_tempest_tests/tests/api/base.py", line 549, in create_shares
        *local_d["args"], **local_d["kwargs"])
      File "/opt/stack/new/manila-tempest-plugin/manila_tempest_tests/tests/api/base.py", line 460, in _create_share
        share = client.create_share(**kwargs)
      File "/opt/stack/new/manila-tempest-plugin/manila_tempest_tests/services/share/v2/json/shares_client.py", line 281, in create_share
        extra_headers=experimental, version=version)
      File "/opt/stack/new/manila-tempest-plugin/manila_tempest_tests/services/share/v2/json/shares_client.py", line 70, in post
        headers=headers)
      File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 283, in post
        return self.request('POST', url, extra_headers, headers, body, chunked)
      File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 687, in request
        self._error_checker(resp, resp_body)
      File "/opt/stack/new/tempest/tempest/lib/common/rest_client.py", line 793, in _error_checker
        raise exceptions.NotFound(resp_body, resp=resp)
    tempest.lib.exceptions.NotFound: Object not found
    Details: {'code': 404, 'message': 'Share network 8f437b19-cef0-4f5d-92a0-46d50409f36a could not be found.'}
==================================

actually thee share network is there:
====================================
$ manila share-network-list
+--------------------------------------+---------------+
| id | name |
+--------------------------------------+---------------+
| 8f437b19-cef0-4f5d-92a0-46d50409f36a | enas_sharenet |
+--------------------------------------+---------------+
=======================================

Seems the test cases failed on all DellEMC manila drivers with DHSS=true.
The errors appeared about 2 weeks ago.

example build logs:
=====================================
http://publiclogs.emc.com/25/703025/5/check/DellEMC_PowerMAX_Manila/e8a5325/DellEMC_PowerMAX_Manila/254/console.html
http://publiclogs.emc.com/25/703025/5/check/DellEMC_Unity_Manila_dhss_true/f18dc3b/DellEMC_Unity_Manila_dhss_true/242/console.html
=====================

[Martin Kopec (mkopec)]

Moving this to manila project as it's a manila plugin.

[Douglas Viroel (dviroel)]

Hi @Saw Wan, thanks for reporting this issue. We'll discuss about this on our next weekly meeting [1], on March 26th at 15:00 UTC. Hope that you can join us there. If you won't be able to attend, you still can read the logs that will be posted here[2].
Thanks!

[1] https://wiki.openstack.org/wiki/Manila/Meetings
[2] http://eavesdrop.openstack.org/meetings/manila/

[Sam Wan (sam-wan)]

Hi Douglas,

That's too late for me and I might not be able to attend.
However please do let us know how we can recover from this error.

thanks and regards

San

[Maurice Escher (maurice-escher)]

Did it stop working after the fix of https://bugs.launchpad.net/manila/+bug/1861485?
I have seen similar issues, were tempest was looking in the wrong project and it was working by accident before because share network get by ID was possible from any project scope.

[Sam Wan (sam-wan)]

Hi Maurice Escher,

Thanks for the hint.
It looks like so.
But how can we fix this?
Seems there's no way to set project id when creaing share-network
=======================
usage: manila share-network-create [--neutron-net-id <neutron-net-id>]
                                   [--neutron-subnet-id <neutron-subnet-id>]
                                   [--name <name>]
                                   [--description <description>]
                                   [--availability-zone <availability_zone>]

Create description for network used by the tenant.

Optional arguments:
  --neutron-net-id <neutron-net-id>, --neutron-net_id <neutron-net-id>, --neutron_net_id <neutron-net-id>, --neutron_net-id <neutron-net-id>
                        Neutron network ID. Used to set up network for share
                        servers.
  --neutron-subnet-id <neutron-subnet-id>, --neutron-subnet_id <neutron-subnet-id>, --neutron_subnet_id <neutron-subnet-id>, --neutron_subnet-id <neutron-subnet-id>
                        Neutron subnet ID. Used to set up network for share
                        servers. This subnet should belong to specified
                        neutron network.
  --name <name> Share network name.
  --description <description>
                        Share network description.
  --availability-zone <availability_zone>, --availability_zone <availability_zone>, --az <availability_zone>
                        Availability zone in which the subnet should be
                        created. Share networks can have one or more subnets
                        in different availability zones when the driver is
                        operating with 'driver_handles_share_servers'
                        extra_spec set to True. Available only for
                        microversion >= 2.51. (Default=None)
===================================

Besides, tempest test cases create projects and we can NOT predict the project ID before tempest run.
====================================
...
| c071eeaad8364f719f1d3a84a1b79d27 | tempest-SharesMetadataTest-1308851634 |
| e1dd6b1fbf9c4c8ba69d8f8cbb9a93da | tempest-SharesMetadataTest-257143021 |
==================================

thanks and regards
Sam

[Sam Wan (sam-wan)]

Hi Maurice Escher,

I've confirmed that the error is indeed caused by merge of https://bugs.launchpad.net/manila/+bug/1861485.
Once I revert the change, the test cases can pass.
I know the patch is necessary to fix an important security bug, but I don't know how our CI's can adapt to this change based on below facts:
1. there's no way to set project id when creating manila share network
2. tempest create projects for cases and we can't predict the project before it run.

Thanks and regards
Sam

[Sam Wan (sam-wan)]

Hi Maurice Escher and Douglas,

This issue can be worked out by using the 'test_accounts_file' approach provided by tempest:
https://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/test-accounts.html
o set use_dynamic_credentials=False
o Use test_accounts_file.yml with manually created openstack accounts, non-admin accounts should be put in non-admin projects and admin accounts should be put in admin project.
Below is an example test accounts file for MANILA_TEMPEST_CONCURRENCY=1
( Normally, 2 x MANILA_TEMPEST_CONCURRENCY non-admin users and MANILA_TEMPEST_CONCURRENCY * admin users are required for running tempest test cases successfully. )
========================
- domain_name: Default
  password: "password"
  project_name: demo
  resources:
    network: unity-net
  username: tempest_user1
- domain_name: Default
  password: "password"
  project_name: demo
  resources:
    network: unity-net
  username: tempest_user2
- domain_name: Default
  password: "password"
  project_name: admin
  resources:
    network: unity-net
  types:
  - admin
  username: tempest_admin1
=======================

Thanks and regards.
Sam

[Goutham Pacha Ravi (gouthamr)]

Thanks for the update, Sam - Since this has a resolution; I'm marking this bug Invalid.

[Vida Haririan (vhariria)]

Additional comments http://eavesdrop.openstack.org/meetings/manila/2020/manila.2020-04-09-15.00.log.html