Under any condition in which replication is enabled but api fails, cinder service-list --withreplication needs to show state of disabled rather than enabled.
In the scenarios that follow I created backends where the source and target backends used the same vserver. This is illegal and failed, yet cinder service-list --withreplication reports replication status as enabled. Only by looking in the logs do you see the failure.
Other scenarios that have the same behavior:
1) target and source aggregates are unknown, yet replictaion status shows enabled.
2) snapmirror is not licensed, yet replication status shows enabled.
[root@scsor0012900001 instances(keystone_admin)]# cinder service-list --withreplication
+------------------+---------------------------------------------------+------+---------+-------+----------------------------+--------------------+-------------------+--------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Replication Status | Active Backend ID | Frozen | Disabled Reason |
+------------------+---------------------------------------------------+------+---------+-------+----------------------------+--------------------+-------------------+--------+-----------------+
| cinder-scheduler | scsor0012900001.rtp.openenglab.netapp.com | nova | enabled | up | 2017-06-02T16:15:55.000000 | | | | - |
| cinder-volume | scsor0012900001.rtp.openenglab.netapp.com@lvm | nova | enabled | up | 2017-06-02T16:15:57.000000 | disabled | - | False | - |
| cinder-volume | scsor0012900001.rtp.openenglab.netapp.com@prodnfs | nova | enabled | up | 2017-06-02T16:15:59.000000 | enabled | - | False | - |
+------------------+---------------------------------------------------+------+---------+-------+----------------------------+--------------------+-------------------+--------+-----------------+
Yet replication is not occuring becuase the vserver origin and target are the same vserver and as such the volume names cannot be replicated
[prodnfs]
volume_backend_name=prodnfs
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_server_hostname=10.63.158.79
netapp_server_port=80
netapp_storage_protocol=nfs
netapp_storage_family=ontap_cluster
netapp_login=admin
netapp_password=Netapp123
netapp_vserver=prod
nfs_shares_config=/etc/cinder/prod_mounts
netapp_pool_name_search_pattern = *CHAD_PROD* # match all r/w FlexVols on the vServer
replication_device = backend_id:drnfs
netapp_replication_aggregate_map = backend_id:drnfs,stlfas2552_7_8_01_AggrGroup1_1:stlfas2552_7_8_02_AggrGroup2_1
[drnfs]
volume_backend_name=drnfs
volume_driver=cinder.volume.drivers.netapp.common.NetAppDriver
netapp_server_hostname=10.63.158.79
netapp_server_port=80
netapp_storage_protocol=nfs
netapp_storage_family=ontap_cluster
netapp_login=admin
netapp_password=Netapp123
netapp_vserver=prod
nfs_shares_config=/etc/cinder/prod_mounts
netapp_pool_name_search_pattern = *CHAD_PROD* # match all r/w FlexVols on the vServer
As the logs show
2017-06-02 12:14:57.042 17894 INFO cinder.volume.manager [req-ad6e441e-b028-4e2d-898d-61b3450f7a2f - - - - -] Initializing RPC dependent components of volume driver LVMVolumeDriver (3.0.0)
2017-06-02 12:14:57.547 17894 INFO cinder.volume.manager [req-ad6e441e-b028-4e2d-898d-61b3450f7a2f - - - - -] Driver post RPC initialization completed successfully.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall [-] Fixed interval looping call 'cinder.volume.drivers.netapp.dataontap.nfs_cmode.NetAppCmodeNfsDriver._handle_housekeeping_tasks' failed
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall Traceback (most recent call last):
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 137, in _run_loop
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall return f(*args, **kwargs)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall return f(*args, **kwargs)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/nfs_cmode.py", line 165, in _handle_housekeeping_tasks
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall self.ssc_library.get_ssc_flexvol_names())
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/utils/data_motion.py", line 467, in ensure_snapmirrors
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall dest_flexvol_name)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/utils/data_motion.py", line 170, in create_snapmirror
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall schedule='hourly')
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall return f(*args, **kwargs)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall return f(*args, **kwargs)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_cmode.py", line 1999, in create_snapmirror
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall self.send_request('snapmirror-create', api_args)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/utils.py", line 827, in trace_method_logging_wrapper
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall return f(*args, **kwargs)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/client_base.py", line 90, in send_request
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall return self.connection.invoke_successfully(request, enable_tunneling)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 222, in invoke_successfully
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall raise NaApiError(code, msg)
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall NaApiError: NetApp API failed. Reason - 17104:Source prod:OPENSTACK_RDO_openstack_01_AggrGroup1_1_OPENSTACK_CHAD_PROD_2 cannot be the same as the destination volume.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.loopingcall
2017-06-02 12:14:58.632 17898 INFO cinder.volume.manager [req-b1bdc590-f7d1-4b90-be64-1ec378e00ebe - - - - -] Driver initialization completed successfully.
2017-06-02 12:14:58.637 17898 INFO cinder.manager [req-b1bdc590-f7d1-4b90-be64-1ec378e00ebe - - - - -] Initiating service 3 cleanup
2017-06-02 12:14:58.641 17898 INFO cinder.manager [req-b1bdc590-f7d1-4b90-be64-1ec378e00ebe - - - - -] Service 3 cleanup completed.
2017-06-02 12:14:58.648 17898 WARNING py.warnings [req-b1bdc590-f7d1-4b90-be64-1ec378e00ebe - - - - -]
Under any condition in which replication is enabled but api fails, cinder service-list --withreplication needs to show state of disabled rather than enabled.
In the scenarios that follow I created backends where the source and target backends used the same vserver. This is illegal and failed, yet cinder service-list --withreplication reports replication status as enabled. Only by looking in the logs do you see the failure.
Other scenarios that have the same behavior:
1) target and source aggregates are unknown, yet replictaion status shows enabled.
2) snapmirror is not licensed, yet replication status shows enabled.
[root@scsor0012 900001 instances( keystone_ admin)] # cinder service-list --withreplication ------- -----+- ------- ------- ------- ------- ------- ------- ------- -+----- -+----- ----+-- -----+- ------- ------- ------- ------+ ------- ------- ------+ ------- ------- -----+- ------- +------ ------- ----+ ------- -----+- ------- ------- ------- ------- ------- ------- ------- -+----- -+----- ----+-- -----+- ------- ------- ------- ------+ ------- ------- ------+ ------- ------- -----+- ------- +------ ------- ----+ .rtp.openenglab .netapp. com | nova | enabled | up | 2017-06- 02T16:15: 55.000000 | | | | - | .rtp.openenglab .netapp. com@lvm | nova | enabled | up | 2017-06- 02T16:15: 57.000000 | disabled | - | False | - | .rtp.openenglab .netapp. com@prodnfs | nova | enabled | up | 2017-06- 02T16:15: 59.000000 | enabled | - | False | - | ------- -----+- ------- ------- ------- ------- ------- ------- ------- -+----- -+----- ----+-- -----+- ------- ------- ------- ------+ ------- ------- ------+ ------- ------- -----+- ------- +------ ------- ----+
+------
| Binary | Host | Zone | Status | State | Updated_at | Replication Status | Active Backend ID | Frozen | Disabled Reason |
+------
| cinder-scheduler | scsor0012900001
| cinder-volume | scsor0012900001
| cinder-volume | scsor0012900001
+------
Yet replication is not occuring becuase the vserver origin and target are the same vserver and as such the volume names cannot be replicated
[prodnfs] backend_ name=prodnfs driver= cinder. volume. drivers. netapp. common. NetAppDriver server_ hostname= 10.63.158. 79 server_ port=80 storage_ protocol= nfs storage_ family= ontap_cluster password= Netapp123 config= /etc/cinder/ prod_mounts pool_name_ search_ pattern = *CHAD_PROD* # match all r/w FlexVols on the vServer replication_ aggregate_ map = backend_ id:drnfs, stlfas2552_ 7_8_01_ AggrGroup1_ 1:stlfas2552_ 7_8_02_ AggrGroup2_ 1
volume_
volume_
netapp_
netapp_
netapp_
netapp_
netapp_login=admin
netapp_
netapp_vserver=prod
nfs_shares_
netapp_
replication_device = backend_id:drnfs
netapp_
[drnfs] backend_ name=drnfs driver= cinder. volume. drivers. netapp. common. NetAppDriver server_ hostname= 10.63.158. 79 server_ port=80 storage_ protocol= nfs storage_ family= ontap_cluster password= Netapp123 config= /etc/cinder/ prod_mounts pool_name_ search_ pattern = *CHAD_PROD* # match all r/w FlexVols on the vServer
volume_
volume_
netapp_
netapp_
netapp_
netapp_
netapp_login=admin
netapp_
netapp_vserver=prod
nfs_shares_
netapp_
As the logs show volume. manager [req-ad6e441e- b028-4e2d- 898d-61b3450f7a 2f - - - - -] Initializing RPC dependent components of volume driver LVMVolumeDriver (3.0.0) volume. manager [req-ad6e441e- b028-4e2d- 898d-61b3450f7a 2f - - - - -] Driver post RPC initialization completed successfully. loopingcall [-] Fixed interval looping call 'cinder. volume. drivers. netapp. dataontap. nfs_cmode. NetAppCmodeNfsD river._ handle_ housekeeping_ tasks' failed loopingcall Traceback (most recent call last): loopingcall File "/usr/lib/ python2. 7/site- packages/ oslo_service/ loopingcall. py", line 137, in _run_loop loopingcall result = func(*self.args, **self.kw) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ utils.py" , line 827, in trace_method_ logging_ wrapper loopingcall return f(*args, **kwargs) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ utils.py" , line 827, in trace_method_ logging_ wrapper loopingcall return f(*args, **kwargs) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ volume/ drivers/ netapp/ dataontap/ nfs_cmode. py", line 165, in _handle_ housekeeping_ tasks loopingcall self.ssc_ library. get_ssc_ flexvol_ names() ) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ volume/ drivers/ netapp/ dataontap/ utils/data_ motion. py", line 467, in ensure_snapmirrors loopingcall dest_flexvol_name) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ volume/ drivers/ netapp/ dataontap/ utils/data_ motion. py", line 170, in create_snapmirror loopingcall schedule='hourly') loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ utils.py" , line 827, in trace_method_ logging_ wrapper loopingcall return f(*args, **kwargs) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ utils.py" , line 827, in trace_method_ logging_ wrapper loopingcall return f(*args, **kwargs) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ volume/ drivers/ netapp/ dataontap/ client/ client_ cmode.py" , line 1999, in create_snapmirror loopingcall self.send_ request( 'snapmirror- create' , api_args) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ utils.py" , line 827, in trace_method_ logging_ wrapper loopingcall return f(*args, **kwargs) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ volume/ drivers/ netapp/ dataontap/ client/ client_ base.py" , line 90, in send_request loopingcall return self.connection .invoke_ successfully( request, enable_tunneling) loopingcall File "/usr/lib/ python2. 7/site- packages/ cinder/ volume/ drivers/ netapp/ dataontap/ client/ api.py" , line 222, in invoke_successfully loopingcall raise NaApiError(code, msg) loopingcall NaApiError: NetApp API failed. Reason - 17104:Source prod:OPENSTACK_ RDO_openstack_ 01_AggrGroup1_ 1_OPENSTACK_ CHAD_PROD_ 2 cannot be the same as the destination volume. loopingcall volume. manager [req-b1bdc590- f7d1-4b90- be64-1ec378e00e be - - - - -] Driver initialization completed successfully. f7d1-4b90- be64-1ec378e00e be - - - - -] Initiating service 3 cleanup f7d1-4b90- be64-1ec378e00e be - - - - -] Service 3 cleanup completed. f7d1-4b90- be64-1ec378e00e be - - - - -]
2017-06-02 12:14:57.042 17894 INFO cinder.
2017-06-02 12:14:57.547 17894 INFO cinder.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:57.679 17898 ERROR oslo.service.
2017-06-02 12:14:58.632 17898 INFO cinder.
2017-06-02 12:14:58.637 17898 INFO cinder.manager [req-b1bdc590-
2017-06-02 12:14:58.641 17898 INFO cinder.manager [req-b1bdc590-
2017-06-02 12:14:58.648 17898 WARNING py.warnings [req-b1bdc590-