Running kolla-ansible Train on RHEL8. On rare occasions (approximately 1/50) recently created DNS records fail deletion at https://github.com/openstack/designate/blob/60edc59ff765b406e4b936deb4d200a2d9b411ce/designate/notification_handler/base.py#L113 when the corresponding VM is deleted.
When this happens we see the "Record not found in recordset" error in our debug logs. I added extra code to step through the recordset and found that the record is there before and after the failed "if." It appears that record_to_delete does not match the record that exists in the recordset.
Code:
LOG.debug(
'Deleting record: %s / recordset: %s / zone: %s',
record_to_delete['id'], recordset_id, zone_id
)
try:
recordset = self.central_api.get_recordset(
context, zone_id, recordset_id
)
for thisrecord in recordset.records:
LOG.debug('Record %s found in recordset %s', thisrecord['id'], recordset_id)
LOG.debug('Dumping recordset %s',recordset)
# Record not longer in recordset. Lets abort.
if record_to_delete not in recordset.records:
LOG.debug(
'Record %s not found in recordset %s',
record_to_delete['id'], recordset_id
)
for thisrecord in recordset.records:
LOG.debug('Record %s found in recordset %s', thisrecord['id'], recordset_id)
return
Output:
2023-04-04 13:24:31.590 35 DEBUG designate.notification_handler.enhanced_base [req-c1ef2ad9-20d8-4956-9846-e65bea436d7e - - - - -] Deleting record: c5d50ad5-cc1e-4192-9e64-e16961602941 / recordset: ad032ea6-8c6f-45f8-8396-5c5532603010 / zone: c46a3fdf-6a2b-4126-88ad-9516856ad9a5 _update_or_delete_recordset /usr/lib/python3.6/site-packages/designate/notification_handler/enhanced_base.py:114
2023-04-04 13:24:31.612 35 DEBUG designate.notification_handler.enhanced_base [req-c1ef2ad9-20d8-4956-9846-e65bea436d7e - - - - -] Record c5d50ad5-cc1e-4192-9e64-e16961602941 found in recordset ad032ea6-8c6f-45f8-8396-5c5532603010 _update_or_delete_recordset /usr/lib/python3.6/site-packages/designate/notification_handler/enhanced_base.py:121
2023-04-04 13:24:31.613 35 DEBUG designate.notification_handler.enhanced_base [req-c1ef2ad9-20d8-4956-9846-e65bea436d7e - - - - -] Dumping recordset <RecordSet id:'ad032ea6-8c6f-45f8-8396-5c5532603010' type:'A' name:'stackwatch-trust-6.infra-stackwatch.de6.ourdomain.com.' zone_id:'c46a3fdf-6a2b-4126-88ad-9516856ad9a5'> _update_or_delete_recordset /usr/lib/python3.6/site-packages/designate/notification_handler/enhanced_base.py:122
2023-04-04 13:24:31.613 35 DEBUG designate.notification_handler.enhanced_base [req-c1ef2ad9-20d8-4956-9846-e65bea436d7e - - - - -] Record c5d50ad5-cc1e-4192-9e64-e16961602941 not found in recordset ad032ea6-8c6f-45f8-8396-5c5532603010 _update_or_delete_recordset /usr/lib/python3.6/site-packages/designate/notification_handler/enhanced_base.py:128
2023-04-04 13:24:31.614 35 DEBUG designate.notification_handler.enhanced_base [req-c1ef2ad9-20d8-4956-9846-e65bea436d7e - - - - -] Record c5d50ad5-cc1e-4192-9e64-e16961602941 found in recordset ad032ea6-8c6f-45f8-8396-5c5532603010 _update_or_delete_recordset /usr/lib/python3.6/site-packages/designate/notification_handler/enhanced_base.py:131
I worked around the problem by checking record_to_delete['id'] against the record that I find in the recordset, and deleting that record instead:
try:
recordset = self.central_api.find_recordset(
context, {'id': recordset_id, 'zone_id': zone_id}
)
def default_json(t):
return f'{t}'
for thisrecord in recordset.records:
LOG.debug('Record %s found in recordset %s', thisrecord['id'], recordset_id)
LOG.debug('Dumping thisrecord: %s',json.dumps(thisrecord, default=default_json))
if thisrecord['id'] == record_to_delete['id']:
try:
recordset.records.remove(thisrecord)
except:
LOG.debug('Failed to delete record %s from recordset %s',thisrecord['id'], recordset_id)
return
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit 2cb42ac9f6e5273
Author: Erik Olof Gunnar Andersson <email address hidden>
Date: Tue Apr 4 22:04:56 2023 -0700
Use ids when removing sink managed records
If the record status changes during the removal process,
the current implementation will fail.
Closes-Bug: 2015762
Change-Id: Iebe609e5f365d0