Failed check if a log retention policy exists in opensearch

[Grzegorz Bialas]

What happened:

During upgrade from Zed to Antelope Task "opensearch : Check if a log retention policy exists" failed with:

TASK [opensearch : Check if a log retention policy exists] *********************************************************************************************************************************************
Tuesday 09 January 2024 10:55:38 +0100 (0:00:02.012) 0:12:25.474 *******
fatal: [redacted ]: FAILED! => changed=false
  action: uri
  connection: close
  content: '{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"}],"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];"},"status":503}'
  content_length: '271'
  content_type: application/json; charset=UTF-8
  elapsed: 0
  json:
    error:
      reason: 'blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];'
      root_cause:
      - reason: 'blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];'
        type: cluster_block_exception
      type: cluster_block_exception
    status: 503
  msg: 'Status code was 503 and not [200, 404]: HTTP Error 503: Service Unavailable'
  redirected: false
  status: 503
  url: http://172.16.4.1:9200/_plugins/_ism/policies/retention

What you expected to happen:
Opensearch was up and ready to pass this check.

How to reproduce it (minimal and precise):

Run upgrade from Zed release to Antelope

**Environment**:
OS: Rocky Linux release 9.2

Kernel: Linux spsrc-nova-03 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Docker version 24.0.6, build ed223bc