Unable to remove Docker Proxy through system service parameter though docker param is deleted

[Yatindra Shashi]

Brief Description
-----------------
During installation, added docker proxy but removed after changed in OAM network which goes without proxy but still service-parameter does not remove it. I have applied parameter and done host-lock, unlock as well.

Severity
--------
Provide the severity of the defect.

<Minor: Effect those changig docker proxy

Steps to Reproduce
------------------
1. Configure Docker Proxy during installation
2. Remove after installation using system service-parameter-delete
3. Apply $ system service-parameter-apply

Expected Behavior
------------------
Should remove proxy and should be able to download docker images from K8s.

Actual Behavior
----------------'
Failed to pull image:

Warning Failed 17s (x3 over 55s) kubelet, controller-0 Failed to pull image "minio/k8s-operator:1.0.4": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: proxyconnect tcp: dial tcp: lookup proxy-xx.xx.com on 192.168.204.1:53: no such host

Reproducibility
---------------
yes

System Configuration
--------------------
Simplex AIO

STX 3.0

Timestamp/Logs
--------------

[sysadmin@controller-0 ~(keystone_admin)]$ system service-parameter-list
+--------------------------------------+----------+-------------+---------------------+-------+-------------+----------+
| uuid | service | section | name | value | personality | resource |
+--------------------------------------+----------+-------------+---------------------+-------+-------------+----------+
| 10b85091-af07-47a2-ae0b-b5ddfbd29236 | horizon | auth | lockout_retries | 3 | None | None |
| ea52483a-963f-49ee-bd3b-b33fd9de3291 | horizon | auth | lockout_seconds | 300 | None | None |
| 6b22fab8-59b0-4005-8625-0169206de7e6 | radosgw | config | fs_size_mb | 25 | None | None |
| 8b6dea76-2572-4797-812b-922fe8ce12f6 | http | config | http_port | 8080 | None | None |
| fbd7d0a3-0fdf-492e-b693-d3448e70a5a7 | http | config | https_port | 8443 | None | None |
| 40352e1d-f0c9-42f5-851d-d4f65d3b57a0 | radosgw | config | service_enabled | false | None | None |
| 507729eb-423b-4bad-bbd7-7a59575b40b7 | identity | config | token_expiration | 3600 | None | None |
| 69a26825-e3d2-4071-ad18-280494dcc47b | platform | maintenance | controller_boot_tim | 1200 | None | None |
| | | | eout | | | |
| | | | | | | |
| 4614f774-00ed-4cba-ba91-f309c1daa0e4 | platform | maintenance | heartbeat_degrade_t | 6 | None | None |
| | | | hreshold | | | |
| | | | | | | |
| ab9f830d-5cc0-4cc7-adf1-458e0c66298d | platform | maintenance | heartbeat_failure_a | fail | None | None |
| | | | ction | | | |
| | | | | | | |
| 5af24981-d50d-45d9-96d4-3864915cc216 | platform | maintenance | heartbeat_failure_t | 10 | None | None |
| | | | hreshold | | | |
| | | | | | | |
| 9db2d014-4da8-4fde-885f-3a0660652deb | platform | maintenance | heartbeat_period | 100 | None | None |
| 0ad27805-2764-4853-8148-57c5f070f453 | platform | maintenance | mnfa_threshold | 2 | None | None |
| 59140647-22f0-46cf-8684-4bca500098a3 | platform | maintenance | mnfa_timeout | 0 | None | None |
| 9c1fd83a-ada2-42a2-a4e0-fa83eea50be1 | platform | maintenance | worker_boot_timeout | 720 | None | None |
+--------------------------------------+----------+-------------+---------------------+-------+-------------+----------+

$ sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf

[Service]
Environment="HTTP_PROXY=http://proxy-xx..com:911"
Environment="HTTPS_PROXY=http://proxy-xx.com:911"
Environment="NO_PROXY=localhost,127.0.0.1,registry.local,192.168.204.1,192.168.204.2,172.28.235.202,172.28.235.203"

 Workaround Tried
 ----------
 Describe workaround if available

Deleted the /etc/systemd/system/docker.service.d/http-proxy.conf file
$ systemctl daemon-reload
$sudo service docker restart

But it stuck on restarting docker service

[Ghada Khalil (gkhalil)]

Marking for investigation in stx.4.0 as this is not a very common scenario

[Ghada Khalil (gkhalil)]

Moving to stx.5.0 as this is not a very common scenario

[Jerry Sun (jerry-sun-u)]

service parameters are handled in "config/sysinv/sysinv/sysinv/sysinv/conductor/manager.py"

here is an example commit that adds service parameters, for reference on how service parameters work in general:
https://opendev.org/starlingx/config/commit/95d8bb436b625c82e78ebb2a2134e0e861bd5574

"config/sysinv/sysinv/sysinv/sysinv/conductor/manager.py" drives the application of puppet classes (platform::kubernetes::master::change_apiserver_parameters in this case) when service parameters change.

The issue looks like the service parameter change failed to drive the puppet proxy configuration changes.
a good place to start is seeing if the puppet changes caused by the service parameters failed.
The puppet classes that control the docker proxy config can be found here: stx-puppet/puppet-manifests/src/modules/platform/manifests/docker.pp

puppet logs can be found at "/var/log/puppet" on a running system

[Ghada Khalil (gkhalil)]

Lowering the priority as this is not a common scenario.

[Scott V Kamp (skamp007)]

medium - low ?? hows that when im experiencing this exact issue right now

[Scott V Kamp (skamp007)]

Warning FailedCreatePodSandBox 10s kubelet, controller-1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "82cc11cb98bb725719107db49bccbf971589feddde14aa57c52684357272054b": Multus: [kube-system/calico-kube-controllers-5cd4695574-t9sdt]: error adding container to network "chain": delegateAdd: error invoking conflistAdd - "chain": conflistAdd: error in getting result from AddNetworkList: error getting ClusterInformation: Get https://[10.96.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default: proxyconnect tcp: dial tcp 46.23.86.244:3128: connect: no route to host