Debian: Auditd app applies after almost 1 hour (slower than centos), progress status remained 0% during apply test

[Karla Felix]

Brief Description

Debian: The following tc attempts to apply auditd. The auditd appears to apply (after almost 1 hr), but takes much more time than on centos. Also there is no progress status updates in that time ie. remains at 0%

functional/security/test_audit.py::test_audit_install

Severity

standard

Steps to Reproduce

auditd had a value of eg. 1 in system service-parameter-list prior to running this test and no alarms existed.

eg. || platform | kernel | audit | 1

Run the following tc from jenkins

test_audit.py::test_audit_install

[2022-09-06 18:23:24,305] 351 DEBUG MainThread ssh.send :: Send '/usr/bin/ssh -o PubkeyAuthentication=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null sysadmin@controller-1' [2022-09-06 18:23:25,350] 351 DEBUG MainThread ssh.send :: Send 'export TMOUT=0' [2022-09-06 18:23:25,454] 351 DEBUG MainThread ssh.send :: Send 'bind 'set enable-bracketed-paste off'' [2022-09-06 18:23:25,559] 351 DEBUG MainThread ssh.send :: Send 'stat /home/sysadmin//images/tis-centos-guest.img' [2022-09-06 18:23:25,748] 351 DEBUG MainThread ssh.send :: Send 'exit' [2022-09-06 18:23:25,814] 351 DEBUG MainThread ssh.send :: Send 'kubectl create secret docker-registry docker-io --docker-server=https://index.docker.io/v1/ --docker-username='wrcloud' --docker-password='9BdjlrdzK1P0YBp'' [2022-09-06 18:23:26,395] 351 DEBUG MainThread ssh.send :: Send 'kubectl patch serviceaccount default -p '{"imagePullSecrets":[
{"name":"docker-io"}

]}'' [2022-09-06 18:23:26,550] 351 DEBUG MainThread ssh.send :: Send 'grep audit=1 /proc/cmdline' [2022-09-06 18:23:26,654] 351 DEBUG MainThread ssh.send :: Send 'find /usr/local/share/applications/helm/ name "auditd[0-9]"' [2022-09-06 18:23:29,648] 351 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-upload -n auditd /usr/local/share/applications/helm/auditd-1.0-1.tgz' [2022-09-06 18:23:43,688] 351 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-apply auditd'

Expected Behavior

Expect apply performance to improve (ie. takes 57 minutes on debian vs ~ 35 min on centos)

Also, expect progress status to be updated during this time instead of remaining at 0% throughout

Actual Behavior

Progress apply status of auditd remained at 0% for almost an hour

$ fm alarm-list
-------{}{}-------------------------{}{}{}{}{}-------------{}{}{}{}----{}{}{}{}{}-----------
Alarm ID Reason Text Entity ID Severity Time Stamp

-------{}{}-------------------------{}{}{}{}{}-------------{}{}{}{}----{}{}{}{}{}-----------
750.004 Application Apply In Progress k8s_application= warning 2022-09-06T1
                                          auditd 8:23:46.
                                                                        524573

sysinv 2022-09-06 18:24:41.485 370457 INFO sysinv.conductor.kube_app [-] Applying app auditd, overall completion: 0%
sysinv 2022-09-06 19:06:22.702 370457 INFO sysinv.conductor.kube_app [-] lifecycle hook for application auditd (1.0-1) started {'lifecycle_type': 'fluxcd-request', 'relative_timing': 'post', 'operation': 'apply', 'extra': {'rc': True}}.
sysinv 2022-09-06 19:06:25.307 370457 INFO sysinv.conductor.kube_app [-] Application auditd (1.0-1) apply completed.
sysinv 2022-09-06 19:06:25.309 370457 INFO sysinv.conductor.kube_app [-] lifecycle hook for application auditd (1.0-1) started {'mode': 'manual', 'lifecycle_type': 'manifest', 'relative_timing': 'post', 'operation': 'apply', 'extra': {'manifest_applied': True}}.
sysinv 2022-09-06 19:06:25.309 370457 INFO sysinv.conductor.kube_app [-] lifecycle hook for application auditd (1.0-1) started {'mode': 'manual', 'lifecycle_type': 'operation', 'relative_timing': 'post', 'operation': 'apply', 'extra': {'manifest_applied': True, 'app_applied': True}}.

Reproducibility

yes

System Configuration

hpe-360-g9-015-016 (duplex debian)

Load info (eg: 2022-03-10_20-00-07)

"2022-09-01_18-00-06"

Last Pass

Did this test scenario pass previously? If so, please indicate the load/pull time info of the last pass.

Use this section to also indicate if this is a new test scenario.

Alarms

Please indicate if there are any alarms observed.

If there are any alarms please list them here

Test Activity

Feature Testing, Regression Testing

Workaround

Describe workaround if available

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/audit-armada-app/+/862504
Committed: https://opendev.org/starlingx/audit-armada-app/commit/586ace04d4ad3105194ea104bd5655db5fee3c4c
Submitter: "Zuul (22348)"
Branch: master

commit 586ace04d4ad3105194ea104bd5655db5fee3c4c
Author: Karla Felix <email address hidden>
Date: Mon Oct 24 13:16:43 2022 -0300

    Decreasing repository update interval

    There is a FluxCD helmrepository for each kubernetes namespace.
    With the previous interval, 60 minutes, of the already existing
    kube-system repository does not update until this 60 minutes
    interval is over.

    This commit changes the update interval from 60 minutes to 5 minutes,
    which seems to be reasonable amount and decrease the time necessary to
    apply auditd from almost one hour to 6-7 minutes.

    Test Plan:

    PASS: Generating new audit package and upload/apply that
           package(it took 6 minutes to fully apply).

    Closes-Bug: 1994151
    Signed-off-by: Karla Felix <email address hidden>
    Change-Id: I4d47ea8bce61f407b5ffe65ca5306dea1e457dd6

[Ghada Khalil (gkhalil)]

The same issue exists for the snmp-app, so we'll make a similar change for that app as well.

Note: I opened https://bugs.launchpad.net/starlingx/+bug/1995748 (Apps take a long time to apply and the progress status remains at 0%) for the app-framework team to look into a more generic solution.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/snmp-armada-app/+/863878
Committed: https://opendev.org/starlingx/snmp-armada-app/commit/65e2bb9fa7a2a3494d9320d4b1c527eff320fdae
Submitter: "Zuul (22348)"
Branch: master

commit 65e2bb9fa7a2a3494d9320d4b1c527eff320fdae
Author: Karla Felix <email address hidden>
Date: Mon Nov 7 10:54:42 2022 -0300

    Decreasing repository update interval for snmp app

    There is a FluxCD helmrepository for each kubernetes namespace.
    With the previous interval, 60 minutes, of the already existing
    kube-system repository does not update until this 60 minutes
    interval is over.

    This commit changes the update interval from 60 minutes to 5 minutes,
    which seems to be reasonable amount and decrease the time necessary to
    apply snmp from almost one hour to 6-7 minutes.

    Test Plan:

    PASS: Generating new snmp package and upload/apply that
           package(it took 6-7 minutes to fully apply).

    Closes-Bug: 1994151
    Signed-off-by: Karla Felix <email address hidden>
    Change-Id: I04735fa64340ff8781e66218924ffb5eb602f1d7