[sru] sosreport does not obfuscate a mac address when --mask is used

[nikhil kshirsagar]

[Impact]

sos 4.3 has a regression in mac address obfuscation. The file etc/netplan/50-cloud-init.yaml ends up with an unobfuscated mac address inspite of using --mask.

[TEST PLAN]

Documentation for Special Cases:
https://wiki.ubuntu.com/SosreportUpdates

[WHERE PROBLEMS COULD OCCUR]

Since we are changing the regex parser code in sos/cleaner/parsers/mac_parser.py we would need to ensure no other regex behavior is changed. The unit tests in autopkgtest will suffice to determine that.

-------------
[Other Info]

Upstream issue is https://github.com/sosreport/sos/issues/2873
Upstream MR is https://github.com/sosreport/sos/pull/2875

Reproducer details:
sos 4.2 shows correct behavior. testing shows..

/etc/netplan/50-cloud-init.yaml contains

network:
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: '52:54:00:12:34:56'
            set-name: ens3
    version: 2

4.2 sos contains the file but with the obfuscated mac address. correct behavior.

# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: '53:4f:53:cf:3a:9e'
            set-name: ens3
    version: 2

----------------------

4.3 testing shows the bug,

the /etc/netplan/50-cloud-init.yaml contains

# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: '52:54:00:12:34:56'
            set-name: ens3
    version: 2

---

generated sosreport (run with --mask) contains

# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: '52:54:00:12:34:56'
            set-name: ens3
    version: 2

So the mac address isnt being obfuscated.

Whats interesting is.. sosreport-host0-2022-03-02-kluxwcz-private_map stored in /tmp does indeed contain that entry and the obfuscated entry too..

{
    "hostname_map": {
        "autopkgtest": "host0"
    },
    "ip_map": {
        "10.0.2.0/24": "100.0.0.0/24",
        "10.0.2.15/24": "100.0.0.1/24",
        "10.0.2.255": "100.0.0.255",
        "10.0.2.3": "100.0.0.2/24",
        "91.189.89.199": "96.79.58.28",
        "5.4.0.102": "73.90.65.47",
        "5.4.0.100": "69.14.29.26",
        "5.4.0.26": "58.82.14.73",
        "224.0.0.1": "34.30.38.86",
        "91.189.89.198": "59.29.50.62",
        "3.192.30.10": "14.98.98.25",
        "5.4.0.99": "74.19.96.35",
        "10.0.2.0/28": "101.0.0.1/28",
        "10.0.2.0/30": "102.0.0.1/30",
        "192.168.200.1": "20.50.36.98",
        "192.168.200.4": "28.58.14.33",
        "192.168.200.9": "97.65.46.86",
        "192.168.201.0/24": "103.0.0.1/24",
        "192.168.201.0/25": "104.0.0.1/25",
        "224.0.0.251": "67.40.56.81",
        "239.255.255.250": "26.23.39.82",
        "123.45.67.89": "20.91.68.26",
        "192.168.0.133": "79.76.79.11"
    },
    "mac_map": {
        "52:54:00:12:34:56": "53:4f:53:79:76:69", <---
        "33:33:00:00:00:16": "53:4f:53:73:7a:0e",
        "33:33:00:00:00:02": "53:4f:53:5e:53:b8",
        "33:33:ff:12:34:56": "53:4f:53:eb:10:e9",
        "33:33:00:00:00:01": "53:4f:53:dc:de:c4",
        "01:00:5e:00:00:01": "53:4f:53:4d:05:93",
        "01:80:c2:00:00:00": "53:4f:53:b6:04:55",
        "01:80:c2:00:00:03": "53:4f:53:34:cb:ef",
        "01:80:c2:00:00:0e": "53:4f:53:f1:77:bc",
        "52:55:0a:00:02:02": "53:4f:53:08:59:72",
        "52:55:0a:00:02:03": "53:4f:53:97:23:da",
        "52:56:00:00:00:02": "53:4f:53:80:ba:7d"
    },
    "keyword_map": {},
    "username_map": {}
}

But somehow the file itself is collected without changing the ip address from original ("52:54:00:12:34:56") to the obfuscated one ("53:4f:53:79:76:69")

----

Tested with upstream sos:

root@autopkgtest:/home/nikhil/sos# git remote -v
origin https://github.com/sosreport/sos/ (fetch)
origin https://github.com/sosreport/sos/ (push)
root@autopkgtest:/home/nikhil/sos# ls
AUTHORS bin docs LICENSE man MANIFEST.in plugins_overview.py po README.md requirements.txt setup.py snap sos sos.conf sos.spec tests tmpfilesd-sos.conf
root@autopkgtest:/home/nikhil/sos#

root@autopkgtest:/home/nikhil/sos/bin# cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: '52:54:00:12:34:56'
            set-name: ens3
    version: 2

Note it contains the mac addr 52:54:00:12:34:56

Now we will run the sos report, (upstream version)

oot@autopkgtest:/home/nikhil/sos/bin# ls
sos sos-collector sosreport
root@autopkgtest:/home/nikhil/sos/bin# pwd
/home/nikhil/sos/bin
root@autopkgtest:/home/nikhil/sos/bin# ./sos report --mask

sosreport (version 4.3)

This command will collect system configuration and diagnostic
information from this Ubuntu system.

For more information on Canonical visit:

        Community Website : https://www.ubuntu.com/
        Commercial Support : https://www.canonical.com

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case id that you are generating this report for []:

 Setting up archive ...
 Setting up plugins ...
[plugin:firewall_tables] skipped command 'nft list ruleset': required kmods missing: nf_tables, nfnetlink. Use '--allow-system-changes' to enable collection.
[plugin:firewall_tables] skipped command 'iptables -vnxL': required kmods missing: nf_tables, iptable_filter.
[plugin:firewall_tables] skipped command 'ip6tables -vnxL': required kmods missing: ip6table_filter, nf_tables.
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec. Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: unix_diag, udp_diag, netlink_diag, af_packet_diag, inet_diag, xsk_diag, tcp_diag. Use '--allow-system-changes' to enable collection.
[plugin:ufw] skipped command 'ufw status numbered': required kmods missing: bpfilter, iptable_filter.
[plugin:ufw] skipped command 'ufw app list': required kmods missing: bpfilter, iptable_filter.
[plugin:wireless] skipped command 'iw list': required kmods missing: cfg80211.
[plugin:wireless] skipped command 'iw dev': required kmods missing: cfg80211.
[plugin:wireless] skipped command 'iwconfig': required kmods missing: cfg80211.
[plugin:wireless] skipped command 'iwlist scanning': required kmods missing: cfg80211.
 Running plugins. Please wait ...

  Finishing plugins [Running: systemd]
  Finished running plugins
Found 1 total reports to obfuscate, processing up to 4 concurrently

sosreport-autopkgtest-2022-03-02-abhwscl : Beginning obfuscation...
sosreport-autopkgtest-2022-03-02-abhwscl : Obfuscation completed [removed 16 unprocessable files]

Successfully obfuscated 1 report(s)

Creating compressed archive...

A mapping of obfuscated elements is available at
 /tmp/sosreport-host0-2022-03-02-abhwscl-private_map

Your sosreport has been generated and saved in:
 /tmp/sosreport-host0-2022-03-02-abhwscl-obfuscated.tar.xz

 Size 2.27MiB
 Owner root
 sha256 e9d19933cfed512a59790edf65f70a0139f8da162f406153c298bb093bfbd939

Please send this file to your support representative.

Lets open the file and see if mac address in that file is left unobfuscated,

root@autopkgtest:/tmp# cat sosreport-host0-2022-03-02-abhwscl/etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        ens3:
            dhcp4: true
            match:
                macaddress: '52:54:00:12:34:56'
            set-name: ens3
    version: 2
root@autopkgtest:/tmp#

Note,

root@autopkgtest:/tmp# ls -lrt
total 9448
drwx------ 3 root root 4096 Mar 2 15:48 systemd-private-e709306472c1435993a0b8d1f15e9dd3-systemd-resolved.service-7kMEUf
drwx------ 3 root root 4096 Mar 2 15:48 systemd-private-e709306472c1435993a0b8d1f15e9dd3-systemd-timesyncd.service-FqCM6e
drwx------ 3 root root 4096 Mar 2 15:48 systemd-private-e709306472c1435993a0b8d1f15e9dd3-systemd-logind.service-xFJpBh
drwx------ 3 root root 4096 Mar 2 15:48 systemd-private-e709306472c1435993a0b8d1f15e9dd3-ModemManager.service-x5UZXh
-rwxr-xr-x 1 root root 691 Mar 2 15:48 eofcat
-rwxr-xr-x 1 root root 285 Mar 2 15:48 autopkgtest-reboot
-rwxr-xr-x 1 root root 269 Mar 2 15:48 autopkgtest-reboot-prepare
drwxrwxrwt 5 root root 4096 Mar 2 15:48 autopkgtest.RixDKr
drwx------ 10 root root 4096 Mar 2 15:48 sosreport-autopkgtest-2022-03-02-zwngejm
drwx------ 3 root root 4096 Mar 2 15:48 systemd-private-e709306472c1435993a0b8d1f15e9dd3-fwupd.service-Zasqxf
drwx------ 3 root root 4096 Mar 2 15:48 systemd-private-e709306472c1435993a0b8d1f15e9dd3-upower.service-qb69Wg
-rw------- 1 root root 1645 Mar 2 15:49 sosreport-host0-2022-03-02-bwcteqj-private_map
-rw------- 1 root root 2389116 Mar 2 15:50 sosreport-host0-2022-03-02-bwcteqj-obfuscated.tar.xz
drwxr-xr-x 12 root root 4096 Mar 2 15:50 sosreport_test
drwx------ 12 root root 4096 Mar 2 15:55 sosreport-autopkgtest-2022-03-02-nwzytde
-rw------- 1 root root 2409380 Mar 2 15:55 sosreport-autopkgtest-2022-03-02-nwzytde.tar.xz
-rw-r--r-- 1 root root 65 Mar 2 15:55 sosreport-autopkgtest-2022-03-02-nwzytde.tar.xz.sha256
-rw------- 1 root root 2411848 Mar 2 15:58 sosreport-autopkgtest-2022-03-02-hkqkbak.tar.xz
-rw-r--r-- 1 root root 65 Mar 2 15:58 sosreport-autopkgtest-2022-03-02-hkqkbak.tar.xz.sha256
drwx------ 12 root root 4096 Mar 2 15:58 sosreport-host0-2022-03-02-abhwscl
-rw------- 1 root root 1645 Mar 2 15:59 sosreport-host0-2022-03-02-abhwscl-private_map <---
-rw------- 1 root root 2378324 Mar 2 15:59 sosreport-host0-2022-03-02-abhwscl-obfuscated.tar.xz
-rw------- 1 root root 65 Mar 2 15:59 sosreport-host0-2022-03-02-abhwscl-obfuscated.tar.xz.sha256

root@autopkgtest:/tmp# cat sosreport-host0-2022-03-02-abhwscl-private_map
{
    "hostname_map": {
        "autopkgtest": "host0"
    },
    "ip_map": {
        "10.0.2.0/24": "100.0.0.0/24",
        "10.0.2.15/24": "100.0.0.1/24",
        "10.0.2.255": "100.0.0.255",
        "10.0.2.3": "100.0.0.2/24",
        "91.189.89.198": "33.43.50.21",
        "5.4.0.102": "80.74.90.96",
        "5.4.0.100": "69.87.15.65",
        "5.4.0.26": "13.16.68.51",
        "224.0.0.1": "92.20.91.63",
        "91.189.94.4": "42.38.68.46",
        "3.192.30.10": "93.87.22.28",
        "5.4.0.99": "37.44.72.50",
        "10.0.2.0/28": "101.0.0.1/28",
        "10.0.2.0/30": "102.0.0.1/30",
        "192.168.200.1": "37.72.13.85",
        "192.168.200.4": "19.35.86.99",
        "192.168.200.9": "39.80.73.13",
        "192.168.201.0/24": "103.0.0.1/24",
        "192.168.201.0/25": "104.0.0.1/25",
        "224.0.0.251": "19.45.84.66",
        "239.255.255.250": "93.52.70.42",
        "123.45.67.89": "92.20.45.84",
        "192.168.0.133": "29.81.60.51"
    },
    "mac_map": {
        "52:54:00:12:34:56": "53:4f:53:45:22:61", <----- never made it into the file collected
        "33:33:00:00:00:16": "53:4f:53:63:ca:e1",
        "33:33:00:00:00:02": "53:4f:53:46:bc:12",
        "33:33:ff:12:34:56": "53:4f:53:9a:2e:fa",
        "33:33:00:00:00:01": "53:4f:53:e6:29:f1",
        "01:00:5e:00:00:01": "53:4f:53:67:6f:e7",
        "01:80:c2:00:00:00": "53:4f:53:20:c4:71",
        "01:80:c2:00:00:03": "53:4f:53:fb:50:f9",
        "01:80:c2:00:00:0e": "53:4f:53:2d:57:8a",
        "52:55:0a:00:02:03": "53:4f:53:50:2b:72",
        "52:55:0a:00:02:02": "53:4f:53:64:ec:34",
        "52:56:00:00:00:02": "53:4f:53:3e:92:21"
    },
    "keyword_map": {},
    "username_map": {}

-------------

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

[Arif Ali (arif-ali)]

This is resolved in 4.4, and was released in bionic and focal, so solved this issue

[Christian Ehrhardt  (paelzer)]

We were scanning old bugs with assigned people.
Mostly to better manage expectations on bugs that have not yet been solved, but this one is good already \o/
Either way, removing the assignee as there is not much more coming e.g. on the "Impish Won't Fix" task