Ubuntu
libmail-dmarc-perl package

[MIR] libmail-dmarc-perl

Bug #2023971 reported by Bryce Harrington on 2023-06-15

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	libmail-dmarc-perl (Ubuntu)	Fix Released	High	Unassigned

Bug Description

Package: libmail-dmarc-perl
-----
Dependency status update (Feb 01, 2024):

On Feb 1, the six needed packages to be promoted to main and their bug status are:

* libmail-dmarc-perl (this bug): bug 2023971, New, Ioanna assigned: pending review, package to be tested at [1] with all the changes required in the other MIR bugs, and the dependency splitting.

* libemail-simple-perl: bug 2031491, New, assigned security team : conditional ACK from security team by George-Andrei Iosif (iosifache)
           -> maintenance by Server Team if we lack upstream support
           -> ask for demotion (in the future) if another package if a more
              suitable can be used as an alternative

* libfile-sharedir-perl: bug 2039566 , In progress, unassigned -> ACK by slyon, all TODOS done.

* libclass-inspector-perl: bug 2039569 , Fix commited, unassigned -> ACK by Didier, All todos done

* libnet-ip-perl: bug 2039456 , In progress, unassigned -> ACK by Ioanna, All todos done

* libregexp-common-perl: bug 2039563 , Fix commited, unassigned-> ACK by Didier, All todos done

[1] https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl-suggested/+sourcepub/15684761/+listing-archive-extra

-----
Please promote libmail-dmarc-perl and its universe dependencies (recursively) to main. According to `check-mir` (+ recursive searching) this would need ~44 packages to be promoted (as runtime binary dependencies, see [Dependencies] section for detailed dependencies tree) .

However, libmail-dmarc-perl is only used by the spamassassin package, and spamassassin only uses the validation feature of DMARC, so we can narrow the necessary binaries dependencies to be promoted too. Those packages are the following ones (11), which I propose for promotion instead of the complete dependencies tree:

The rest would be moved to Suggested dependencies.

[Availability]
The package libmail-dmarc-perl is already in Ubuntu universe.
The package libmail-dmarc-perl builds for the architectures it is designed to work on.
It currently builds and works for architectures: amd64 (all)
Link to package https://launchpad.net/ubuntu/+source/libmail-dmarc-perl

[Rationale]

tldr; DMARC support in SpamAssassin is important for stronger spam filtering.

Spam email is an ever-present and ever-evolving presence in our online lives, and SpamAssassin is a key tool for end users and service providers to identify likely spam for filtering. SpamAssassin 4.0, introduced in Ubuntu "lunar" 22.10, introduced a number of major new features including three new plugins, the most significant of which is the DMARC policy checker.

DMARC (or "Domain-based Message Authentication, Reporting & Conformance") [0] is a new convention for email service providers to communicate to email recipient programs about how to handle authentication failures. It builds on prior protocols (namely, SPF and DKIM) to address their limitations. Essentially, DMARC protects against direct domain spoofing, such that when an email purports to be from a given domain (say, @gmail.com or @irs.gov) but fails proper authentication using the authentication methods published by that domain, it tells the email receiver whether to reject the email as spam, quarantine it for evaluation, or something else. DMARC also establishes a way for the email receiver to give feedback back to the sender about emails that failed to pass this check.

libmail-dmarc-perl contains the official Perl implementation of DMARC support. SpamAssassin is the primary user of this package

❯ apt-cache rdepends libmail-dmarc-perl
libmail-dmarc-perl
Reverse Depends:
spamassassin

and requires it in order to perform this DMARC evaluation: Spamassassin dmarc plugin [1] only uses the MAIL::DMARC::PurePerl module [2] and, inside it, the validate function in particular [3]. It means that Spamassasin only uses by default the DMARC's validation feature, and optionally the DMARC's Reporting feature.

libmail-dmarc-perl is a relatively new source package that was proposed for lunar [4] but deleted on 2023-05-01 due to build/autopkgtest issues and later re-introduced to mantic once those issues were resolved, on 2023-06-13.

[0] (https://dmarc.org/)
[1] (https://git.launchpad.net/ubuntu/+source/spamassassin/tree/lib/Mail/SpamAssassin/Plugin/DMARC.pm)
[2] (https://git.launchpad.net/ubuntu/+source/spamassassin/tree/lib/Mail/SpamAssassin/Plugin/DMARC.pm#n242)
[3] (https://git.launchpad.net/ubuntu/+source/spamassassin/tree/lib/Mail/SpamAssassin/Plugin/DMARC.pm#n322)
[4] (https://bugs.launchpad.net/ubuntu/+source/libmail-dmarc-perl/+bug/2023606)

[Security]
No CVEs/security issues in this software in the past:
  - (0) https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=libmail-dmarc-perl
  - (0) https://ubuntu.com/security/cves?q=&package=libmail-dmarc-perl
  - (0) https://security-tracker.debian.org/tracker/source-package/libmail-dmarc-perl
Broadening the search to "spamassassin dmarc" shows only a single issue within the past few years, CVE-2022-3620, which appears to be more of an issue in exim4 specifically, and has been deemed not relevant to the exim4 package shipped in Ubuntu (See https://ubuntu.com/security/CVE-2022-3620).

No `suid` or `sgid` binaries.
No executables in `/sbin` and `/usr/sbin`.
Package does not install services, timers or recurring jobs, but contains an example of a cron job for updating the public_suffix_list (https://publicsuffix.org/).
Package does not open privileged ports (ports < 1024).
Package does not expose any external endpoints.
Package contains extensions to security-sensitive software (filters, scanners, plugins, UI skins, ...): It is the basis for the DMARC plugin of spamassassin, implementing the DMARC policy validation (and reporting).

Upstream's source code includes the build of a web server (dmarc_httpd) and an HTTP client (dmarc_http_client), but these binaries are not installed in the debian package (they are suppressed on build time for autoinstallation).

[Quality assurance - function/usage]
The package works well right after installed with its required installation dependencies.

Integration with spamassassin has been tested by co-installing it, updating the configuration, running sa-compile, and checking for errors. Behavioral performance was checked by running:

$ spamassassin --debug < /home/bryce/pkg/Spamassassin/spam.mbox

Processing of emails by spamassassin using its DMARC plugin that uses libmail-dmarc-perl validation function was tested also grabbing an email (i.e, from gmail) and doing:

$ spamc -R < message-mattermost-hey.eml | grep DMARC
-0.0 DMARC_PASS DMARC pass policy

also, non pass results with an already classified as spam mail (by gmail, cleaning the header):
0.9 DMARC_NONE DMARC none policy

or introducing typos to malform domains/header:
0.0 DMARC_MISSING Missing DMARC policy

For rejection examples, we use the data from spamassassin's test, and we can see:

root@Mspamassasin-suggested:~# l spam_mails/
nodmarc.eml noneko.eml quarko.eml rejectko.eml strictrejectko.eml
root@Mspamassasin-suggested:~# spamc -R < spam_mails/rejectko.eml | grep DMARC
1.8 DMARC_REJECT DMARC reject policy
root@Mspamassasin-suggested:~# spamc -R < spam_mails/strictrejectko.eml | grep DMARC
1.8 DMARC_REJECT DMARC reject policy
root@Mspamassasin-suggested:~# spamc -R < spam_mails/quarko.eml | grep DMARC
1.2 DMARC_QUAR DMARC quarantine policy
root@Mspamassasin-suggested:~# spamc -R < spam_mails/nodmarc.eml | grep DMARC
0.0 DMARC_MISSING Missing DMARC policy
root@Mspamassasin-suggested:~# spamc -R < spam_mails/noneko.eml | grep DMARC
0.9 DMARC_NONE DMARC none policy

I'm working on an autopkgtest that covers this: The code is uploaded but not working yet. For manual testing, you can get the eml messages from this commit:

https://git.launchpad.net/~mirespace/ubuntu/+source/libmail-dmarc-perl/commit/?id=feb85cd6f97508c98bdeb732fe25f876f33790d4

where d/t/data/nice contains messages that qualifies as PASS due to the DMARC rules (DMARC_PASS) and d/t/data/spam contains messages that are market as REJECT, QUARENTINE, MISSING or NONE.

The specific test on spamassassin for DMARC (t/dmarc.t) was also run successfully (without the implicit net tests on this file):

  $ make test TEST_FILES="t/dmarc.t"
  "/usr/bin/perl" build/mkrules --exit_on_no_src --src rulesrc --out rules --manifest MANIFEST --manifestskip MANIFEST.SKIP
  mkrules: no rules updated
  "/usr/bin/perl" build/preprocessor -Mvars -DVERSION="4.000000" -DPREFIX="/usr/local" -DDEF_RULES_DIR="/usr/local/share/spamassassin" -DLOCAL_RULES_DIR="/etc/mail/spamassassin" -DLOCAL_STATE_DIR="/var/lib/spamassassin" -DINSTALLSITELIB="/usr/local/share/perl/5.36.0" -DCONTACT_ADDRESS="the administrator of that system" -DRE2C_BIN="re2c" -Msharpbang -Mconditional -DPERL_BIN=""/usr/bin/perl"" -DPERL_WARN="" -DPERL_TAINT="" -m755 -isa-update.raw -osa-update
  cp sa-update blib/script/sa-update
  "/usr/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/sa-update
  PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/dmarc.t
  t/dmarc.t .. Nov 3 12:36:16.977 [8195] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. 1/18 Nov 3 12:36:18.906 [8197] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. 3/18 Nov 3 12:36:21.247 [8199] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. 5/18 Nov 3 12:36:23.737 [8201] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. 7/18 Nov 3 12:36:26.153 [8203] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. 9/18 Nov 3 12:36:29.117 [8205] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. 11/18 Nov 3 12:36:32.796 [8207] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. 13/18 Nov 3 12:36:35.338 [8209] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. 15/18 Nov 3 12:36:37.374 [8211] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
  t/dmarc.t .. ok
  All tests successful.
  Files=1, Tests=18, 31 wallclock secs ( 0.02 usr 0.00 sys + 12.18 cusr 0.88 csys = 13.08 CPU)
  Result: PASS

[Quality assurance - maintenance]
The package is maintained well in Debian/Ubuntu and does
not have too many, long-term & critical, open bugs:
   - Ubuntu (0) https://bugs.launchpad.net/ubuntu/+source/libmail-dmarc-perl/+bug
   - Debian (0) https://bugs.debian.org/cgi-bin/pkgreport.cgi?src=libmail-dmarc-perl
   - Upstream's bug tracker (4) https://github.com/msimerson/mail-dmarc/issues
     + Upstream's repo last activity: https://github.com/msimerson/mail-dmarc/issues
       - last commit: in master, Oct 25, 2023
       - Issues without answer: 2
       - Updated issue/PR: Oct 25, 2023
       - last fixed/closed/merged issue: Oct 25, 2023
       - last merged PR: Oct 25, 2023
The package has one important/old open bugs on upstream: Invalid XML in generated reports, https://github.com/msimerson/mail-dmarc/issues/190
The package does not deal with exotic hardware we cannot support

[Quality assurance - testing]
The package runs a test suite on build time, if it fails
it makes the build fail: https://launchpadlibrarian.net/677780063/buildlog_ubuntu-mantic-amd64.libmail-dmarc-perl_1.20230215-1_BUILDING.txt.gz :

Creating new 'Build' script for 'Mail-DMARC' version '1.20230215'
dh_auto_build
/usr/bin/perl Build
Building Mail-DMARC
dh_auto_test
/usr/bin/perl Build test --verbose 1

The package doesn't run an autopkgtest.

[Quality assurance - packaging]
debian/watch is present and works.

debian/control defines a correct Maintainer field : Noah Meyerhans <email address hidden> ( https://git.launchpad.net/ubuntu/+source/libmail-dmarc-perl/tree/debian/control#n2)

This package does not yield massive lintian Warnings, Errors
  - recent build log of the package https://launchpadlibrarian.net/677780063/buildlog_ubuntu-mantic-amd64.libmail-dmarc-perl_1.20230215-1_BUILDING.txt.gz
  - full output from `lintian --pedantic` :
    #source
    ❯ lintian -EvIL +pedantic --show-overrides
      E: libmail-dmarc-perl changes: bad-distribution-in-changes-file noble
      W: libmail-dmarc-perl: changelog-distribution-does-not-match-changes-file unstable != noble [usr/share/doc/libmail-dmarc-perl/changelog.Debian.gz:1]
      W: libmail-dmarc-perl changes: distribution-and-changes-mismatch noble unstable
      I: libmail-dmarc-perl: typo-in-manual-page messsage message [usr/share/man/man3/Mail::DMARC::Report::Receive.3pm.gz:162]
      I: libmail-dmarc-perl: typo-in-manual-page messsage message [usr/share/man/man3/Mail::DMARC::Result.3pm.gz:184]
      P: libmail-dmarc-perl: repeated-path-segment share [usr/share/perl5/auto/share/]
      X: libmail-dmarc-perl: application-in-library-section perl [usr/bin/dmarc_lookup]
      X: libmail-dmarc-perl: application-in-library-section perl [usr/bin/dmarc_receive]
      X: libmail-dmarc-perl: application-in-library-section perl [usr/bin/dmarc_send_reports]
      X: libmail-dmarc-perl: application-in-library-section perl [usr/bin/dmarc_update_public_suffix_list]
      X: libmail-dmarc-perl: application-in-library-section perl [usr/bin/dmarc_view_reports]
      X: libmail-dmarc-perl: library-package-name-for-application [usr/bin/dmarc_lookup]
      X: libmail-dmarc-perl: library-package-name-for-application [usr/bin/dmarc_receive]
      X: libmail-dmarc-perl: library-package-name-for-application [usr/bin/dmarc_send_reports]
      X: libmail-dmarc-perl: library-package-name-for-application [usr/bin/dmarc_update_public_suffix_list]
      X: libmail-dmarc-perl: library-package-name-for-application [usr/bin/dmarc_view_reports]
    #binary
    ❯ lintian -EvIL +pedantic --show-overrides ../libmail-dmarc-perl_1.20230215-1.dsc
      I: libmail-dmarc-perl source: out-of-date-standards-version 4.6.0 (released 2021-08-18) (current is 4.6.2)
      X: libmail-dmarc-perl source: debian-watch-does-not-check-openpgp-signature [debian/watch]
      X: libmail-dmarc-perl source: libmodule-build-perl-needs-to-be-in-build-depends
      X: libmail-dmarc-perl source: update-debian-copyright 2022 vs 2023 [debian/copyright:11]

This package does not rely on obsolete or about to be demoted packages.
This package has no python2 or GTK2 dependencies.

The package will not be installed by default.

Packaging and build is easy, link to debian/rules: https://git.launchpad.net/ubuntu/+source/libmail-dmarc-perl/tree/debian/rules

[UI standards]
Application is not end-user facing (does not need translation).

[Dependencies]

As exposed at the beginning, this package needed initially +-44 MIR bugs (depending on the processing of the Recommends ones).
This is the original dependency tree, only listing on first occurrence:

* libdbd-sqlite3-perl: universe
    MIR bug: https://bugs.launchpad.net/ubuntu/+source/libdbd-sqlite3-perl/+bug/2029379
* libdbix-simple-perl: universe
   MIR bug: https://bugs.launchpad.net/ubuntu/+source/libdbix-simple-perl/+bug/2030731
* libemail-mime-perl: universe
   MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-mime-perl/+bug/2030880
   - libemail-messageid-perl: universe
     MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-messageid-perl/+bug/2030956
   - libemail-mime-contenttype-perl: universe
     MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-mime-contenttype-perl/+bug/2030962
     + libtext-unidecode-perl: universe
        MIR bug: https://bugs.launchpad.net/ubuntu/+source/libtext-unidecode-perl/+bug/2031109
   - libemail-mime-encodings-perl: universe
     MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-mime-encodings-perl/+bug/2031487
   - libemail-simple-perl: universe
     MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-simple-perl/+bug/2031491
* libemail-sender-perl: universe
     MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-sender-perl/+bug/2037389
   - libemail-abstract-perl: universe
     MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-abstract-perl/+bug/2037405
     + libmodule-pluggable-perl: universe
     + libmro-compat-perl: universe
       • libclass-c3-perl: universe
         ◦ libalgorithm-c3-perl: universe
       • libclass-c3-xs-perl: universe (Recommends)
   - libmoox-types-mooselike-perl: universe
   - libscalar-list-utils-perl: universe
   - libthrowable-perl: universe
     MIR bug https://bugs.launchpad.net/ubuntu/+source/libthrowable-perl/+bug/2037392
     + libmoose-perl: universe (Recommends)
       • libclass-load-perl: universe
       • libclass-load-xs-perl: universe
       • libdevel-globaldestruction-perl: universe
       • libdevel-overloadinfo-perl: universe
       • libeval-closure-perl: universe
         ◦ libdevel-lexalias-perl: universe (Recommends)
           · libdevel-caller-perl: universe
              ·· libpadwalker-perl: universe
       • libmodule-runtime-conflicts-perl: universe
         ◦ libdist-checkconflicts-perl: universe
       • libpackage-deprecationmanager-perl: universe
         ◦ libscalar-list-utils-perl: universe
       • libdevel-partialdump-perl: universe (Recommends)
         ◦ libclass-tiny-perl: universe
* libfile-sharedir-perl: universe
   MIR bug https://bugs.launchpad.net/ubuntu/+source/libfile-sharedir-perl/+bug/2039566
   + libclass-inspector-perl: universe
     MIR bug https://bugs.launchpad.net/ubuntu/+source/libclass-inspector-perl/+bug/2039569
* libnet-idn-encode-perl: universe
   MIR bug https://bugs.launchpad.net/ubuntu/+source/libnet-idn-encode-perl/+bug/2038929
* libnet-imap-simple-perl: universe (Recommends)
   - libparse-recdescent-perl: universe (Recommends)
* libnet-ip-perl: universe
   https://bugs.launchpad.net/ubuntu/+source/libnet-ip-perl/+bug/2039456
* libnet-smtps-perl: universe
* libregexp-common-perl: universe
   MIR bug https://bugs.launchpad.net/ubuntu/+source/libregexp-common-perl/+bug/2039563
* libtest-file-sharedir-perl: universe
   - libfile-copy-recursive-perl: universe
   - libfile-sharedir-perl: universe
     + libclass-inspector-perl: universe
   - libscope-guard-perl: universe
* libtest-output-perl: universe

But, attending on how spamassassin works -explained above-, I believe we can split these dependencies in dependencies used by the reporting feature and dependencies used by validation feature. Walking this way,
the dependencies used in validation remains as needed runtime binary dependencies, and the dependencies used in reporting goes to suggested runtime dependencies. Therefore, only 11 packages would need to go
through the MIR process to satisfy libmail-dmarc-perl dependencies:

  * libemail-mime-perl: universe
    MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-mime-perl/+bug/2030880
    - libemail-messageid-perl: universe
      MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-messageid-perl/+bug/2030956
    - libemail-mime-contenttype-perl: universe
      MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-mime-contenttype-perl/+bug/2030962
      + libtext-unidecode-perl: universe
         MIR bug: https://bugs.launchpad.net/ubuntu/+source/libtext-unidecode-perl/+bug/2031109
    - libemail-mime-encodings-perl: universe
      MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-mime-encodings-perl/+bug/2031487
    - libemail-simple-perl: universe
      MIR bug: https://bugs.launchpad.net/ubuntu/+source/libemail-simple-perl/+bug/2031491
  * libfile-sharedir-perl: universe
    MIR bug https://bugs.launchpad.net/ubuntu/+source/libfile-sharedir-perl/+bug/2039566
    + libclass-inspector-perl: universe
      MIR bug https://bugs.launchpad.net/ubuntu/+source/libclass-inspector-perl/+bug/2039569
  * libnet-idn-encode-perl: universe
    MIR bug https://bugs.launchpad.net/ubuntu/+source/libnet-idn-encode-perl/+bug/2038929
  * libnet-ip-perl: universe
    https://bugs.launchpad.net/ubuntu/+source/libnet-ip-perl/+bug/2039456
  * libregexp-common-perl: universe
    MIR bug https://bugs.launchpad.net/ubuntu/+source/libregexp-common-perl/+bug/2039563

a package following this scheme was built and it is available for testing on mantic here:

ppa:mirespace/libmail-dmarc-perl-suggested
https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl-suggested

and the debdiff (for noble now) is here: https://pastebin.ubuntu.com/p/2kW5HwSsCq/ .
Note that, in this package, there is a suppression of a perl library (on main) that I suspect is not needed, as the package builds well and tests passed also. I opened this question to upstream (https://github.com/msimerson/mail-dmarc/issues/215).

Also, there is this MP (https://code.launchpad.net/~mirespace/ubuntu/+source/libmail-dmarc-perl/+git/libmail-dmarc-perl/+merge/456713) opened for tracking changes to the package. For the moment, apart from the split, we have:
- refactor for not using libemail-mime-perl (done)
- adding autopkgtest for validating the splitting (wip)

This package passes all the tests explained in the [Quality assurance - function/usage] section.

[Standards compliance]
This package correctly follows FHS and Debian Policy

[Maintenance/Owner]
Owning Team will be the Ubuntu Server Team, if they're not covered already by other teams (the Foundations team has covered maintenance at points in the past).
Team is not yet, but will subscribe to the package before promotion.
This does not use static builds.
This does not use vendored code.
This package is not rust based.

The package successfully built during the most recent test rebuild : https://launchpad.net/ubuntu/+archive/test-rebuild-20230830-mantic/+build/26600016/+files/buildlog_ubuntu-mantic-amd64.libmail-dmarc-perl_1.20230215-1_BUILDING.txt.gz

[Background information]
The Package description explains the package well.
Upstream Name is Mail-DMARC .
Link to upstream project https://metacpan.org/release/Mail-DMARC

About SSLeay (https://github.com/msimerson/mail-dmarc/issues/215) and for the moment I keep the library on the dependencies,
but libhttp-tiny-perl is not in use by the debian package because that is provided by perl package itself, so I'm wondering about it yet.

This has been in the archive since this year (Lunar, 1.20211209-1).

See original description

Tags:

Related branches

~mirespace/+git/team-subscriptions:mir_libmail-dmarc-perl_lp2023971

Merged into ~canonical-server/+git/team-subscriptions:main at revision b27924f729397206420ddb3877e7a3eb96498bbb

Lucas Kanashiro: Approve on 2023-11-27

Canonical Server Reporter: Pending requested 2023-11-24

Bryce Harrington (bryce) on 2023-06-30

description:	updated
Changed in libmail-dmarc-perl (Ubuntu):
status:	Incomplete → New

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2023-07-04 (last edit on 2023-07-04):

Actually, this shouldn't be tasks nowadays. Individual bugs would be better.
Since they are so similar all the meta-discussion can happen here.
And a link farm to reach the others is useful, see how Lucas has done pcs.

So the others can - for now - be shallow forks for tooling to work well.

Miriam España Acebal (mirespace) on 2023-07-06

Changed in libmail-dmarc-perl (Ubuntu):
assignee:	nobody → Miriam España Acebal (mirespace)

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2023-07-06:

Incomplete as Miriam will prep a bit more for all the dependencies

Changed in libmail-dmarc-perl (Ubuntu):
status:	New → Incomplete

Miriam España Acebal (mirespace) on 2023-08-02

description:

updated

Miriam España Acebal (mirespace) on 2023-08-08

description:

updated

Miriam España Acebal (mirespace) on 2023-08-09

description:

updated

Miriam España Acebal (mirespace) on 2023-08-10

description:

updated

Miriam España Acebal (mirespace) on 2023-08-10

description:

updated

Miriam España Acebal (mirespace) on 2023-08-10

description:

updated

Miriam España Acebal (mirespace) on 2023-08-11

description:

updated

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2023-08-16:

Miriam will continue spreading out the prep work by Bryce to create more individual bugs to process.
But for now the initial 6 have been opened and are ready for review (to not stall getting this started further).

We can see on those if our hope for "most should be trivial and quick perl cases" is true.

Miriam España Acebal (mirespace) on 2023-08-16

description:

updated

Miriam España Acebal (mirespace) on 2023-08-16

description:

updated

Miriam España Acebal (mirespace) on 2023-09-01

description:

updated

Miriam España Acebal (mirespace) on 2023-09-26

description:

updated

Miriam España Acebal (mirespace) on 2023-09-26

description:

updated

Miriam España Acebal (mirespace) on 2023-09-26

description:

updated

Christian Ehrhardt  (paelzer) on 2023-09-28

description:

updated

Miriam España Acebal (mirespace) on 2023-09-28

description:

updated

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2023-09-28:

AFAICS libtest-file-sharedir-perl is only used in tests:

root@m:~/libmail-dmarc-perl-1.20230215# grep -Hrn use * | grep Test::File::ShareDir
t/09.HTTP.t:7:use Test::File::ShareDir
t/03.Base.t:7:use Test::File::ShareDir
t/00.Dmarc.t:7:use Test::File::ShareDir
t/06.Result.t:7:use Test::File::ShareDir
t/26.Report.Sender.t:10:use Test::File::ShareDir
t/21.Report.Send.t:7:use Test::File::ShareDir
t/04.PurePerl.t:8:use Test::File::ShareDir
t/17.Report.Aggregate.Schema.t:7:use Test::File::ShareDir -share => { -dist => { 'Mail-DMARC' => 'share' } };
t/11.Report.Store.t:7:use Test::File::ShareDir

It is only there as it is listed manually in d/control.

That would eliminate this whole block:
* libtest-file-sharedir-perl: universe
   - libclass-tiny-perl: universe
   - libfile-copy-recursive-perl: universe
   - libfile-sharedir-perl: universe
     + libclass-inspector-perl: universe
   - libscope-guard-perl: universe

Looking at d/control:

Package: libmail-dmarc-perl
Architecture: all
Depends: ${misc:Depends},
         ${perl:Depends},
         libconfig-tiny-perl,
         libio-socket-ssl-perl,
         libfile-sharedir-perl,
         libnet-dns-perl,
         libnet-idn-encode-perl,
         libnet-ip-perl,
         libnet-ssleay-perl,
         libemail-mime-perl,
         libtest-file-sharedir-perl,
         libemail-sender-perl,
         libdbix-simple-perl,
         libdbd-sqlite3-perl,
         libtest-output-perl,
         libregexp-common-perl,
         libsocket6-perl,
         liburi-perl,
         libxml-libxml-perl,
         publicsuffix,

That is a long list of manual depends which might be wrong.

I think there are two tasks here, now seeming like work but potentially eliminating a lot more work.
0. check code for candidates e.g.
   # Install libs listed
   $ apt install lib...
   $ dpkg -L lib...
   # find its include path like
   # /usr/share/perl5/Config/Tiny.pm
   # Check for usage of that

Example of a potentially real dependency:
# grep -Hrn use * | grep Config::Tiny
lib/Mail/DMARC/Base.pm:8:use Config::Tiny;

1. Remove all the manual dependencies, see what just ${perl:Depends} detects
2. Use libmail-dmarc-perl without them installed, does it still work

Once confirmed, update the dependency tree in the bug and in the package.

As you @Miriam have found yourself furthermore there is this in Install.md

NOTE: Most of the dependencies are optionally required for the DMARC reporting features. Mail::DMARC will perform validation with only these modules:

    Regexp::Common
    Config::Tiny
    File::ShareDir
    Net::DNS::Resolver
    Net::IP
    Socket6

So an approach could be a much more limited libmail-dmarc-perl in main and a separated only suggested package libmail-dmarc-perl-reporting which adds the rest. Doing this does not prevent considering to add the reporting as well in the future and is independent to the check which dependencies might be entirely non-needed.

Of these only the two here might be needed then:
* libfile-sharedir-perl: universe
+ libclass-inspector-perl: universe

AFAICS libtest-file-sharedir-perl is only used in tests:

It is only there as it is listed manually in d/control.

That would eliminate this whole block:
 * libtest-file-sharedir-perl: universe
   - libclass-tiny-perl: universe
   - libfile-copy-recursive-perl: universe
   - libfile-sharedir-perl: universe
     + libclass-inspector-perl: universe
   - libscope-guard-perl: universe

Looking at d/control:

That is a long list of manual depends which might be wrong.

Example of a potentially real dependency:
# grep -Hrn use * | grep Config::Tiny
lib/Mail/DMARC/Base.pm:8:use Config::Tiny;

1. Remove all the manual dependencies, see what just ${perl:Depends} detects
2. Use libmail-dmarc-perl without them installed, does it still work

Once confirmed, update the dependency tree in the bug and in the package.

As you @Miriam have found yourself furthermore there is this in Install.md

NOTE: Most of the dependencies are optionally required for the DMARC reporting features. Mail::DMARC will perform validation with only these modules:

Regexp::Common
    Config::Tiny
    File::ShareDir
    Net::DNS::Resolver
    Net::IP
    Socket6

Of these only the two here might be needed then:
 * libfile-sharedir-perl: universe
   + libclass-inspector-perl: universe

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-10-05:

Download full text (4.0 KiB)

TL;DR We could split the dmarc package into validation (in main) and reporting (suggested to validation, in universe). Still working in checking it through spamassassin. Packages to be MIR:

     libfile-sharedir-perl
       + libclass-inspector-perl
     libnet-idn-encode-perl
     libnet-ip-perl
     libregexp-common-perl

I went throught the list of dependencies declared in the control file as exposed in the comment above, plus doing a matching of what perl modules were used in the test build suite provided in the libmail-dmarc-perl package, in the following sense:

- the module MAIL::DMARC::* does a "use" or "require" of perl modules provided by these X perl packages.
- The t/*.test test in libmail-dmarc-perl's source tree that test the above module use or require these X perl packages.

After that, I splitted the libmail-dmarc-perl package into validation functionality and reporting feature according to the above [1], wich coincides with the ones that are mentioned in the INSTALL.md for validation:

     libconfig-tiny-perl,
     libfile-sharedir-perl,
  libnet-dns-perl,
     libnet-idn-encode-perl,
     libnet-ip-perl,
     libregexp-common-perl,
     libsocket6-perl,
  liburi-perl,

I used the above tests as a usability check of the split packages, doing a prove -v t/*.test with the packages installed (not at build time). First, only with the validation package, later installing also the reporting package for the t/*.Report.*t tests.

Furthermore, the doubt about all of this could be ... is enough the validation functionality of dmarc for the spamassassin plugin? Yes, with an exception that it's already optional: saving reports, that due to the comment, seems that the Store module of DMARC wasn't a submodule of DMARC::Report at the beginning of this perl implementation of DMARC.

Spamassasin dmarc plugin [2] only uses the MAIL::DMARC::PurePerl module [3] and, inside it, the validate function in particular [4]. However, is true that it also could use the save_aggregate function if the dmarc_save_reports variable is set throught mail-dmarc.ini to 1 (defaults is 0 [5], and also in the ini file [6]). That function will need the Mail::DMARC::Report::Store module and the Mail::DMARC::Report::URI module.

It would need a third verification for the splitting: to test it from spamassasin. It has also a t/dmarc.t test [7], that is skipped in building time nowadays because tests involving net are disabled (logical in our infra). I'm working on how to run this test properly or how it can be re-used for this needed check.

So, the packages to be MIR processed in the case of splitting (validation package in main, reporting package in universe and suggested to validation package) are:

libfile-sharedir-perl
+ libclass-in...

TL;DR We could split the dmarc package into validation (in main) and reporting (suggested to validation, in universe). Still working in checking it through spamassassin. Packages to be MIR:

libfile-sharedir-perl
       + libclass-inspector-perl
     libnet-idn-encode-perl
     libnet-ip-perl
     libregexp-common-perl

- Creating two d/*.install files for deploying MAIL::DMARC::* modules according with validation/reporting: all the DMARC::Report::* modules go to the reporting package, and associated  script located in the source tree on /bin folder.
- Splitting the binary dependencies between validation functionality and reporting in the d/control file, as ${perl:Depends} is not calculating anything  on its end (maybe due to [8])
  
     libconfig-tiny-perl,
     libfile-sharedir-perl,
	 libnet-dns-perl,
     libnet-idn-encode-perl,
     libnet-ip-perl,
     libregexp-common-perl,
     libsocket6-perl,
	 liburi-perl,

Spamassasin dmarc plugin [2] only uses the MAIL::DMARC::PurePerl module [3] and, inside it, the validate function in particular [4]. However, is true that it also could use the save_aggregate function if the dmarc_save_reports variable is set throught mail-dmarc.ini to 1 (defaults is 0 [5],  and also in the ini file [6]). That function will need the Mail::DMARC::Report::Store module and the Mail::DMARC::Report::URI module.

So, the packages to be MIR processed in the case of splitting (validation package in main, reporting package in universe and suggested to validation package) are:

libfile-sharedir-perl
       + libclass-inspector-perl
     libnet-idn-encode-perl
     libnet-ip-perl
     libregexp-common-perl

[1] (https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl)
[2] (https://git.launchpad.net/ubuntu/+source/spamassassin/tree/lib/Mail/SpamAssassin/Plugin/DMARC.pm)
[3] (https://git.launchpad.net/ubuntu/+source/spamassassin/tree/lib/Mail/SpamAssassin/Plugin/DMARC.pm#n242)
[4] (https://git.launchpad.net/ubuntu/+source/spamassassin/tree/lib/Mail/SpamAssassin/Plugin/DMARC.pm#n322)
[5] (https://git.launchpad.net/ubuntu/+source/spamassassin/tree/lib/Mail/SpamAssassin/Plugin/DMARC.pm#n111)
[6] (https://git.launchpad.net/ubuntu/+source/libmail-dmarc-perl/tree/share/mail-dmarc.ini#n21)
[7] (https://git.launchpad.net/ubuntu/+source/spamassassin/tree/t/dmarc.t)
[8] (https://www.debian.org/doc/packaging-manuals/perl-policy/ch-module_packages.html#id-1.6.5.4.2.4)

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-10-05:

libmail-dmarc-perl-split.debdiff Edit (9.2 KiB, text/plain)

The first unpolished approach of splitting the package (used mainly to test the needed-to-be dependencies).
Debdiff between archive version and ppa package

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-10-06:

Bad news... for making spamassassin's dmarc.t test working, we need the following files to allow the requires of the dmarc PurePerl module (perl -e "requires MAIL::DMARC::PurePerl;"):

lib/Mail/DMARC/Report/URI.pm
lib/Mail/DMARC/Report.pm
lib/Mail/DMARC/Report/Aggregate.pm
lib/Mail/DMARC/Report/Aggregate/Metadata.pm
lib/Mail/DMARC/Report/Send.pm
lib/Mail/DMARC/Report/Send/SMTP.pm
lib/Mail/DMARC/Report/Send/HTTP.pm
lib/Mail/DMARC/Report/Store.pm
lib/Mail/DMARC/Report/Receive.pm
lib/Mail/DMARC/Report/Aggregate/Record.pm
lib/Mail/DMARC/Report/Aggregate/Record/Identifiers.pm
lib/Mail/DMARC/Report/Aggregate/Record/Auth_Results/
lib/Mail/DMARC/Report/Aggregate/Record/Auth_Results.pm
lib/Mail/DMARC/Report/Aggregate/Record/Row.pm
lib/Mail/DMARC/Report/Aggregate/Record/Row/Policy_Evaluated.pm

plus

apt install libxml-libxml-perl
apt install libemail-mime-perl

which will introduce this needed MIR bug tree, apart from reconsidering this initial splitting by pure Report modules:

* libemail-mime-perl
   - libemail-messageid-perl
   - libemail-mime-contenttype-perl
     + libtext-unidecode-perl
   - libemail-mime-encodings-perl
   - libemail-simple-perl

Still is 11 versus 44 to be MIR processed.

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-10-06:

The above is still for the use of the save_aggregate function on MAIL:DMARC that is called on MAIL::DMARC:PurePerl at line 48, inside the validate function called from Mail::SpamAssassin::Plugin::DMARC
at line 322.

Thinking about what from the dmarc tests on spamassasin is totally necessary or not if we could consider the autosave report disabled by default (that's it).

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-10-10:

#10

Another possibility is not to split the package but to move all the dependencies related to Reporting features from binary dependencies to Suggested. With this, we could avoid touching dmarc or spamassassin code (importing partially subs with autoload) and supporting only validation feature, in theory.

Focussing on:

- the modules that need to be imported in all the cases (still wip on filling this bugs).
- Check the not-split-but-reporting-dependencies-suggested way.

Next step: checking with upstream this separation on reporting/validation

Miriam España Acebal (mirespace) on 2023-10-16

description:

updated

Miriam España Acebal (mirespace) on 2023-10-16

description:	updated
description:	updated

Miriam España Acebal (mirespace) on 2023-10-17

description:

updated

Miriam España Acebal (mirespace) on 2023-10-17

description:

updated

Miriam España Acebal (mirespace) on 2023-10-17

description:

updated

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-10-17:

#11

Dear MIR approval team:

The bugs for the essential 11 perl modules to be promoted are filed. At the time I'm writing this, pending your review are the following:

https://bugs.launchpad.net/ubuntu/+source/libfile-sharedir-perl/+bug/2039566
https://bugs.launchpad.net/ubuntu/+source/libclass-inspector-perl/+bug/2039569
https://bugs.launchpad.net/ubuntu/+source/libregexp-common-perl/+bug/2039563
https://bugs.launchpad.net/ubuntu/+source/libnet-idn-encode-perl/+bug/2038929
https://bugs.launchpad.net/ubuntu/+source/libnet-ip-perl/+bug/2039456

Three more bugs were filed in advance to the above (for libemail-sender-perl, libemail-abstract-perl and libthrowable-perl). Please don't prioritize these three over the others above.

Checking in the meantime the dependency separation on validation/reporting without splitting the code in two packages.

Thanks in advance and for your patient!

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-11-03:

#12

The t/dmarc.t on spamassassin passes using the libmail-dmarc-perl package that puts all the non-necessary modules for validation as suggested packages (it's on ppa:mirespace/libmail-dmarc-perl-suggested).

Following spamassassin/t/README, and commenting the line 17 on t/dmarc.t for net tests disabled (while looking for a more proper way):

root@Mspamassasin-suggested:~/spamassassin# make test TEST_FILES="t/dmarc.t"
"/usr/bin/perl" build/mkrules --exit_on_no_src --src rulesrc --out rules --manifest MANIFEST --manifestskip MANIFEST.SKIP
mkrules: no rules updated
"/usr/bin/perl" build/preprocessor -Mvars -DVERSION="4.000000" -DPREFIX="/usr/local" -DDEF_RULES_DIR="/usr/local/share/spamassassin" -DLOCAL_RULES_DIR="/etc/mail/spamassassin" -DLOCAL_STATE_DIR="/var/lib/spamassassin" -DINSTALLSITELIB="/usr/local/share/perl/5.36.0" -DCONTACT_ADDRESS="the administrator of that system" -DRE2C_BIN="re2c" -Msharpbang -Mconditional -DPERL_BIN=""/usr/bin/perl"" -DPERL_WARN="" -DPERL_TAINT="" -m755 -isa-update.raw -osa-update
cp sa-update blib/script/sa-update
"/usr/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/sa-update
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/dmarc.t
t/dmarc.t .. Nov 3 12:36:16.977 [8195] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 1/18 Nov 3 12:36:18.906 [8197] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 3/18 Nov 3 12:36:21.247 [8199] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 5/18 Nov 3 12:36:23.737 [8201] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 7/18 Nov 3 12:36:26.153 [8203] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 9/18 Nov 3 12:36:29.117 [8205] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 11/18 Nov 3 12:36:32.796 [8207] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 13/18 Nov 3 12:36:35.338 [8209] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 15/18 Nov 3 12:36:37.374 [8211] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. ok
All tests successful.
Files=1, Tests=18, 31 wallclock secs ( 0.02 usr 0.00 sys + 12.18 cusr 0.88 csys = 13.08 CPU)
Result: PASS

So I believe we can continue only with the 11 perl modules that are implied on Validation, and moving the rest to suggested.

Following spamassassin/t/README, and commenting the line 17 on t/dmarc.t for net tests disabled (while looking for a more proper way):

root@Mspamassasin-suggested:~/spamassassin# make test TEST_FILES="t/dmarc.t"
"/usr/bin/perl" build/mkrules --exit_on_no_src --src rulesrc --out rules --manifest MANIFEST --manifestskip MANIFEST.SKIP
mkrules: no rules updated
"/usr/bin/perl" build/preprocessor  -Mvars -DVERSION="4.000000" -DPREFIX="/usr/local" -DDEF_RULES_DIR="/usr/local/share/spamassassin" -DLOCAL_RULES_DIR="/etc/mail/spamassassin" -DLOCAL_STATE_DIR="/var/lib/spamassassin" -DINSTALLSITELIB="/usr/local/share/perl/5.36.0" -DCONTACT_ADDRESS="the administrator of that system" -DRE2C_BIN="re2c" -Msharpbang -Mconditional -DPERL_BIN=""/usr/bin/perl"" -DPERL_WARN="" -DPERL_TAINT="" -m755 -isa-update.raw -osa-update
cp sa-update blib/script/sa-update
"/usr/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/sa-update
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/dmarc.t
t/dmarc.t .. Nov  3 12:36:16.977 [8195] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 1/18 Nov  3 12:36:18.906 [8197] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 3/18 Nov  3 12:36:21.247 [8199] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 5/18 Nov  3 12:36:23.737 [8201] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 7/18 Nov  3 12:36:26.153 [8203] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 9/18 Nov  3 12:36:29.117 [8205] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 11/18 Nov  3 12:36:32.796 [8207] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 13/18 Nov  3 12:36:35.338 [8209] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 15/18 Nov  3 12:36:37.374 [8211] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. ok     
All tests successful.
Files=1, Tests=18, 31 wallclock secs ( 0.02 usr  0.00 sys + 12.18 cusr  0.88 csys = 13.08 CPU)
Result: PASS

So I believe we can continue only with the 11 perl modules that are implied on Validation, and moving the rest to suggested.

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-11-13 (last edit on 2023-11-13):

#13

I'm sure we can go with this dependency splitting into validation (for binary/runtime dependencies ) and reporting (suggested dependencies): after a talk with Bryce (thanks!), passing a mail (downloaded from your inbox) through spamassassin directly and checking if the DMARC line is present in the spamassassin report:

root@Mspamassasin-suggested:~# spamc -R < message-mattermost-hey.eml | grep DMARC
-0.0 DMARC_PASS DMARC pass policy

with

root@Mspamassasin-suggested:~# apt-cache policy libmail-dmarc-perl
libmail-dmarc-perl:
  Installed: 1.20230215-1~mirespace1
  Candidate: 1.20230215-1
  Version table:
     1.20230215-1 500
        500 http://archive.ubuntu.com/ubuntu mantic/universe amd64 Packages
*** 1.20230215-1~mirespace1 500
        500 https://ppa.launchpadcontent.net/mirespace/libmail-dmarc-perl-suggested/ubuntu mantic/main amd64 Packages
        100 /var/lib/dpkg/status

after

root@Mspamassasin-suggested:~# sudo apt install libmail-dmarc-perl=1.20230215-1~mirespace1
[...]
Suggested packages:
libdbd-sqlite3-perl libdbix-simple-perl libemail-sender-perl libnet-imap-simple-perl libnet-server-perl libnet-smtps-perl libtest-file-sharedir-perl
libtest-output-perl libmojolicious-perl libscalar-number-perl libxml-sax-expatxs-perl
[..]
0 upgraded, 26 newly installed, 0 to remove and 0 not upgraded.
[..]
root@Mspamassasin-suggested:~#

In a fresh installation of libmail-dmarc-perl in Mantic (without spamassassin previously installed), we will have:

0 upgraded, 149 newly installed, 0 to remove and 0 not upgraded.

and if we install the splitted-dependencies package, we will have:

0 upgraded, 66 newly installed, 0 to remove and 0 not upgraded.

I'll make the modifications for the Noble package (that would be -ubuntu1 suffixed then) and
I will update the description in this bug also for:
- mentioning there the packages used in the DMARC validation process only that will need MIR
- testing the package [updating some of the Quality assurance sections]

root@Mspamassasin-suggested:~# spamc -R  < message-mattermost-hey.eml | grep DMARC
-0.0 DMARC_PASS             DMARC pass policy

with

root@Mspamassasin-suggested:~# apt-cache policy libmail-dmarc-perl
libmail-dmarc-perl:
  Installed: 1.20230215-1~mirespace1
  Candidate: 1.20230215-1
  Version table:
     1.20230215-1 500
        500 http://archive.ubuntu.com/ubuntu mantic/universe amd64 Packages
 *** 1.20230215-1~mirespace1 500
        500 https://ppa.launchpadcontent.net/mirespace/libmail-dmarc-perl-suggested/ubuntu mantic/main amd64 Packages
        100 /var/lib/dpkg/status

after

root@Mspamassasin-suggested:~# sudo apt install libmail-dmarc-perl=1.20230215-1~mirespace1
[...]
Suggested packages:
  libdbd-sqlite3-perl libdbix-simple-perl libemail-sender-perl libnet-imap-simple-perl libnet-server-perl libnet-smtps-perl libtest-file-sharedir-perl
  libtest-output-perl libmojolicious-perl libscalar-number-perl libxml-sax-expatxs-perl
[..]
0 upgraded, 26 newly installed, 0 to remove and 0 not upgraded.
[..]
root@Mspamassasin-suggested:~#

In a fresh installation of libmail-dmarc-perl in Mantic (without spamassassin previously installed), we will have:

0 upgraded, 149 newly installed, 0 to remove and 0 not upgraded.

and if we install the splitted-dependencies package, we will have:

0 upgraded, 66 newly installed, 0 to remove and 0 not upgraded.

I'll make the modifications for the Noble package (that would be -ubuntu1 suffixed then) and
I will update the description in this bug also for:
 -  mentioning there the packages used in the DMARC validation process only that will need MIR  
 -  testing the package [updating some of the Quality assurance sections]

Miriam España Acebal (mirespace) on 2023-11-13

description:

updated

Revision history for this message

Bryce Harrington (bryce) wrote on 2023-11-14:

#14

Glad to see you are able to verify the DMARC entry after running spamc. Why don't you also find an email that does *not* pass DMARC to verify a negative result. If you have some spam emails on hand I'll bet at least one of them fails. Or you could create one synthetically by putting typos into the From header to a non-existent domain.

Changed in libmail-dmarc-perl (Ubuntu):
importance:	Undecided → High

Miriam España Acebal (mirespace) on 2023-12-05

description:	updated
description:	updated

Miriam España Acebal (mirespace) on 2023-12-05

Changed in libmail-dmarc-perl (Ubuntu):
status:	Incomplete → New

Miriam España Acebal (mirespace) on 2023-12-05

description:

updated

Miriam España Acebal (mirespace) on 2023-12-05

Changed in libmail-dmarc-perl (Ubuntu):
assignee:	Miriam España Acebal (mirespace) → nobody

Lukas Märdian (slyon) on 2023-12-05

Changed in libmail-dmarc-perl (Ubuntu):
assignee:	nobody → Ioanna Alifieraki (joalif)

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2023-12-12:

#15

I forwarded the split idea (and also a patch) to Debian Maintainer (Noah Meyerhans):
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058492

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2024-01-09 (last edit on 2024-01-09):

#16

The status of the dependencies is the following (as a recap/resume):

* libemail-mime-perl: bug 2030880 , Won't fix -> Refactored to use MIME::Entity and MIME::Parser form MIME::Tools (fordwarded upstream [1])
   - libemail-messageid-perl: bug 2030956 , In progress-> ACK by Ioanna, All todos done
   - libemail-mime-contenttype-perl: bug 2030962 , In progress -> ACK by Ioanna, All todos done
       + libtext-unidecode-perl: bug 2031109 , In progress -> ACK by Ioanna, All todos done
   - libemail-mime-encodings-perl: bug 2031487 , Fix commited -> ACK by slyion,
   - libemail-simple-perl: bug 2031491 , New -> ACK by slyon, All todos done, (on sc review sec-2674) . We still need this package although we won't use libemail-mime-perl.

* libfile-sharedir-perl: bug 2039566 , In progress -> ACK by slyon, pending 1 recommend TODO adressed, waiting on MIR-team comment (if needed).
+ libclass-inspector-perl: bug 2039569 , Fix commited-> ACK by Didier, All todos done

* libnet-idn-encode-perl: bug 2038929 , New -> Skip to review by slyon. Package replaced by libnet-libidn-perl

* libnet-ip-perl: bug 2039456 , In progress -> ACK by Ioanna, All todos done

* libregexp-common-perl: bug 2039563 , Fix commited-> ACK by Didier, All todos done

The libmail-dmarc-perl package, with all these changes (including the dependency splitting), is in [2].

[1] https://github.com/msimerson/mail-dmarc/pull/217
[2] https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl-suggested/+sourcepub/15684761/+listing-archive-extra

Miriam España Acebal (mirespace) on 2024-02-01

description:

updated

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2024-02-01:

#17

I updated the dependency tree to be promoted as a result of the replacements of:

libemail-mime-perl -> libmime-tools-perl
libnet-idn-encode-perl -> libnet-libidn-perl

The replacements are already in main.

We still need libemail-simple-perl (libemail-mime-perl) because it is used directly by libmail-dmarc-perl.

description:

updated

Revision history for this message

Ioanna Alifieraki (joalif) wrote on 2024-02-13 (last edit on 2024-02-13):

#18

Download full text (4.4 KiB)

Review for Source Package: libmail-dmarc-perl

[Summary]
The review is based on the package provided by Miriam here:
https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl-suggested/

MIR team ACK under the constraint to resolve the below listed
required TODOs and as much as possible having a look at the
recommended TODOs.
This does need a security review, so I'll assign ubuntu-security, after the
required TODOs are addressed.

List of specific binary packages to be promoted to main: libmail-dmarc-perl
Specific binary packages built, but NOT to be promoted to main: <None>

Notes:
Required TODOs:
1. There 5 dependencies waiting to get in main, all of them have an ACK:
  a. libemail-simple-perl: https://bugs.launchpad.net/ubuntu/+source/libemail-simple-perl/+bug/2031491
  b. libfile-sharedir-perl: https://bugs.launchpad.net/ubuntu/+source/libfile-sharedir-perl/+bug/2039566
  c. libclass-inspector-perl: https://bugs.launchpad.net/ubuntu/+source/libclass-inspector-perl/+bug/2039569 - as a dependency of the above
  d. libnet-ip-perl: https://bugs.launchpad.net/ubuntu/+source/libnet-ip-perl/+bug/2039456
  e. libregexp-common-perl: https://bugs.launchpad.net/ubuntu/+source/libmail-dmarc-perl/+bug/2023971

2. Could you please clarify the status of the autopkgtest ?

Recommended TODOs:

3. It would be nice to have https://github.com/msimerson/mail-dmarc/pull/217 merged upstream.
4. It would be nice to have https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058492 in debian.

[Rationale, Duplication and Ownership]
There is no other package in main providing the same functionality.
A team is committed to own long term maintenance of this package.
The rationale given in the report seems valid and useful for Ubuntu

[Dependencies]
OK:
- no -dev/-debug/-doc packages that need exclusion
- No dependencies in main that are only superficially tested requiring
more tests now.

Problems:
- other Dependencies to MIR due to this

[Embedded sources and static linking]
OK:
- no embedded source present
- no static linking
- does not have unexpected Built-Using entries
- not a go package, no extra constraints to consider in that regard
- not a rust package, no extra constraints to consider in that regard
- Does not include vendored code

Problems: None

[Security]
OK:
- history of CVEs does not look concerning
- does not run a daemon as root
- does not use webkit1,2
- does not use lib*v8 directly
- does not expose any external endpoint (port/socket/... or similar)
- does not use centralized online accounts
- does not integrate arbitrary javascript into the desktop
- does not deal with system authentication (eg, pam), etc)
- does not deal with security attestation (secure boot, tpm, signatures)
- this makes appropriate (for its exposure) use of established risk
  mitigation features (dropping permissions, using temporary environments,
  restricted users/groups, seccomp, systemd isolation features,
  apparmor, ...)

Problems:
- does parse data formats (files [images, video, audio,
xml, json, asn.1], network packets, structures, ...) from
an untrusted source.
- does process arbitrary web content
- does not deal with cryptography (en-/decryption, certi...

Review for Source Package: libmail-dmarc-perl

[Summary]
The review is based on the package provided by Miriam here:
https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl-suggested/

List of specific binary packages to be promoted to main: libmail-dmarc-perl
Specific binary packages built, but NOT to be promoted to main: <None>

Notes:
Required TODOs:
1. There 5 dependencies waiting to get in main, all of them have an ACK:
  a. libemail-simple-perl: https://bugs.launchpad.net/ubuntu/+source/libemail-simple-perl/+bug/2031491
  b. libfile-sharedir-perl: https://bugs.launchpad.net/ubuntu/+source/libfile-sharedir-perl/+bug/2039566
  c. libclass-inspector-perl: https://bugs.launchpad.net/ubuntu/+source/libclass-inspector-perl/+bug/2039569 - as a dependency of the above 
  d. libnet-ip-perl: https://bugs.launchpad.net/ubuntu/+source/libnet-ip-perl/+bug/2039456
  e. libregexp-common-perl: https://bugs.launchpad.net/ubuntu/+source/libmail-dmarc-perl/+bug/2023971

2. Could you please clarify the status of the autopkgtest ?

Recommended TODOs:

3. It would be nice to have https://github.com/msimerson/mail-dmarc/pull/217 merged upstream.
4. It would be nice to have https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1058492 in debian.

[Dependencies]
OK:
- no -dev/-debug/-doc packages that need exclusion
- No dependencies in main that are only superficially tested requiring
  more tests now.

Problems:
- other Dependencies to MIR due to this

Problems: None

Problems:
- does parse data formats (files [images, video, audio,
  xml, json, asn.1], network packets, structures, ...) from
  an untrusted source.
- does process arbitrary web content
- does not deal with cryptography (en-/decryption, certificates,
  signing, ...)

[Common blockers]
OK:
- does not FTBFS currently
- does have a test suite that runs at build time
  - test suite fails will fail the build upon error.
- no new python2 dependency

Problems:
- does not have a non-trivial test suite that runs as autopkgtest

[Packaging red flags]
OK:
- Ubuntu does not carry a delta
- symbols tracking not applicable for this kind of code.
- debian/watch is present and looks ok (if needed, e.g. non-native)
- Upstream update history is good
- Debian/Ubuntu update history is good
- the current release is packaged
- promoting this does not seem to cause issues for MOTUs that so far
  maintained the package
- no massive Lintian warnings
- debian/rules is rather clean
- It is not on the lto-disabled list

Problems: None

[Upstream red flags]
OK:
- no Errors/warnings during the build
- no incautious use of malloc/sprintf (the language has no direct MM)
- no use of sudo, gksu, pkexec, or LD_LIBRARY_PATH (usage is OK inside
  tests)
- no use of user nobody
- no use of setuid / setgid
- no dependency on webkit, qtwebkit, seed or libgoa-*
- not part of the UI for extra checks
- no translation present, but none needed for this case (user visible)?

Problems:
- one important open bug upstream https://github.com/msimerson/mail-dmarc/issues/190

Changed in libmail-dmarc-perl (Ubuntu):
status:	New → Incomplete
assignee:	Ioanna Alifieraki (joalif) → nobody

Christian Ehrhardt  (paelzer) on 2024-02-20

Changed in libmail-dmarc-perl (Ubuntu):
assignee:	nobody → Miriam España Acebal (mirespace)

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2024-02-21 (last edit on 2024-02-21):

#19

Hi Ioanna,

sorry, I forgot to update the bug: I finished the autopackage tests (they were OK, and I removed the task from my mind), and it can be run:

https://autopkgtest.ubuntu.com/results/autopkgtest-noble-mirespace-libmail-dmarc-perl-suggested/

  - libmail-dmarc-perl/1.20230215-1ubuntu1~mirespace7
    + ✅ libmail-dmarc-perl on noble for amd64 @ 21.02.24 11:08:26 Log️ 🗒️
    + ✅ libmail-dmarc-perl on noble for arm64 @ 21.02.24 11:08:55 Log️ 🗒️
    + ✅ libmail-dmarc-perl on noble for armhf @ 21.02.24 11:13:34 Log️ 🗒️
    + ✅ libmail-dmarc-perl on noble for i386 @ 21.02.24 11:08:16 Log️ 🗒️
    + ✅ libmail-dmarc-perl on noble for ppc64el @ 21.02.24 11:09:25 Log️ 🗒️
    + ✅ libmail-dmarc-perl on noble for s390x @ 21.02.24 11:39:48 Log️ 🗒️

Log for amd64 https://autopkgtest.ubuntu.com/results/autopkgtest-noble-mirespace-libmail-dmarc-perl-suggested/noble/amd64/libm/libmail-dmarc-perl/20240221_110826_ac1bc@/log.gz

Changed in libmail-dmarc-perl (Ubuntu):
assignee:	Miriam España Acebal (mirespace) → nobody

Revision history for this message

Ioanna Alifieraki (joalif) wrote on 2024-02-22:

#20

Hi Miriam,

Thanks for the autopackage tests.
We still need to wait for the dependencies but I'll assign this to security to get it into the pipeline.

Changed in libmail-dmarc-perl (Ubuntu):
assignee:	nobody → Ubuntu Security Team (ubuntu-security)

Mark Esler (eslerm) on 2024-02-22

tags:

added: sec-3890

Revision history for this message

Miha Purg (mihap) wrote on 2024-03-21:

#21

Hi all,

During the security review I noticed that several dependencies for this package were replaced with those that already exist in main:
- Net::IDN::Encode -> Net::LibIDN [1]
- Email::MIME -> MIME::Parser & MIME::Entity [2]
To my understanding, this was done to avoid introducing unnecessary and/or duplicate functionality to main, and to mitigate a known security vulnerability in the Email::MIME library [3].

I have some concerns regarding the patch needed to replace Email::MIME which I wanted to bring up for discussion, especially in light of the fact that the vulnerability has since been confirmed fixed [4] . Although an elegant solution, the patch makes non-trivial changes to the source, and, although it passes all tests, these change have not been battle tested. Moreover, I suspect there will be some unwanted implications on maintenance and support for the modified library on the long run if upstream does not accept the proposed changes (see ongoing discussion in [5]). Should we still consider this the same library as upstream in the end and who will maintain the modified code? I'm wondering if this is still the best approach considering that the vulnerability has been fixed, and that upstream is not leaning in our direction [5].

[1]: https://bugs.launchpad.net/ubuntu/+source/libnet-idn-encode-perl/+bug/2038929
[2]: https://bugs.launchpad.net/ubuntu/+source/libemail-mime-perl/+bug/2030880
[3]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960062
[4]: https://github.com/msimerson/mail-dmarc/issues/216#issuecomment-1945033737
[5]: https://github.com/msimerson/mail-dmarc/pull/217

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2024-03-21 (last edit on 2024-03-21):

#22

Hi Miha,

First of all, thanks for your work on this Miha.

You've summarized it very well and you're right that moving in a different direction than upstream always involves more delicate work in the future. Upstream's discussion is ongoing [1], and we don't know the decision they could make even if we (I) provide as fast as possible what they need to take it:

- specific tests
- data (that we have from this MIR study already)
- Extending the patch: completely switching even from libemail-simple-perl for general email handling.
- and as extra and beyond: making/checking it is compliant with latest RFC DMARC/DMARCbisfor, making our proposal more robust (although it is out-of-scope of the original PR) [1.1]

since for the moment they don't find a compelling reason to change the dependency [1.2] because:

- the security issue is solved [2] (as you rightly point out too)
- they are perfectly within their rights that our main/universe distinctions are not a priority for them.

Coming back to the Requires TODO for ACK in the MIR review of libemail-mime-perl [3], the #0 is that libemail-mime-perl passes the security review: Would that package be reviewed by security to see if we can go back to it? The security review (SEC-2671) was blocked and later rejected due to the switching to libmime-tools-perl.

I have no objection to reverting back to libemail-mime-perl if:
- The libemail-mime-perl security review (SEC-2671) passes with an ACK/OK.
- TODO #2 (duplicity) can be overcome.

Thank you in advance!

[1] https://github.com/msimerson/mail-dmarc/pull/217
[1.1] https://github.com/msimerson/mail-dmarc/pull/217#issuecomment-2010843067
[1.2] https://github.com/msimerson/mail-dmarc/pull/217#issuecomment-2010862484
[2] https://github.com/msimerson/mail-dmarc/issues/216#issuecomment-1945033737
[3] https://bugs.launchpad.net/ubuntu/+source/libemail-mime-perl/+bug/2030880/comments/1

Revision history for this message

Lukas Märdian (slyon) wrote on 2024-03-26:

#23

Thanks for the summary, Miriam!

We have to make a call between duplicated work because of two similar packages in "main". Or extra work because of carrying non-mainstream patches...

Both libemail-mime-perl and libmime-tools-perl are owned by ~ubuntu-server, so I'd like to get their opinion.

> sarnold> one of the comments on https://github.com/msimerson/mail-dmarc/pull/217 suggested that the original requirements are also requirements for spamassassin 4.0, so suddenly it feels more plausible to use the original requirements..

https://github.com/msimerson/mail-dmarc/pull/217#issuecomment-2010087005

Should we really need to support libemail-mime-perl in the future for spamassassin 4.0, it could make sense to introduce duplication in the archive.

Revision history for this message

Miha Purg (mihap) wrote on 2024-03-28:

#24

Download full text (9.6 KiB)

I reviewed libmail-dmarc-perl 1.20230215-1 as checked in noble. This shouldn't be considered a full
audit but rather a quick gauge of maintainability.

libmail-dmarc-perl is a Perl module implementing DMARC. It can be used by:
- MTAs and filtering tools like SpamAssassin to **validate** that incoming messages are aligned with purported sender's policy
- email senders, to **receive DMARC reports** from other mail servers and display them via CLI and web interfaces
- MTA operators to **send DMARC reports** to DMARC author domains

The main use-case and the only reverse dependency of libmail-dmarc-perl in Ubuntu is SpamAssassin.
It requires only the validation functionality by default, thus, as was discussed in the MIR process,
unnecessary dependencies related to reporting functionality were moved to Suggested and will not be
promoted to main at this time, thus limiting the functionality and reducing attack surface in the main archive.

- CVE History
  - None
  - Github issues with potential security implications (and not related to *reporting* functionality):
    - https://github.com/msimerson/mail-dmarc/issues/121 (does not seem exploitable)
  - Dependency issues:
    - https://github.com/rjbs/Email-MIME/issues/66#issuecomment-626260473 (fixed)

I reviewed libmail-dmarc-perl 1.20230215-1 as checked in noble. This shouldn't be considered a full
audit but rather a quick gauge of maintainability.

- Build-Depends
  - large dependency tree when both validation and reporting functionality is
    required, including DBD::SQLite, Data::Dumper, HTTP::Tiny, IO::Socket::SSL,
    NET::DNS::Resolver, Email::Sender, NET::SSLeay, Sys::Syslog, Net::IDN::Encode... 
  - the main use-case (SpamAssassin) only uses the validation functionality,
    which significantly reduced the dependency count (Regexp::Common, Config::Tiny,
    File::ShareDir, Net::DNS::Resolver, Net::IP, Net::IDN::Encode).The original MIR
    proposes splitting dependencies into main (validation) and universe (reporting).
  - Suggested change for including in main is to replace `Net::IDN::Encode` (pure perl
    implementation of IDN, last update 2018, unresponsive maintainer - 
    https://github.com/cfaerber/Net-IDN-Encode/pull/11#issuecomment-1919484551)
    with `Net::LibIDN` (perl binding for gnu-libidn, last update 2009, last update for
    gnu-libidn 2024) to avoid adding duplicate functionality to main. This replaces an
    unmaintained single-purpose library for a maintained and common library, but memory
    unsafe, which is an acceptable tradeoff.
  - Suggested change for including in main is to replace `Email::MIME` (last update 2023-1,
    several open issues) with `MIME::Tools`  (last update 2024-02, infrequent updates,
    many very old open issues), in an attempt to fix a DoS issue, requiring non-trivial
    modifications to source code (see previous comment in MIR bug report). 
- pre/post inst/rm scripts
  - none
- init scripts
  - none
- systemd units
  - none
- dbus services
  - none
- setuid binaries
  - none
- binaries in PATH
  - Several cmdline tools for looking up, receiving, and sending DMARC reports.
    The threat model is the same as for the library itself (reporting functionality)
    and depends on the user context. The tools should ideally not be run by a privileged user.
    - rwxr-xr-x root/root      2212 2023-06-17 17:38 ./usr/bin/dmarc_lookup
    - rwxr-xr-x root/root      1973 2023-06-17 17:38 ./usr/bin/dmarc_receive
    - rwxr-xr-x root/root       387 2023-06-17 17:38 ./usr/bin/dmarc_send_reports
    - rwxr-xr-x root/root      1864 2023-06-17 17:38 ./usr/bin/dmarc_update_public_suffix_list
    - rwxr-xr-x root/root      7846 2023-06-17 17:38 ./usr/bin/dmarc_view_reports
  - The tools are generally not applicable to the Spamassassin validation use-case,
    unless aggregation of reports is enabled in Spamassassin and the Suggested
    dependencies from universe are installed.
- sudo fragments
  - none
- polkit files
  - none
- udev rules
  - none
- unit tests / autopkgtests
  - Unit tests using Perl::More, continuous integration using Travis
  - Coverage is 78% (low mostly due to reporting and DB functionalities)
  - Perl::Critic is evaluated in 'stern' mode for each build
  - Locally all tests pass (required dependencies: libtest-file-sharedir-perl libdbd-sqlite3-perl
    libtest-exception-perl libxml-validator-schema-perl  libemail-sender-perl libdbix-simple-perl
    libtest-output-perl)
- cron jobs
  - An example cronjob is provided for updating the public suffix list. It calls
    `dmarc_update_public_suffix_list` as root, which only mirrors the PSL file from
    https://publicsuffix.org/list/effective_tld_names.dat to the path defined in the config file.
    Generally safe, but would be better to run as a dedicated service account.
- Build logs
  - none

- Processes spawned
  - Unsafe use of string-`eval` for loading modules at run time 
    (see issue https://github.com/msimerson/mail-dmarc/issues/234)
  - No backticks, `qx`, `system`.
- Memory management
  - none
- File IO
  - Generally ok - all calls to open() use the safe 3-argument version 
  - The config file is first searched for in potentially insecure directories
    (order is `MAIL_DMARC_CONFIG_FILE, ./, /usr/local/etc, /opt/local/etc, /etc/, ./etc`) 
    (see issue https://github.com/msimerson/mail-dmarc/issues/231).
    In the *very unlikely* event that the malicious user controls the env.var. or working directory,
    this could open attack vectors via malicious configuration options (e.g. command execution via
    the `backend` variable or DoS by using a large PSL file).
  - Public_suffix_file is trusted and read into hash without checking type/size, which could
    lead to DoS in certain unlikely scenarios (see above)
- Logging
  - syslog is used for logging in Sender submodule - no obvious issues
- Environment variable usage
  - MAIL_DMARC_CONFIG_FILE can be used to specify a user-controlled config file. See FileIO.
- Use of privileged functions
  - none
- Use of cryptography / random number sources etc
  - See Networking.
  - Cryptographicaly insecure rand() is used in PurePerl module to generate a percentage (0-100) for
    checking against DMARC PCT value. This is not an issue as it is not security-sensitive use-case.
- Use of temp files
  - none
- Use of networking
  - A CGI HTTP server for viewing reports is bundled with the library but is removed from the Ubuntu
    package and was thus not reviewed.
  - Primary use-case of the library in the context of Spamassassin is DMARC validation, where possible attack
    vectors are via a malicious email header (`header_from_raw`) or malicious dmarc record (`policy->parse()`).
    The possible injection points have been reviewed and are sanitized and thoroughly validated.
  - Aggregate DMARC reports can optionally be stored in a SQL database. Database queries are properly
    parameterized. Few contain concatenated values but these are not user controllable.
  - Reports stored in the DB can be sent using `dmarc_send_reports` program. STARTTLS is used when
    relaying via a "smarthost" SMTP server with authentication (credentials in config file). When
    the report is sent directly to "mailto:", STARTTLS is used if available, otherwise no security,
    which is acceptable due to non-sensitivity of the data.
  - Received DMARC reports are obtained with `dmarc_receive` either from local file, mbox,
    or from IMAP (credentials in config file). TLS via port 993 is used by default, but will downgrade
    to plaintext if Socket::SSL is not loadable (only prints a warning) or will accept any certificate
    if Mozilla::Ca is not loadable (no warning). Upstream issue: https://github.com/msimerson/mail-dmarc/issues/233
- Use of WebKit
  - none
- Use of PolicyKit
  - none

- Any significant cppcheck results
  - none
- Any significant Coverity results
  - none
- Any significant shellcheck results
  - none
- Any significant bandit results
  - none
- Any significant govulncheck results
  - none
- Any significant Semgrep results
  - none
- Other notes
  - HTTP client functionality is implemented for sending reports but it is not used
    internally.  The stale code should ideally be removed.
  - Poor code coverage on reporting functionality should be improved.
  - Missing `security.md`.

The threat model for the primary use-case (validation in Spamassassin) includes
the processing of malicious email headers and dns dmarc records, potentially leading to
improper DMARC validation, denial of service or code exection on the host.
The possible injection points are generally well sanitized and validated to prevent abuse.
Minor issues related to file loading were noticed and raised upstream:
https://github.com/msimerson/mail-dmarc/issues/231

The threat model for the reporting functionality extends the above to processing
of malicious received reports (email subject, body, and attachement metadata), as
well as tampering and information disclosure due to insecure connection to external
(authenticated) SMTP/IMAP servers. In this case the user-controlled data is largely
processed with external libraries (IO::Uncompress, Email::Simple, Email::MIME,
XML::LibXML), and is thus outside the scope of this review.

The connection to IMAP was identified to not strictly enforce TLS, and the issue was
raised upstream: https://github.com/msimerson/mail-dmarc/issues/233

Additionally, an unsafe use of string-`eval` was discovered, which allows for
arbitrary code execution and possibly privilege escalation in the unlikely scenario
that the library is executed in a privileged context AND
in a world-writable direcory (e.g. root runs `dmarc_view_reports` in /tmp).
Upstream issue: https://github.com/msimerson/mail-dmarc/issues/234

The upstream maintainer responded promptly is resolving the issues.

Security team ACK for promoting libmail-dmarc-perl to main. This is irrespective
of the decision on whether the Email::MIME dependency is to be replaced or not.

Changed in libmail-dmarc-perl (Ubuntu):
status:	Incomplete → In Progress
assignee:	Ubuntu Security Team (ubuntu-security) → nobody

Revision history for this message

Bryce Harrington (bryce) wrote on 2024-04-03:

#25

> We have to make a call between duplicated work because of two similar packages in "main". Or extra work because of carrying non-mainstream patches...
>
> Both libemail-mime-perl and libmime-tools-perl are owned by ~ubuntu-server, so I'd like to get their opinion.

From a ~ubuntu-server perspective, perl packages are very, very low maintenance work, so having two similar packages is not a problem in the slightest since they sync from debian without needing any attention at all. Maintaining some non-mainstream patches going forward would be a small bit of work but absolutely acceptable if that is desired.

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2024-04-09:

#26

Download full text (5.0 KiB)

After Miha did the security review, he found that dmarc_receive was failing. It wasn't covered here as dmarc's binary scripts are not used by spamassassin, but I thought it was worth checking.

The steps to reproduce (thanks Miha!) were:

1) Create dmarc xml based on google's example

2) Convert to gzipped base64

$ cat dmarc_example.xml | gzip | base64 > dmarc_example.xml.gz.base64

3) Copy base64 into example email (test_email.eml) from https://datatracker.ietf.org/doc/html/rfc7489#appendix-B.5

4) Run dmarc_receive

$ dmarc_receive --file test_mail.eml &> out.txt

The bad output contains "Can't locate object method "getline" via package "From: <email address hidden> ..."

I caught the bug and amended it:

diff -Nru libmail-dmarc-perl-1.20230215/debian/patches/use-MIME-Entity-and-MIME-Parser-from-libmime-tools-p.patch libmail-dmarc-perl-1.20230215/debian/patches/use-MIME-Entity-and-MIME-Parser-from-libmime-tools-p.patch
--- libmail-dmarc-perl-1.20230215/debian/patches/use-MIME-Entity-and-MIME-Parser-from-libmime-tools-p.patch 2023-12-11 14:43:31.000000000 +0000
+++ libmail-dmarc-perl-1.20230215/debian/patches/use-MIME-Entity-and-MIME-Parser-from-libmime-tools-p.patch 2023-12-11 14:43:31.000000000 +0000
@@ -125,7 +125,7 @@
- foreach my $part ( Email::MIME->new( $email->as_string )->parts ) {
- my ($c_type) = split /;/, $part->content_type || '';
+ my $parser = MIME::Parser->new;
-+ foreach my $part ( $parser->parse( $email->as_string )->parts_DFS ) {
++ foreach my $part ( $parser->parse_data( $email->as_string )->parts_DFS ) {
+ next if defined(!$part->bodyhandle); # something to process
+ my ($c_type) = split /;/, $part->effective_type || '';
next if $c_type eq 'text/plain';

dmarc building tests are passing [1] and also autopkgtests that cover the splitting:

autopkgtest [14:02:18]: test splitting-check: [-----------------------
autopkgtest [14:03:08]: test splitting-check: -----------------------]
autopkgtest [14:03:09]: test splitting-check: - - - - - - - - - - results - - - - - - - - - -
splitting-check PASS
autopkgtest [14:03:10]: @@@@@@@@@@@@@@@@@@@@ summary
splitting-check PASS

Spamassassin test for dmarc also passes:

After Miha did the security review, he found that dmarc_receive was failing. It wasn't covered here as dmarc's binary scripts are not used by spamassassin, but I thought it was worth checking.

The steps to reproduce (thanks Miha!) were:

1) Create dmarc xml based on google's example

2) Convert to gzipped base64

$ cat dmarc_example.xml | gzip | base64 > dmarc_example.xml.gz.base64

3) Copy base64 into example email (test_email.eml) from https://datatracker.ietf.org/doc/html/rfc7489#appendix-B.5

4) Run dmarc_receive

$ dmarc_receive --file test_mail.eml &> out.txt

The bad output contains "Can't locate object method "getline" via package "From: dmarc-reporting@mail.receiver.example ..."

I caught the bug and amended it:

diff -Nru libmail-dmarc-perl-1.20230215/debian/patches/use-MIME-Entity-and-MIME-Parser-from-libmime-tools-p.patch libmail-dmarc-perl-1.20230215/debian/patches/use-MIME-Entity-and-MIME-Parser-from-libmime-tools-p.patch
--- libmail-dmarc-perl-1.20230215/debian/patches/use-MIME-Entity-and-MIME-Parser-from-libmime-tools-p.patch	2023-12-11 14:43:31.000000000 +0000
+++ libmail-dmarc-perl-1.20230215/debian/patches/use-MIME-Entity-and-MIME-Parser-from-libmime-tools-p.patch	2023-12-11 14:43:31.000000000 +0000
@@ -125,7 +125,7 @@
 -    foreach my $part ( Email::MIME->new( $email->as_string )->parts ) {
 -        my ($c_type) = split /;/, $part->content_type || '';
 +    my $parser = MIME::Parser->new;
-+    foreach my $part ( $parser->parse( $email->as_string )->parts_DFS ) {
++    foreach my $part ( $parser->parse_data( $email->as_string )->parts_DFS ) {
 +        next if defined(!$part->bodyhandle); # something to process
 +        my ($c_type) = split /;/, $part->effective_type || '';
          next if $c_type eq 'text/plain';

dmarc building tests are passing [1] and also autopkgtests that cover the splitting:

autopkgtest [14:02:18]: test splitting-check: [-----------------------
autopkgtest [14:03:08]: test splitting-check: -----------------------]
autopkgtest [14:03:09]: test splitting-check:  - - - - - - - - - - results - - - - - - - - - -
splitting-check      PASS
autopkgtest [14:03:10]: @@@@@@@@@@@@@@@@@@@@ summary
splitting-check      PASS

Spamassassin test for dmarc also passes:

root@NDmarc-spamassassin-tests:~/spamassassin# make test TEST_FILES="t/dmarc.t"
"/usr/bin/perl" build/mkrules --exit_on_no_src --src rulesrc --out rules --manifest MANIFEST --manifestskip MANIFEST.SKIP
mkrules: no rules updated
"/usr/bin/perl" build/preprocessor  -Mvars -DVERSION="4.000000" -DPREFIX="/usr/local" -DDEF_RULES_DIR="/usr/local/share/spamassassin" -DLOCAL_RULES_DIR="/etc/mail/spamassassin" -DLOCAL_STATE_DIR="/var/lib/spamassassin" -DINSTALLSITELIB="/usr/local/share/perl/5.38.2" -DCONTACT_ADDRESS="the administrator of that system" -DRE2C_BIN="re2c" -Msharpbang -Mconditional -DPERL_BIN=""/usr/bin/perl"" -DPERL_WARN="" -DPERL_TAINT="" -m755 -isa-update.raw -osa-update
cp sa-update blib/script/sa-update
"/usr/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/sa-update
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/dmarc.t
t/dmarc.t .. Apr  9 12:29:54.780 [9230] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 1/18 Apr  9 12:29:59.035 [9232] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 3/18 Apr  9 12:30:03.208 [9234] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 5/18 Apr  9 12:30:06.210 [9236] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 7/18 Apr  9 12:30:09.197 [9238] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 9/18 Apr  9 12:30:12.643 [9240] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 11/18 Apr  9 12:30:17.225 [9242] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 13/18 Apr  9 12:30:21.491 [9244] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. 15/18 Apr  9 12:30:25.294 [9246] warn: deprecated method; size() is an alias of "UDPsize()" at ../blib/lib/Mail/SpamAssassin/DnsResolver.pm line 602.
t/dmarc.t .. ok     
All tests successful.
Files=1, Tests=18, 34 wallclock secs ( 0.02 usr  0.00 sys + 12.72 cusr  1.55 csys = 14.29 CPU)
Result: PASS

I upload the fixed package at https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl-suggested/ as I've been doing (fixed package is ~mirespace10)

[1] https://launchpadlibrarian.net/723909463/buildlog_ubuntu-noble-amd64.libmail-dmarc-perl_1.20230215-1ubuntu1~mirespace10_BUILDING.txt.gz

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2024-04-09 (last edit on 2024-04-09):

#27

Regarding the debate about libmime-tools-perl vs libemail-mime-perl, I checked that libemail-mime-perl is not being used in the last version of spamassassin directly:

https://salsa.debian.org/debian/spamassassin/-/blob/master/debian/control?ref_type=heads

Also, using the example of dmarc_receive in the previous comment, using libmime-tools-perl is enough to make it work but, if we use libemail-mime-perl, we would need to MIRs :
- libdbd-sqlite3-perl bug 2029379 : ready for promotion
- libdbix-simple-perl bug 2030731: needs to move some dependencies from Recommends to Suggests for unblock the MIR and check usability

I checked it by using libemail-mime-perl as suggested in dmarc's dependencies - it was one of the last checks I did for validating the change to libmime-tools-perl before libemail-mime-perl passed the security review-. I had to install by hand all that is needed to run the dmarc_receive tool (the libdb* packages above).

I'm not saying it's time to review all the binary Perl scripts since they were considered outside the validation feature of DMARC and are not in use by spamassassin, but it's a minor collateral win that some of them could work, too, with the changes already made to the dmarc Perl package.

Revision history for this message

Lukas Märdian (slyon) wrote on 2024-04-09:

#28

Setting status back to "Incomplete" as there are still discussions happening and we don't want it to get lost from the MIR report.

Also see update in https://github.com/rjbs/Email-MIME/issues/66#issuecomment-2024085120

Changed in libmail-dmarc-perl (Ubuntu):
status:	In Progress → Incomplete

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2024-04-15 (last edit on 2024-04-15):

#29

Version of spamassassin using dmarc has been uploaded to [1] . I'm opening a FFe for spamassassin for this.

Using the libmail-dmarc-perl package that has been ack to be MIRed that is at [2], we can confirm the installation of both and the packages to finally be promoted (that coincides with the summary at the beginning of the description):

#1 Checking smapassassin and libmail-dmarc-perl version to be installed if we do am 'apt install spamassassin' in Noble with the spamassassin package in PPA for FFe.

root@Ndmarc-final:~# apt-cache policy libmail-dmarc-perl
libmail-dmarc-perl:
  Installed: (none)
  Candidate: 1.20230215-1ubuntu1~mirespace10
  Version table:
     1.20230215-1ubuntu1~mirespace10 500
        500 https://ppa.launchpadcontent.net/mirespace/libmail-dmarc-perl-suggested/ubuntu noble/main amd64 Packages
     1.20230215-1 500
        500 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages
root@Ndmarc-final:~# apt-cache policy spamassassin
spamassassin:
  Installed: (none)
  Candidate: 4.0.0-8ubuntu5
  Version table:
     4.0.0-8ubuntu5 500
        500 https://ppa.launchpadcontent.net/mirespace/spamassassin-dmarc-mir/ubuntu noble/main amd64 Packages
     4.0.0-8ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages

#2 Checking packages in universe for this, to get the list to be promoted finally:

root@Ndmarc-final:~# for p in $(apt install -s spamassassin | grep Inst | cut -d' ' -f2); do echo -n ${p} && echo -n " " && apt-cache policy ${p} | grep archive.ubuntu | cut -d'/' -f5 | cut -d' ' -f1 ; done | grep universe

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libclass-inspector-perl universe
libemail-simple-perl universe
libfile-sharedir-perl universe
libnet-ip-perl universe
libregexp-common-perl universe
libmail-dmarc-perl universe

[1] https://launchpad.net/~mirespace/+archive/ubuntu/spamassassin-dmarc-mir
[2] https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl-suggested

Version of spamassassin using dmarc has been uploaded to [1] . I'm opening a FFe for spamassassin for this.

#1 Checking smapassassin and libmail-dmarc-perl version to be installed if we do am 'apt install spamassassin' in Noble with the spamassassin package in PPA for FFe.

#2 Checking packages in universe for this, to get the list to be promoted finally:

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libclass-inspector-perl universe
libemail-simple-perl universe
libfile-sharedir-perl universe
libnet-ip-perl universe
libregexp-common-perl universe
libmail-dmarc-perl universe

[1] https://launchpad.net/~mirespace/+archive/ubuntu/spamassassin-dmarc-mir
[2] https://launchpad.net/~mirespace/+archive/ubuntu/libmail-dmarc-perl-suggested

Revision history for this message

Miriam España Acebal (mirespace) wrote on 2024-04-16:

#30

FFe granted at https://bugs.launchpad.net/ubuntu/+source/spamassassin/+bug/2061379/comments/1

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2024-04-17:

#31

Thanks for also ensuring an FFE.

All MIR requested modifications have been implemented, thereby this is now ready to go and seen in component mismatches already.
Updating the state.

Changed in libmail-dmarc-perl (Ubuntu):
status:	Incomplete → Fix Committed

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2024-04-17:

#32

The list of mismatches in proposed matches the expected set, all of those cases are ready from the MIR and security POV. Furthermore there is a FFE approved for the same context.

I've ensured our team subscriptions are complete (they already are) on each of those.

After the next run of component mismatches hopefully confirming all this I will promote them to clear -proposed.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2024-04-17:

#33

Full stack ready, all dependencies seen in component mismatches, FFE approved, MIR approved, only the top level change pulling this in is in -proposed and ready there other than this mismatch -> promoting.

Override component to main
libmail-dmarc-perl 1.20230215-1ubuntu1 in noble: universe/misc -> main
libmail-dmarc-perl 1.20230215-1ubuntu1 in noble amd64: universe/perl/optional/100% -> main
libmail-dmarc-perl 1.20230215-1ubuntu1 in noble arm64: universe/perl/optional/100% -> main
libmail-dmarc-perl 1.20230215-1ubuntu1 in noble armhf: universe/perl/optional/100% -> main
libmail-dmarc-perl 1.20230215-1ubuntu1 in noble i386: universe/perl/optional/100% -> main
libmail-dmarc-perl 1.20230215-1ubuntu1 in noble ppc64el: universe/perl/optional/100% -> main
libmail-dmarc-perl 1.20230215-1ubuntu1 in noble riscv64: universe/perl/optional/100% -> main
libmail-dmarc-perl 1.20230215-1ubuntu1 in noble s390x: universe/perl/optional/100% -> main
Override [y|N]? y
8 publications overridden.