Enabling SPF checks with CHECK_RCPT_SPF doesn't work

[Dominic]

As I understand it, enabling SPF validation in Exim4 simply requires setting CHECK_RCPT_SPF to true and installing the spf-tools-perl package.

However, in mantic, every email has this header, regardless of whether the sender's domain has an SPF TXT record set:

Received-SPF: none

[Dominic (triatic)]

Note: this bug relates to inbound IPv4 addresses. IPv6 addresses are affected by a separate bug: #2056443

[Mitchell Dzurick (mitchdz)]

Hi Dominic,

Thanks for taking the time to submit this bug report and make Ubuntu better!

Where are you setting the variable CHECK_RCPT_SPF = true ?

Doing a quick google search I found[0] which looks like it may be useful. It says to do the following -

1) Add `CHECK_RCPT_SPF = true` to the top of /etc/exim4/exim4.conf.template
2) Run `update-exim4.conf`
3) Run `systemctl restart exim4`

If you could supply all the steps on how you configured your server and reproduced the issue, that may be helpful as well.

[0] - https://j11g.com/2023/07/24/correctly-configuring-incoming-spf-in-exim-on-debian/

[Dominic (triatic)]

Hi Mitchell,

I am setting CHECK_RCPT_SPF = true in /etc/exim4/exim4.conf.localmacros where local macros are typically set.

I have run `update-exim4.conf` and `systemctl restart exim4`.

Then I send myself an email from an outlook.com account, which has SPF enabled, and the headers report:

Received-SPF: none

I am expecting to see:

Received-SPF: pass

[Lucas Kanashiro (lucaskanashiro)]

Could you provide your config file? Also state if you applied any customization in your set up? Those things would be useful to reproduce what you described locally.

[Dominic (triatic)]

exim4.conf.template is completely unmodified from stock mantic.

Bug can be replicated by:

1. Installing mantic
2. Installing exim4 and spf-tools-perl
3. `dpkg-reconfigure exim4-config` and make it accept mail
4. Adding CHECK_RCPT_SPF = true to /etc/exim4/exim4.conf.localmacros
5. Sending yourself an email from Gmail or Outlook and checking the spf header

That's all I did.

[Dominic (triatic)]

Also: `update-exim4.conf` and restart exim before sending mail to yourself.

[Athos Ribeiro (athos-ribeiro)]

Hi Dominic,

Thanks for the additional information.

Are you using exim4-daemon-heavy or exim4-daemon-light? In case you are using the light version, could you try it with the heavy one as well and report it back here?

[Dominic (triatic)]

I'm using exim4-daemon-heavy.

$ apt list --installed | grep exim4

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

exim4-base/mantic-updates,mantic-security,now 4.96-17ubuntu2.2 amd64 [installed,automatic]
exim4-config/mantic-updates,mantic-security,now 4.96-17ubuntu2.2 all [installed,automatic]
exim4-daemon-heavy/mantic-updates,mantic-security,now 4.96-17ubuntu2.2 amd64 [installed,automatic]
exim4/mantic-updates,mantic-security,now 4.96-17ubuntu2.2 all [installed]

[Dominic (triatic)]

According to the exim4 changelog, the spf logic was modified in Aug 2023.

exim4 (4.96-17ubuntu1) mantic; urgency=medium

  * Merge with Debian unstable (LP: #2030098). Remaining changes:
     - Disable external SPF support to avoid Build-Depends on libspf2-dev
       (only available in universe). SPF can still be implemented via
       spf-tools-perl, as documented in exim4.conf.template. This reverts
       Vcs-Git commit 494f1fe, first released in 4.95~RC0-1.
       (LP #1952738)
       + d/control: drop Build-Depends on libspf2-dev.
       + d/EDITME.exim4-heavy.diff: disable support for libspf2.
       + d/d/c/a/30_exim4-config_check_rcpt: restore SPF logic based
         on spfquery.mail-spf-perl from spf-tools-perl, but without
         the previously supported helo detection.

[Paride Legovini (paride)]

Hello Dominic,

The fact that you get that

  Received-SPF: none

headers means that spf-tools-perl gets called by exim4. What I suggest to debug this is to manually call spfquery.mail-spf-per with the relevant parameters (see the spfquery.mail-spf-perl manpage), and check if what kind of reply you get for your incoming outlook mail. The tool's help screen (spfquery.mail-spf-perl --help) has examples on how to invoke it.

[Dominic (triatic)]

I get the correct result when running spfquery.mail-spf-perl manually.

$ /usr/bin/spfquery.mail-spf-perl --ip 40.92.113.65 --scope mfrom --identity <email address hidden>
pass
outlook.com: Sender is authorized to use '<email address hidden>' in 'mfrom' identity (mechanism 'include:spf.protection.outlook.com' matched)
outlook.com: Sender is authorized to use '<email address hidden>' in 'mfrom' identity (mechanism 'include:spf.protection.outlook.com' matched)
Received-SPF: pass (outlook.com: Sender is authorized to use '<email address hidden>' in 'mfrom' identity (mechanism 'include:spf.protection.outlook.com' matched)) receiver=alpha.cress.org.uk; identity=mailfrom; <email address hidden>"; client-ip=40.92.113.65

[Dominic (triatic)]

Still an issue in noble / 24.04.

"Received-SPF: none" is reported in the headers of received mail for senders known to have an spf record configured.

[Dominic (triatic)]

Also, this should not be marked as "incomplete" as I have supplied all the required information.

[Mitchell Dzurick (mitchdz)]

Dominic, did you upgrade to mantic and start seeing this failure, or did you make a new mantic install and set things up from scratch?

I wonder if some logic elsewhere was changed in Debian that we missed, or has this always been broken?

This scenario would also be a nice thing to have a dep8 test going forward if that can be setup, I'm just not sure if that's feasible in the testbed.

[Dominic (triatic)]

It was a clean install of Noble, for some reason the spf check always returns "none". It is very easily reproducible, as per my comment #5.

[Dominic (triatic)]

Possibly related to: https://bugs.launchpad.net/ubuntu/+source/exim4/+bug/1998678

[Mitchell Dzurick (mitchdz)]

I was looking into bug #1998678 as a potential reason, do you see any concerning logs in your /var/log/exim4/mainlog (or wherever your logs are)?

[Dominic (triatic)]

No errors in /var/log/exim4/mainlog, just normal mail flow logs.

I suspect some sort of error was introduced in the code for the #1998678 fix.

[Mitchell Dzurick (mitchdz)]

That could be a possibility.

On a side note, I tried to setup a testbed to reproduce this (and start working on a dep8 test) and can't seem to get the setup working correctly. I followed the steps in bug #1998678, see the mail in `/var/mail/ubuntu` but don't see any Received-SPF headers at all, is that expected? I think I must be missing something in my setup.

[Dominic (triatic)]

No, you should be seeing a header. Are the steps you followed the same as that which I wrote in comment #5 of this report?

[Dominic (triatic)]

(and comment #6)

[Mitchell Dzurick (mitchdz)]

Hi, just want to add a quick comment that I’m on vacation until the end of the week so don’t expect a reply from me until then. Also will be attending the Canonical sprint next week so might not be able to update this next week either.

Feel free to take a stab at this if you are triaging this bug without me.

[Dominic (triatic)]

Re-marking as new since I've supplied all requested information.

[Mitchell Dzurick (mitchdz)]

Apologies for the delay, I'm picking this back up today.

[Mitchell Dzurick (mitchdz)]

I have reproduced this issue on a live noble server running exim4. Digging deeper now.

[Mitchell Dzurick (mitchdz)]

I think I have found a fix for this issue.

My hunch is there is something buggy about the run{} command when invoking programs directly. I wrapped the contents of run with an explicit shell invocation, e.g.

    condition = ${run{/usr/bin/spfquery.mail-spf-perl \
                   --ip ${quote:$sender_host_address} \
                   --scope mfrom \
                   --identity ${quote:$sender_address}} \
                   {no}{${if eq {$runrc}{1}{yes}{no}}}}

now becomes

    condition = ${run{/bin/bash -c "/usr/bin/spfquery.mail-spf-perl \
                   --ip ${quote:$sender_host_address} \
                   --scope mfrom \
                   --identity ${quote:$sender_address}"} \
                   {no}{${if eq {$runrc}{1}{yes}{no}}}}

Dominic, are you still using mantic or are you on noble by now? I'd be glad to prepare a package for you to test if you are comfortable with that. Alternatively you can test the changes I mentioned here. This code is found in /etc/exim4/exim4.conf.template but of course be careful if you are modifying this file on a live system.

I've tested this code change with a valid E2E case with SPF, and I also tested a local lxc system with no SPF to ensure SPF validation failures occur correctly.

[Dominic (triatic)]

Hi Mitchell,

Your version works for me as well.

In the interim I have been successfully using the following configuration. Would this be potentially useful, or are there issues with removing ${quote:} from --ip and --identity?

    condition = ${run{/usr/bin/spfquery.mail-spf-perl \
                   --ip $sender_host_address \
                   --scope mfrom \
                   --identity $sender_address} \
                   {no}{${if eq {$runrc}{1}{yes}{no}}}}

[Mitchell Dzurick (mitchdz)]

Nice catch, I like that updated version better. This also leads more credence to an issue with parsing.

[Mitchell Dzurick (mitchdz)]

With this said, I have reservations about removing a quote macro without research. It does more than just quote wrap a variable, it also escapes quotes and other special characters from the variable

To be honest, I'm no expert in this source code, but if the variables $sender_host_address or $sender_address come from metadata that the user controls, they can create a malicious payload and achieve ACE in a subprocess.

This might not be an issue at all, and the variable could be properly sanitized by this point, but it's a concern I'd like to investigate before removing it.

[Dominic (triatic)]

I agree, I also do not know the potential risks of removing ${quote:} from tainted variables, so this needs further research before implementation. Or we go with yours.

[Mitchell Dzurick (mitchdz)]

I'll do a bit of git archaeology tomorrow to try and find out why the quote macros were added. Otherwise, my vote is to go with the bash invocation in the sub-process so that way we can keep the quote macros.

[Dominic (triatic)]

Thinking more about this, wouldn't invoking /bin/bash actually increase the attack potential, by allowing for backticks and $() to execute via user-supplied data?

Also, it's not clear to me that ${quote} escapes backticks or $().

[Mitchell Dzurick (mitchdz)]

I was also thinking about similar things.

FYI - I did find out why ${quote:} is being used - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=697057

[Mitchell Dzurick (mitchdz)]

I have a few leads I want to investigate, will report back once I do some testing.

[Dominic (triatic)]

Good find, I am still running my ${quote:}-less version at the moment so I was able to test the scenarios in that bug report.

All three of the following senders were correctly rejected as spf fail by my server (example.com hard fails all senders in its spf record):

MAIL FROM: "$<email address hidden>
MAIL FROM: "x --help "@example.com
MAIL FROM: <email address hidden>

RCPT TO: <snip>
550-[SPF] <snip> is not allowed to send mail
550 from example.com.

So whatever the reason it was allowing an spf fail to pass back then, it no longer appears to be the case now.

[Mitchell Dzurick (mitchdz)]

Thanks for doing some testing around the quotes, was that with using /usr/bin/bash?

Also the lead I had was the changelog entry from upstream:
JH/24 The ${run} expansion item now expands its command string elements after
      splitting. Previously it was before; the new ordering makes handling
      zero-length arguments simpler. The old ordering can be obtained by
      appending a new option "preexpand", after a comma, to the "run".

So I tried using ,preexpand to hopefully get it to work and unfortunately just see a parsing error now, similar to [0].

[0] - https://www.mail-archive.com/search?<email address hidden>&q=subject:%22%5C%5Bexim%5C%5D+Failed+to+expand+ACL+string%22&o=newest&f=1

[Dominic (triatic)]

No, I was not invoking bash, I was using the code I quoted previously and testing the scenarios at port 25.

An empty $sender_address variable would create a missing argument and likely break spfquery.mail-spf-perl, I haven't tested that scenario yet. $sender_host_address should always be present and sanitised by definition.

    condition = ${run{/usr/bin/spfquery.mail-spf-perl \
                   --ip $sender_host_address \
                   --scope mfrom \
                   --identity $sender_address} \
                   {no}{${if eq {$runrc}{1}{yes}{no}}}}

[Mitchell Dzurick (mitchdz)]

Thanks for clarifying, I misread what you meant, now I understand.

I was looking through https://github.com/Exim/exim/blob/master/doc/doc-txt/ChangeLog to see if there were any other changes between version 4.95 and 4.96 and I don't see much else other than JH/24.

Currently I think that [0] and [1] broke something in how we are using it.

It looks like [1] attempts to fix the run{ } parsing, even going as far to add

${run{DIR/aux-fixed/0002.runfile ${quote:1}}{$value}{2}}

as a testcase, which is supposed to test using quotes in the run{} command, but of course isn't doing variable substitution within the quote and is not as complicated.

Unfortunately right now I'm not sure exactly what to test, I'm going to dig into the actual source changes in [0] and [1] to see if there is an implementation detail that isn't caught in the docs.

[0] - https://github.com/Exim/exim/commit/cfe6acff2ddc7eb03b3489770219edf829abd323
[1] - https://github.com/Exim/exim/commit/44b6e099b76f403a55e77650821f8a69e9d2682e

[Mitchell Dzurick (mitchdz)]

One extra thing I tried, is to use the condition check logic Jammy is using by default (which I tested a jammy system does have working SPF), except I added preexpand like [0] mentions to use the "old" parsing method. This didn't work, but just putting my attempt here for future reference:

    condition = ${run,preexpand {/usr/bin/spfquery.mail-spf-perl --ip \
                   ${quote:$sender_host_address} --identity \
                   ${if def:sender_address_domain \
                       {--scope mfrom --identity ${quote:$sender_address}}\
                       {--scope helo --identity ${quote:$sender_helo_name}}}}\
                   {no}{${if eq {$runrc}{1}{yes}{no}}}}

[0] - https://github.com/Exim/exim/commit/cfe6acff2ddc7eb03b3489770219edf829abd323

[Mitchell Dzurick (mitchdz)]

I ran `sudo exim4 -bd -d+all` on my server and sent myself an email with the current config in noble to see what the errors look like, and I get a ton of information, it looks like the parsing worked correctly at least (argv[6]).

23:35:50.263 410533 direct command:
23:35:50.263 410533 argv[0] = '/usr/bin/spfquery.mail-spf-perl'
23:35:50.263 410533 argv[1] = '--ip'
23:35:50.263 410533 argv[2] = '${quote:$sender_host_address}'
23:35:50.263 410533 argv[3] = '--scope'
23:35:50.263 410533 argv[4] = 'mfrom'
23:35:50.263 410533 argv[5] = '--identity'
23:35:50.263 410533 argv[6] = '${quote:$sender_address}'
23:35:50.263 410533 direct command after expansion:
23:35:50.263 410533 argv[0] = '/usr/bin/spfquery.mail-spf-perl'
23:35:50.263 410533 argv[1] = '--ip'
23:35:50.263 410533 argv[2] = '185.125.188.123'
23:35:50.263 410533 argv[3] = '--scope'
23:35:50.263 410533 argv[4] = 'mfrom'
23:35:50.263 410533 argv[5] = '--identity'
23:35:50.263 410533 argv[6] = '"<email>@canonical.com"'
23:35:50.263 410533 (tainted)
23:35:50.263 410533 daemon-accept forking for expand-run
23:35:50.265 410533 daemon-accept forked for expand-run: 410540

The unfortunate thing is no real information is shown in expand-run which is what I really care about.

It'd be nice to see what was ran in that forked process

[Dominic (triatic)]

It was floated before but it's worth raising again. What are the chances of bringing libspf2 to main and falling back into line with Debian?

[Mitchell Dzurick (mitchdz)]

I'll ask internally again, but the original reasoning why we deviated from debian is explained at https://bugs.launchpad.net/bugs/1952738

It is definitely open to discussion, and these woes with the parser can help justify it.

[Dominic (triatic)]

Thank you, I would appreciate you reiterating the case for libspf2.

Aside from the fact that this implementation is looking (recurrently) fragile, this Ubuntu delta, disabled by default, is probably not getting enough attention to spot bugs. This bug has been open for three months and only one other person said "me too".

We can't say if this is because few people enable it, or whether a lot of people are yet to notice everything is "Received-SPF: none", but it's a concern in itself.

[Mitchell Dzurick (mitchdz)]

I agree. I think we may have a case to do libspf2 inclusion, I'll be happy to start investigating that package and situation a bit deeper next week.

For now, the default SPF code in mantic/noble is broken out of the box and I'm not sure which solution (calling bash, removing the quote macro) I like better. They both work, but I have reservations on the security implications that I don't believe I have the bandwidth right now to properly address.

[Dominic (triatic)]

Option three would be to remove the spf section completely until a non-broken sustainable solution is implemented, especially if libspf2 is going to take some time to implement.

[Bryce Harrington (bryce)]

Given that we're at LTS+1, this would seem to be a good point for a re-think on this piece of delta. I also think Dominic makes good points in comments #43 and #45, and the course of action seems logical to me as well. The delta's been proving time-intensive to maintain, and changes upstream not in our control are limiting what auto-detect functionality can be done.

It may be helpful (and perhaps more maintainable), to improve the Exim4 docs in the Ubuntu Server Guide to describe how to do the SPF configuration manually. Here's the current docs:

    https://discourse.ubuntu.com/t/install-and-configure-exim4/11530

We should also ensure this gains an entry in the Release Notes for Oracular, whatever approach is taken.

[Dominic (triatic)]

Postfix already has a configuration page for SPF [0], albeit requiring a package from universe. So although removing the SPF snippet from Exim could be an option in the short-term, I do think in the long-term Ubuntu's SPF support for Exim should be somewhat equivalent to that of Postfix's.

Adding a fragile snippet to a docs would come with the same caveats as including a fragile snippet in the stock Exim config, just with less culpability.

Ultimately using Exim's native SPF facility, i.e. that of libspf2, is the right approach in my mind.

[0] - https://help.ubuntu.com/community/Postfix/SPF