DevHeads.net

Removing trace records on submission MSA

Hi,

I have a question in regards to removing some trace records when providing submission on Postfix 3.1.x and later.

While reading RFC 6409 (“Message Submission for Mail”), I note that the RFC observes that:

"Even when submitted messages are complete, local site policy may dictate that the message text be
examined or modified in some way, e.g., to conceal local name or address spaces.”

By this I take it that I could remove perhaps the initial trace message that returns information about internal addresses
and network names. It seems to me that both Hotmail/Outlook and Gmail do this.

Is this acceptable ? The only bad side to it would appear to be possibly some increased difficulty in troubleshooting.

If it is an acceptable process, how would I configure Postfix to do this only on submission ?

Thanks,

- J

Comments

Re: Removing trace records on submission MSA

By Philip Paeps at 05/01/2018 - 05:02

On 2018-03-10 22:01:01 (+0100), J Doe wrote:
Apologies for resurrecting an old thread.

I had some time to kill yesterday and I came up with this PCRE monster:

/^Received:.*([\n]).*sender: (.+?)\).*(by.+?)\).*(id \w+).*(;.*)/
REPLACE Received: ${3}, authenticated sender ${2})${1} ${4}${5}

It's a bit hairy but it makes the Received: header of a submission user
look a lot like the Received: header added by local delivery:

Received: by rincewind.trouble.is (Postfix, authenticated sender
philip)
id XXXXXXXXX; Tue, 1 May 2018 09:56:20 +0000 (UTC)

I wonder if it wouldn't be easier to add a configuration option to smtpd
to suitably expurgate Received: headers of sensitive information.

This is working for me though. It's ugly but it seems to work for all
my users and the exotic devices they use.

Philip

Re: Removing trace records on submission MSA

By LuKreme at 05/02/2018 - 13:52

On 2018-05-01 (04:02 MDT), Philip Paeps < ... at trouble dot is> wrote:
What information in the Received header do you consider sensitive?

Re: Removing trace records on submission MSA

By Philip Paeps at 05/02/2018 - 15:57

On 2018-05-02 20:52:46 (+0200), @lbutlr wrote:
When it comes in over submission from authenticated users, I consider
the HELO hostname, the IP address and the reverse lookup of the IP
address sensitive. Those data allow the user to be tracked around the
internet based on where they send email from.

The queue id, the date and the sasl username are sufficient trace
information to grep in logfiles if something needs to be debugged.

Note that I'm only talking about submission. The trace headers added on
mail being relayed are perfectly fine.

I'm not sure if there's a tidy way to implement this as an option. The
hairy header_checks hack also "just works". My mind just rebels against
something so conceptually simple requiring such a crazy regular
expresion. :)

Philip

Re: Removing trace records on submission MSA

By LuKreme at 05/03/2018 - 11:29

On 2018-05-02 (14:57 MDT), Philip Paeps < ... at trouble dot is> wrote:

Hmm. OK, I do not see my home IP address in any mail header (and yes, I use submission and that is the only way to send mail from my server), nor do i see the helo hostname.

These are the received headers from a message I sent to an iCloud account I have:

Received: from mr28p00im-smtpin039.mac.com ([17.110.71.38])
by ms20521.mac.com (Oracle Communications Messaging Server 8.0.1.3.20170906
64bit (built Sep 6 2017)) with ESMTP id < ... at ms20521 dot mac.com>
for *user*@icloud.com; Thu, 03 May 2018 16:06:10 +0000 (GMT)
Original-recipient: rfc822;*user*@icloud.com
Received: from mail.covisp.net (<a href="http://www.covisp.net" title="www.covisp.net">www.covisp.net</a> [65.121.55.45])
by mr28p00im-smtpin039.me.com
(Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7
2017)) with ESMTPS id <0P8500EVTU27DY10@mr28p00im-smtpin039.me.com> for
*user*@icloud.com (ORCPT *user*@icloud.com); Thu, 03 May 2018 16:06:10 +0000 (GMT)
Received-SPF: pass (mr21p00im-spfmilter008.me.com: domain of <a href="mailto: ... at kreme dot com"> ... at kreme dot com</a>
designates 65.121.55.45 as permitted sender)
receiver=mr21p00im-spfmilter008.me.com; client-ip=65.121.55.45;
helo=mail.covisp.net; envelope-from= ... at kreme dot com

I also tried it with a local domain to another local domain, sending to a gmail address, and sending from an iCloud account to gmail (don't think this would involve postfix at all). In none of these where my home IP or helo name or user login name included in the headers.

Now I wonder why?

Re: Removing trace records on submission MSA

By Wietse Venema at 05/02/2018 - 18:30

Philip Paeps:
Instead of saying what to remove from headers, it would be more
natural to say what should be in headers. Configurable headers (not
just From: or Recived:) are on the wishlist for as long as Postfix
exists.

Wietse

Re: Removing trace records on submission MSA

By Philip Paeps at 03/12/2018 - 01:10

On 2018-03-10 16:01:01 (-0500), J Doe wrote:
I anonymise the initial Received: header with a header_checks on the
submission service.

In master.cf, I add `-o cleanup_service_name=subcleanup` to the
submission service. That service is defined as:

subcleanup unix n - n - 0 cleanup
-o syslog_name=postfix/subcleanup
-o header_checks=pcre:$config_directory/submission_header_checks.pcre

The submission_header_checks.pcre file contains:

/^\s*(Received: from .+?(?=\s\())[^\n]*(.*for <.*)/ REPLACE $1 (localhost [127.0.0.1])$2

I'm sure there are better ways to do this, but this works for me.

It doesn't interfere with debugging much because the logs will mentain
the replacement and it's easy to grep for.

Philip

Re: Removing trace records on submission MSA

By J Doe at 04/04/2018 - 19:54

Hi Phillip,

Thank you for your reply.

I currently use DKIM and as per the RFC for DKIM, I don’t include trace headers in the message hash that makes up the DKIM signature. I am under the impression that my DKIM signatures should be correct in this case if I use your solution and it re-writes the first trace header - is that true or are there any other DKIM issues I might run into ?

Thanks,

- J

Re: Removing trace records on submission MSA

By Philip Paeps at 04/04/2018 - 21:49

Unless you have specifically configured your DKIM setup to include trace
headers in the hash (which you should not do according to the RFC), your
DKIM signatures will continue to be correct if you anonymise the first
trace header like I do.

Philip

Re: Removing trace records on submission MSA

By J Doe at 04/06/2018 - 15:27

Hi Philip,

Thank you for your reply.

I configured master.cf and created the regular expression lookup table, but my installation of Postfix (3.1.0), does not appear to support PCRE as placing “pcre:” as the dictionary type in master.cf generated an error that "this dictionary type is unsupported".

Some Googling revealed that I may be able to install support for that, but rather than install something else I switched to “regexp:”. Unfortunately, regexp stated there was an error in the regular expression string (the error indicated the line but not the character in the regexp that it did not like).

My regular expression skills are rusty, so I went with an unoptimized search string:

/etc/postifx/submission_header_rules

/Received: from/ REPLACE Received: from [127.0.0.1] (localhost [127.0.0.1]) by server.com <http://server.com/>

… where server.com <http://server.com/> is the FQDN for my mail server.

As I have this configured for submission, I then tested sending e-mail to Gmail and can confirm that my DKIM is still valid (as expected - I don’t include Received: headers in the DKIM hash, as the DKIM RFC recommends), and this is not doing any unwanted edits on mail over port 25.

I figured this was sufficient but further reading indicates that some anti spam software pays attention to the Received: headers (although most sources noted this was an issue when configuring Postfix to *DELETE* the first header, which I don’t want to do).

With that in mind, I had two questions:

** Is there any anti spam software that checks for the date and time at the end of the Received: string ? My very basic search string does not capture the date and time after the semicolon and therefore does not show up.

** If there is anti spam software that looks for the date and time, could you help me construct a “regexp:” compatible search string ? I experimented with captures but again, my regular expression skills are bad at the moment.

Thanks,

- J

Re: Removing trace records on submission MSA

By Karol Augustin at 04/06/2018 - 16:01

On 06/04/2018 09:27 PM, J Doe wrote:
/^(Received:) from.*]\).*(.{2}by mail\.nimitz\.pl.*Postfix.*) (with
[E]{0,1}SMTP[S]{0,1}[A]{0,1}) (.*)/ REPLACE $1 from mail.nimitz.pl
(localhost [127.0.0.1])$2 with SMTP $4

Just change 'mail.nimitz.pl' with FQDN of your server. This expression
works for me and also removes information about the connection, which in
my case can tell if the mail was sent from webmail (unencrypted
connection from webmail host to postfix host) or client's MUA
(encrypted).

It can probably fail on some systems due to .* matching, which is
greedy, but I wrote it many years ago and it works, so I am not fixing
it.

k.

Re: Removing trace records on submission MSA

By J Doe at 04/06/2018 - 23:34

Hi Karol,

Thanks for this. I’m looking to mask out the DDNS name of a xDSL connection.

I tried the following with a visual regex program (to make checking captures easier):

/etc/postfix/submission_privacy_header
/(Received\:\s*from)[^\;]+(\;\s[A-Z]{1}[a-z]{2,3}\,)\s+(\d{1,2}[^\n]+)/ REPLACE $1 [127.0.0.1] (localhost [127.0.0.1]) by myserver.com$2 $3

…however this does not match when Postfix evaluates it. I am attempting three captures:

1. Received: from
2. The first part of the date from the ; to the , after the day (eg: “Fri”)
3. The last part of the date from the numerical day number to the end

The reason for the two part handling of the date is I want to strip out whitespace between “Day, 6 Apr . . .”. If I don’t strip this out it puts a line break inline in the Received: header that breaks the date over two lines.

Does anyone know what I’m doing wrong and/or is there a way to make Postfix provide more debug output for a regexp: operation ?

Thanks,

- J

Re: Removing trace records on submission MSA

By Viktor Dukhovni at 04/06/2018 - 23:36

That's PCRE syntax.

You're using a "regexp" table, those don't support PCRE.

Re: Removing trace records on submission MSA

By J Doe at 04/07/2018 - 00:23

Hi Viktor,

Thanks for your response and that’s definitely the problem I had.

I did some Googling for doing PCRE to POSIX regular expressions and updated the string:

/^(Received:\sfrom)[^;]+(;\s[A-Z]{1}[a-z]{2,3},)\s+([[:digit:]]{1,2}[^\n]+)/ REPLACE $1 [127.0.0.1] (localhost [127.0.0.1]) by myserver.com$2 $3

… and it works!

Not sure why I had it in my head that regexp: was PCRE (especially when there’s another dictionary type that explicitly says pcre:), but at least I’ve learned something.

- J

Re: Removing trace records on submission MSA

By Viktor Dukhovni at 04/07/2018 - 00:26

It can't, the above is still PCRE, "\s" for whitespace is PCRE, not "regexp". Perhaps that's no the string you're using.

My advice is to ditch regexp and use PCRE. Install the package that adds PCRE support.

Re: Removing trace records on submission MSA

By J Doe at 04/07/2018 - 00:34

Hi Viktor,

Hmmm. I just sent a test message via submission to a Gmail account and checked the headers and the replacement works.

According to the site [1] \s is shorthand for POSIX regular expressions.

Perhaps the POSIX regex library compiled with Postfix now supports this ?

Sources:

[1] <a href="https://www.regular-expressions.info/posixbrackets.html" title="https://www.regular-expressions.info/posixbrackets.html">https://www.regular-expressions.info/posixbrackets.html</a>

Re: Removing trace records on submission MSA

By Viktor Dukhovni at 04/07/2018 - 00:50

No "\s" is not a POSIX feature, it is however a GNU extension, so you may have
a GNU regexp library, that supports "\s" (outside bracket expressions):

<a href="https://www.regular-expressions.info/gnu.html" title="https://www.regular-expressions.info/gnu.html">https://www.regular-expressions.info/gnu.html</a>

Re: Removing trace records on submission MSA

By J Doe at 04/07/2018 - 00:59

Hi Viktor,

Ah, interesting - that must be it, then.

This is on an Ubuntu 16.04 LTS server. I can see the dependencies compiled in from Ubuntu’s page [1] and GNU libc is listed. [2] seems to suggest that regular expressions are part of GNU libc.

Is there another way I can confirm that ?

Thanks,

- J

Sources:

[1] <a href="https://packages.ubuntu.com/xenial/postfix" title="https://packages.ubuntu.com/xenial/postfix">https://packages.ubuntu.com/xenial/postfix</a> <https://packages.ubuntu.com/xenial/postfix>
[2] <a href="https://www.gnu.org/software/libc/manual/html_node/Regular-Expressions.html" title="https://www.gnu.org/software/libc/manual/html_node/Regular-Expressions.html">https://www.gnu.org/software/libc/manual/html_node/Regular-Expressions.html</a> <https://www.gnu.org/software/libc/manual/html_node/Regular-Expressions.html>

Re: Removing trace records on submission MSA

By Viktor Dukhovni at 04/07/2018 - 01:04

FreeBSD 11 (POSIX):

$ echo "1 b" | egrep '\d\s\w'
$

MacOS High Sierra (POSIX with GNU or similar extensions):

$ echo "1 b" | egrep '\d\s\w'
1 b
$

Your Ubuntu system most likely will match the MacOS results. Which means that your regexp table is not portable, but happens to work on your system.

Re: Removing trace records on submission MSA

By J Doe at 04/07/2018 - 01:14

Hi Viktor,

I can confirm the expected result for MacOS High Sierra . . . but interestingly enough, I get the FreeBSD 11 results on Ubuntu 16.04 LTS.

Are there any other tests I can try to confirm the existence of GNU extensions to POSIX regular expressions ?

Thanks,

- J

Re: Removing trace records on submission MSA

By J Doe at 04/07/2018 - 01:39

Hi Viktor and Dominic,

If I do the following on Ubuntu 16.04 LTS:

$ echo "1 2" | egrep '[[:digit:]]\s[[:digit:]]’
1 2

… where “1 2” are highlighted in bash

Am I correct that since this POSIX regex for the digits AND the \s is still being interpreted, my system must support the GNU regex extensions ?

Thanks,

- J

Re: Removing trace records on submission MSA

By Dominic Raferd at 04/07/2018 - 01:46

​It is standard Ubuntu it will support GNU regex extensions, but why not
use pcre? It is more powerful, more standardised, and - my impression​ -
more widely used for Postfix tables. Just do:
$ sudo apt-get install postfix-pcre

Re: Removing trace records on submission MSA

By J Doe at 04/07/2018 - 02:05

Hi Viktor and Dominic,

Thank you for your reply.

It occurred to me that I could side-step the issue of GNU extensions and whether they’re supported by converting the string I e-mailed a couple of e-mails back to full POSIX regex (in this case removing the \s). I ended up with:

/^(Received:[[:space:]]from)[^;]+(;[[:space:]][A-Z]{1}[a-z]{2,3},)[[:space:]]+([[:digit:]]{1,2}[^\n]+)/ REPLACE $1 [127.0.0.1] (localhost [127.0.0.1]) by myserver.com$2 $3

…and it works.

Have I missed anything else that needs to be converted so that the regular expression is POSIX only ?

- J

Re: Removing trace records on submission MSA

By Viktor Dukhovni at 04/07/2018 - 12:32

It is now portable POSIX. For the record, in email the allowed whitespace is more narrow than
is recognized by [[:space:]], you're not likely to run into any false positives. The email
header whitespace consists of just SPACE, TAB, CR and LF. VT and FF (vertical tab and form feed)
are not valid whitespace in email headers.

Re: Removing trace records on submission MSA

By J Doe at 04/07/2018 - 15:42

Hi Viktor,

Ok, great! Thank you for those observations about whitespace

Thanks to everyone else on this thread who also provided examples and suggestions. Going forward I will take the advice to install PCRE support.

- J

Re: Removing trace records on submission MSA

By Dominic Raferd at 04/07/2018 - 01:10

On 7 April 2018 at 07:04, Viktor Dukhovni <postfix- ... at dukhovni dot org>
wrote:

On my Ubuntu 16.04.4:
$ echo "1 b" | grep -E '\d\s\w'
$ echo "1 b" | grep -E '[0-9]\s\w'
1 b

But I agree OP should switch to using pcre tables, it is supported under
Ubuntu.