DevHeads.net

smtpd_delay_reject with rspamd milter

I'm having trouble with access_maps kicking in after an upgrade from a
Postfix 2.something to Postfix 3.1. on Ubuntu 14.06 and using postscreen
and rspamd milter.

After some testing I'm not sure yet, but it looks like the recommended
smtpd_delay_reject = yes in connection with having the access_map checks
listed in the smtpd_recipient_restrictions is the cause of this. It allows
the milter to kick in, so that the access_check (almost?) never happens.

After changing to
smtpd_delay_reject = no
and moving the client access_maps to smtpd_client_restrictions they seem
to be working again.

rspamd is used as a milter (as specified in the rspamd docs) and not as a
policy service (which would allow putting it at the end of restrictions).

# rspamd
smtpd_milters = inet:localhost:11332
milter_protocol = 6
milter_mail_macros = i {mail_addr} {client_addr} {client_name}
{auth_authen}
milter_default_action = accept

Is this a known problem? Other workarounds? Can I specify to use the
milter only after the restrictions? (Maybe doesn't make sense?)
Or am I misinterpreting something?

Thanks!

Kai

Comments

Re: smtpd_delay_reject with rspamd milter

By Carsten Rosenberg at 11/07/2018 - 10:23

Kai,

both are running simultaneously. So at smtpd_recipient_restriction stat
the milter will also get the recipients. As far as I have seen the
postfix restriction react faster.

So if you reject somebody with an access_map, you won't see any scan
result in rspamd. Only the milter connect, because rspamd starts to work
at the end_of_data_restrictions.

Do you have any problems with this situation?

Re: smtpd_delay_reject with rspamd milter

By Kai Schaetzl at 11/07/2018 - 11:26

Carsten Rosenberg wrote on Wed, 7 Nov 2018 16:23:54 +0100:

This would be fine ;-)

Yes, it's the other way around here. e.g. there is no rejection happening
by postfix, but the milter kicks in and greylists the mail (if it scores
enough the first time) and after greylisting scans it and scores
accordingly. But I would rather like it to get rejected by postfix because
of the access_map.

I have some generic TLDs listed that deliver only garbage, like .site,
host, .review etc. They were getting scored as spam by rspamd most of the
time but I wondered why they weren't getting rejected by postfix, anyway.
First I thought I might be using wrong syntax (site vs. .site), but I
scanned the postfix docs and found that the default compatibility setting
for access_maps should allow "site" to be used for subdomain matching as
well.
Now, after removing the delay it seems that postfix is now rejecting them.

I'm not 100% sure if that did it, because I have some sender rejects that
*may* have been before my changes. But never a client reject. I'm not sure
because I made several changes over the course of the day and am not sure
about exact times.

So, this seems to work now, but I've just realized I hit a new problem.
After smtpd_delay_reject = no the option permit_sasl_authenticated doesn't
work in permit_sasl_authenticated anymore. I had to revert to yes,
otherwise the checks *after* permit_sasl_authenticated hit the message and
reject it. After thinking about this, it's clear that if I check at helo
stage there hasn't been any authentication yet, permit_sasl_authenticated
is moot at this stage. If I want it and still use some rejections because
of helo I *have* to delay.

Is there a workaround for this which allows client and sender rejections
and have the milter kick in only after this?
Here's my current conf in this area:
(smtpd_client_restrictions was empty before today and most of the
restrictions had been in recipient_restrictions)

smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_relay_restrictions = permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination
smtpd_helo_restrictions =
permit_sasl_authenticated,(obviously in vain)
#permit_mynetworks,
reject_invalid_helo_hostname,
reject_non_fqdn_helo_hostname,
reject_unknown_helo_hostname,
check_helo_access hash:/etc/mail/access,
check_helo_access hash:/etc/mail/disallow_my_domains,
permit
#http://www.postfix.org/postconf.5.html#smtpd_relay_restrictions:
smtpd_client_restrictions =
permit_sasl_authenticated,
permit_mynetworks,
sleep 1,
reject_unauth_pipelining,
check_client_access hash:/etc/mail/allow_clients,
check_client_access hash:/etc/mail/access,
reject_invalid_hostname,
reject_unknown_client_hostname,
permit
smtpd_sender_restrictions =
smtpd_recipient_restrictions =
reject_non_fqdn_sender,
reject_non_fqdn_recipient,
permit_sasl_authenticated,
permit_mynetworks,
reject_unauth_destination,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
reject_unlisted_recipient,
check_recipient_access hash:/etc/mail/allow_recipients,
check_sender_access hash:/etc/mail/allow_senders,
#check_client_access hash:/etc/mail/allow_clients,
#check_client_access hash:/etc/mail/access,
check_sender_access hash:/etc/mail/access,
#reject_invalid_hostname,
#reject_unknown_client_hostname,
#reject_rbl_client ix.dnsbl.manitu.net,
#check_policy_service inet:127.0.0.1:10023,
check_policy_service inet:127.0.0.1:10024,
permit
smtpd_data_restrictions = reject_unauth_pipelining,
reject_multi_recipient_bounce

Kai

Re: smtpd_delay_reject with rspamd milter

By Kai Schaetzl at 11/07/2018 - 12:01

Addendum.

Currently, I get client rejections with the setup shown in my last mail
(despite the delay). I don't know if it hits *always*, though. I can't
check if it didn't hit for some client where the name matches, there are
too many entries.

I expected it to carry out the helo checks before client checks.
e.g. in a "natural" order of helo, client, sender, rcpt.
Was this assumption wrong?

Example:

Nov 7 17:35:53 b04 postfix/smtpd[30957]: NOQUEUE: reject: RCPT from
com.check-prfofessional.online[185.52.2.216]: 554 5.7.1 <com.check-
prfofessional.online[185.52.2.216]>: Client host rejected: Access denied;
from=< ... at com dot check-prfofessional.online> to=< ... at example dot com>
proto=ESMTP helo=<com.check-prfofessional.online>

The helo contains the same name, so a helo check should have hit it. Which
means, helo checks are done after the client check?

Kai

Re: smtpd_delay_reject with rspamd milter

By Wietse Venema at 11/07/2018 - 12:10

Kai Schaetzl:
[ Charset ISO-8859-1 converted... ]
The order of evaluation is in

<a href="http://www.postfix.org/SMTPD_ACCESS_README.html" title="http://www.postfix.org/SMTPD_ACCESS_README.html">http://www.postfix.org/SMTPD_ACCESS_README.html</a>

Postfix evaluates client restrictions before helo_restrictions
before sender restrictions before recipient restrictions.

HOWEVER, by default Postfix evaluates all of these at RCPT TO time.

Wietse

Re: smtpd_delay_reject with rspamd milter

By Kai Schaetzl at 11/07/2018 - 13:40

Wietse Venema wrote on Wed, 7 Nov 2018 12:10:40 -0500 (EST):

which means smtpd_delay_reject = yes is the default?

Am I correct in assuming that with "yes" it doesn't matter if I list the
client restrictions in smtpd_client_restrictions or in
smtpd_recipient_restrictions?
If so, does the order matter?
I mean it should matter in general, but if I mix different stages like
shown in my earlier mail like the following, is it still getting evaluated
in this order or getting reordered? See below for an exception I saw.

smtpd_recipient_restrictions =
reject_non_fqdn_sender,
reject_non_fqdn_recipient,
permit_sasl_authenticated,
permit_mynetworks,
reject_unauth_destination,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
reject_unlisted_recipient,
check_recipient_access hash:/etc/mail/allow_recipients,
check_sender_access hash:/etc/mail/allow_senders,
check_client_access hash:/etc/mail/allow_clients,
check_client_access hash:/etc/mail/access,
check_sender_access hash:/etc/mail/access,
and some more.

I'm asking because I've seen rejections by sender earlier, although
client_access should have hit first. An example:

Nov 7 14:15:24 b04 postfix/smtpd[6584]: NOQUEUE: reject: RCPT from
mx17.a.outbound.createsend.com[203.55.21.17]: 554 5.7.1
< ... at cmail20 dot com>: Sender address rejected: Access denied;
from=< ... at cmail20 dot com> to=< ... at example dot com> proto=ESMTP
helo=<mx17.a.outbound.createsend.com>

with this in /etc/mail/access
createsend.com REJECT
cmail20.com REJECT
and the exact same order of maps as above.

Shouldn't the client restriction have kicked in here instead of sender?

Thanks,

Kai

Re: smtpd_delay_reject with rspamd milter

By Noel Jones at 11/07/2018 - 14:30

On 11/7/2018 12:40 PM, Kai Schaetzl wrote:
Yes, that's the default, and generally should not be changed.

Postfix always evaluates the smtpd_*_restrictions statements in the
documented order; they are never reordered. Always
client-helo-sender-recipient. This evaluation is by default delayed
until the client sends the first recipient, but the order stays the
same.

Within each smtpd_*_restrictions section, the restrictions are
checked in the order YOU specify.

This will evaluate in exactly the order you have listed above. They
are never reordered.

With the above list, check_sender_access comes first. Postfix does
not reorder the list you have specified.

No, they are executed in the order you specify.

-- Noel Jones

Re: smtpd_delay_reject with rspamd milter

By Kai Schaetzl at 11/07/2018 - 19:31

Noel Jones wrote on Wed, 7 Nov 2018 13:30:08 -0600:

Thanks for the answer. But, please look again.

/etc/mail/access:
createsend.com REJECT
cmail20.com REJECT

The order is:

Kai

Re: smtpd_delay_reject with rspamd milter

By Matus UHLAR - f... at 11/08/2018 - 06:06

On 08.11.18 01:31, Kai Schaetzl wrote:
you should specify .createsend.com, because the connecting domain is
mx17.a.outbound.createsend.com, not createsend.com. You apparently did not
set append_dot_mydomain=yes (don't!).

Nov 7 14:15:24 b04 postfix/smtpd[6584]: NOQUEUE: reject: RCPT from
mx17.a.outbound.createsend.com[203.55.21.17]: 554 5.7.1
< ... at cmail20 dot com>: Sender address rejected: Access denied;
from=< ... at cmail20 dot com> to=< ... at example dot com> proto=ESMTP
helo=<mx17.a.outbound.createsend.com>

Re: smtpd_delay_reject with rspamd milter

By Matus UHLAR - f... at 11/08/2018 - 06:12

On 08.11.18 12:06, Matus UHLAR - fantomas wrote:
sorry, short brain outage.
the problem lies in "parent_domain_matches_subdomains" which is (and should
be) empty in postfix and apparently even is in new postfix version.

Re: smtpd_delay_reject with rspamd milter

By Kai Schaetzl at 11/08/2018 - 06:58

Matus UHLAR - fantomas wrote on Thu, 8 Nov 2018 12:12:49 +0100:

As I wrote earlier, it *is* set implicitely by backwards-compatibilty.

postconf -d|grep parent_domain_matches_subdomains
parent_domain_matches_subdomains =
debug_peer_list,fast_flush_domains,mynetworks,permit_mx_backup_networks,qmqpd
_authorized_clients,relay_domains,smtpd_access_maps

At the moment it seems to be working as intended. But I can't see a config
change vs. yesterday morning that could have made a change with this. The
only change I made is stop the delay (temporarily) and move the
check_client_access from recipient_restrictions to client_restrictions (and
not move them back after delaying the checks again).
I'll see if I can come up with more clues/reproduction next week. At the
moment I'm happy that it *seems* to be working, although the order is now not
what I wanted.

Kai