Design: sender-dependent default_transport

By Wietse Venema at 11/21/2009 - 22:11

Recently there have been requests for sending mail with source IP
addresses that depend on the envelope sender. Sometimes the request
appeared to be related to showshoe spamming, and sometimes it
appeared to be a legitimate attempt to protect IP-based domain
reputations of different customers.

Current solution
================
The current solution that Postfix offers is to use multiple instances:
one back-end instance per source IP address, and one front-end
instance that uses sender_dependent_relayhost_maps to choose the
right back-end instance. That is a "heavy" solution, even though
Postfix 2.6 multi-instance support hides most of the complexity.

Past solutions
==============
Attempts to modify existing features for this job make Postfix
harder to explain, or have negative performance impact for content
inspection as with a recent proposal to change the meaning of a
FILTER actions with an empty destination (this would replace FIFO
selection by domain-based round-robin selection).

Going back in time, the old Postfix sender_dependent_routing feature
was withdrawn a few years ago because it replaced ALL mail routing
decisions by sender-based routing. That broke deliveries to local
recipients, and was therefore not good for a general-purpose MTA.

A promising solution
====================
The solution is not to make ALL routing decisions dependent on the
sender address, but ONLY the routing decisions for mail that leaves
the machine.

A sender-dependent default_transport would change the meaning of
default_transport (namely, giving default_transport a lower precedence
than sender_dependent_default_transport_maps) and introduces a few
new parameters.

(It looks like a sender-dependent default_transport would do the
job. It does not seem to make sense to make relay_transport also
sender-dependent, because relay_transport is supposed to be used
only for domains Postfix is MX host for.)

The draft design looks like this:

default_transport (default: smtp)
The default mail delivery transport and next-hop destination for desti-
nations that do not match $mydestination, $inet_interfaces,
$proxy_interfaces, $virtual_alias_domains, $virtual_mailbox_domains, or
$relay_domains. In order of decreasing precedence, the nexthop desti-
nation is taken from $default_transport, $sender_dependent_relay-
host_maps, $relayhost, or from the recipient domain. This information
can be overruled with the sender_dependent_default_transport_maps
parameter and with the transport(5) table.

Specify a string of the form transport:nexthop, where transport is the
name of a mail delivery transport defined in master.cf. The :nexthop
part is optional. For more details see the transport(5) manual page.

Example:

default_transport = uucp:relayhostname

sender_dependent_default_transport_maps (default: empty)
A sender-dependent override for the global default_transport parameter
setting. The tables are searched by the envelope sender address and
@domain. A lookup result of DUNNO terminates the search without over-
riding the global default_transport parameter setting. This informa-
tion is overruled with the transport(5) table.

For safety reasons, this feature does not allow $number substitutions
in regular expression maps.

This feature is available in Postfix 2.7 and later.

empty_address_default_transport_maps_lookup_key (default: <>)
The sender_dependent_default_transport_maps search string that will be
used instead of the null sender address.

This feature is available in Postfix 2.7 and later.

address_verify_sender_dependent_default_transport_maps (default: empty)
Overrides the sender_dependent_default_transport_maps parameter setting
for address verification probes.

This feature is available in Postfix 2.7 and later.

A first stab at some code looks promising. I'll run tests as time
is available, but I am currently busy with reviewing research
proposals, and that will take most cycles until early December.

Wietse

5 comments

Re: Design: sender-dependent default_transport

By Victor Duchovni at 11/21/2009 - 23:54

Is "DUNNO" the right choice here? This is not an access(5) table, and
"DUNNO" may muddy the user's understanding or expectations? If we really
want to allow one to make exceptions for specific users or sub-domains
without explicitly specifying the default transport, I think that:

except.example.com DEFAULT
example.com transport:nexthop

or perhaps

except.example.com :
example.com transport:nexthop

is even intuitive, with the latter choice matching similar behaviour
in transport(5).

Re: Design: sender-dependent default_transport

By Noel Jones at 11/22/2009 - 00:08

No, adding a new magic result is not the solution.

I like this. +1 for ":"

-- Noel Jones

Re: Design: sender-dependent default_transport

By Wietse Venema at 11/22/2009 - 11:42

Noel Jones:

The sender-dependent default_transport user interface is a clone
of sender_dependent_relayhost_maps. I think that similar mechanisms
should have similar user interfaces where possible (this is a human
interface issue, flattening the learning curve by reusing mental
concepts).

When you compare sender-dependent default_transport versus relayhost
then you will notice that there are some differences:

- sender-dependent relayhost overrides relayhost, which affects
both the "default" address class and the "relay" address class.
This is desirable because sender-dependent relayhost is intended
for people who must send non-local mail through a specific ISP.

- sender-dependent default_transport overrides only default_transport,
meaning this affects the "default" address class but not the
"relay" address class.

While desirable, such differences may not be immediately obvious.

Finally, this is not the only way source IP address selection can
be implemented. My current list of options is:

1) Sender-dependent default_transport override as discussed above.

2) Two-level routing. First, the recipient address class is chosen
(result: local, virtual, relay, default). Then we extend the
address class concept, such that each address class may override
its own per-class transport with a sender-dependent lookup table.
This way, local recipients will be given to the local (or virtual)
address class, and will not be affected by the default class's
sender-dependent transport lookups. And once we have "default
class" transport overrides, we get "relay class" transport
overrides for free.

Like 1), feature 2) builds on the existing address class
infrastructure. On the down side, one has to understand the
subtle precedence order of selection mechanisms. While an override
is easy enough to add to Postfix, the approach does not scale well
if we want to introduce other lookup mechanisms (transport
selection by client name/address, client helo, sasl login, or
some other property such as the message size).

If we really want fully-generalized routing then we should
provide a transport policy protocol plus a proof-of-concept
implementation, similar to the SMTPD policy protocol. It is not
obvious that this would be used in the real world. Lookup tables
sacrifice functionality for better simplicity and robustness.

3) New access/header_checks/body_checks actions that set an SMTPSOURCE
attribute and that is used only by the SMTP client. This decouples
source IP address selection from the sender envelope address,
and requires no transport selection.

Feature 3) while it appears straightforward, it also completely
ignores the existing infrastructure of transport and nexthop
selection. It has the appeal of a short-sighted^h^h^h^h^h^h^hterm
solution and needs more analysis to understand its implications.

Wietse

Re: Design: sender-dependent default_transport

By Rainer Frey (In... at 11/25/2009 - 06:12

Hi Wietse,

thanks for addressing this issue, a scaling solution for this problem will be
really appreciated.

We are an E-Mail Marketing provider and offer a hosted solution for a broad
customer base for which we want to be able separate the reputation (so
individual customers can take part in reputation systems for content
originators as opposed to sending servers, and those often require dedicated
IP addresses).

[...]

As a user with limited knowledge of the internal infrastructure of Postfix, I
like this option best. For my understanding, outgoing SMTP configuration
parameters and mail routing are logically separate concerns, and it is easier
to understand if these are configured separately.

More important, we need to tune other parameters (destination concurrency,
timeouts, ...) based on the recipient domain. So only a way of combining
sender and recipient specific configuration solves our problem completely.
This option 3) seems to offer the greatest flexibility in this regard.

Which, as said above, can be seen as a good thing from a user perspective.

One thing that comes to my mind: there are anti-spam measures that want the
reverse lookup of the sending IP address to correspond with the SMTP HELO
hostname. I know that this is not required by any standards, but some
providers use such rules to at least give bad score to non-complying senders.
An example is UCEProtect, which is used quite often in some parts of Germany.
Therefore it would be necessary (or at least desirable) to not only select
the source address, but also the HELO name.

Rainer

Re: Design: sender-dependent default_transport

By Wietse Venema at 11/25/2009 - 12:06

Rainer Frey (Inxmail GmbH):

I notice that you mention other SMTP client settings (HELO name)
that would also need to vary in addition to the source IP address.
And when there are two now, then there will be more later.

This is a good reason for me to avoid introducing a whole new set
of override mechanisms that duplicate the controls that already
exist (override local IP address, override HELO hostname, and the
things that other people will need in the future).

Right now, Postfix provides an override mechanism via master.cf
settings that give access to all the main.cf parameters that control
that particular process. The downside is that this works at the
granularity of transports, but the benefit is that all the parameters
are already implemented (it reuses both code and user experience).

I don't really buy that you need a global scheduler to balance load
that's being sent from different IP addresses to the same sites.
Such balancing is also not happening between independent MTAs. As
long as Postfix does not worse than a bunch of independent MTAs
then it is good enough.

The problem with a general-purpose system like Postfix is that it
can never be optimal for every possible scenario. Optimize too
much in one direction and I end up paying a large price later
when some functionality needs to be added. If there are a bazillion
ways to do X then I have to make the new functionality work with
all those ways, because software with exceptions is a source of
frustration.

Wietse