DevHeads.net

Best practice - concurrent delivery to remote sites

Hi,

we're running a small relay-service and looking for best practice to
deliver mails to remote sites regarding concurrent delivery and so on.

Sometimes, we have customers that are sending several mails per second to
same recipients.

What is best practice to handle this?

We would like to avoid getting blacklisted or throttled by remote sites due
to sending too many mails or in an non compliant way. How should this be
handled/configured in postfix?

so far all settings are default in postfix.

thank you.

Comments

Re: Best practice - concurrent delivery to remote sites

By Andrey Repin at 12/06/2018 - 06:40

Greetings, Stefan Bauer!

This has nothing to do with postfix itself.
Social issues can't be solved by technical means.

Re: Best practice - concurrent delivery to remote sites

By Stefan Bauer at 12/06/2018 - 06:54

Its no user issue. Its a real and legal use case that customers send
several mails / second to same recipient over a long period (software tests
whatever).

Am Do., 6. Dez. 2018 um 12:50 Uhr schrieb Andrey Repin < ... at yandex dot ru

Re: Best practice - concurrent delivery to remote sites

By Andrey Repin at 12/06/2018 - 07:03

Greetings, Stefan Bauer!

Did I say anything about reality or legality?
The decision of remote host to block you is 100% social, not technical or
legal.
How do they judge you is entirely up to them, as long as you conform to
standards, you can't do anything short of communicating with the owners and
solving any arising issues as they happen.

Re: Best practice - concurrent delivery to remote sites

By Stefan Bauer at 12/06/2018 - 07:19

ack. but i was looking for advices like e.g:

initially defer mail delivery for lets say a minute to be able to send out
a bunch of mails to same recipient in a single session instead of having
100 independant sessions.

stuff/best practice that makes the process more effective.

i'm certain that remote sites prefer one way over the other.

Stefan

Am Donnerstag, 6. Dezember 2018 schrieb Andrey Repin :

Re: Best practice - concurrent delivery to remote sites

By Wietse Venema at 12/06/2018 - 08:41

Stefan Bauer:
I don't think that there is a 'standard' policy that 'works' for
delivery from every site to every site.

Nowadays you get a policy exception from 'big' receivers, and you
come up with transport_maps with different 'classes' of delivery
agents that are configured with different rate_delay (no concurrency),
with limited concurrency, and/or with different source IP address,
then pick the agent depending on destination.

Or you just pay a mail sending company for doing the job.

Wietse

Re: Best practice - concurrent delivery to remote sites

By Stefan Bauer at 12/06/2018 - 09:05

Thank you Wietse,

wouldn't default_transport_rate_delay = 15s

be a safe setting to relax the whole transport a bit?

from a receivers perspective, that's something i would like to see instead
of having ongoing delivery.

Am Do., 6. Dez. 2018 um 14:41 Uhr schrieb Wietse Venema <
... at porcupine dot org>:

Re: Best practice - concurrent delivery to remote sites

By Wietse Venema at 12/06/2018 - 10:20

Stefan Bauer:
It is a big sledgehammer that allows 4 delivers per minute per
destination (or per transport). If that works for you, great.
Just keep in mind that 'postfix reload' will reset the rate
delay timers.

Wietse

Re: Best practice - concurrent delivery to remote sites

By Andrey Repin at 12/06/2018 - 13:28

Greetings, Wietse Venema!

I'd like to ask for clarification, as man page wording is not clear.

The original wording is

To me, it is unclear,
- what considered "individual deliveries"? Individual messages? Individual
connects to the destination?
- what "one at a time" means exactly? Will queue manager connect and
disconnect for each message in queue? Will it try to deliver multiple messages
to the same destination in parallel, over multiple connections?

Re: Best practice - concurrent delivery to remote sites

By Viktor Dukhovni at 12/06/2018 - 13:43

One delivery at a time. A delivery is a handoff of a message with a set of
recipients of that message to a delivery agent for processing.

Less than two or more in parallel.

The queue manager does not connect to remote destinations, delivery
agents make connections. The queue manager asks delivery agents to
perform work, and collects the results.

One delivery at a time, with the configured delay between deliveries.
[ where one is less than two. ]

Re: Best practice - concurrent delivery to remote sites

By Andrey Repin at 12/06/2018 - 14:19

Greetings, Viktor Dukhovni!

In other words, if I have multiple different messages to the same destination,
I can't know if they will be delivered through single connection?
And can't control it?

Re: Best practice - concurrent delivery to remote sites

By Viktor Dukhovni at 12/06/2018 - 14:40

If the inter-message spacing exceeds the either of:

<a href="http://www.postfix.org/postconf.5.html#connection_cache_ttl_limit" title="http://www.postfix.org/postconf.5.html#connection_cache_ttl_limit">http://www.postfix.org/postconf.5.html#connection_cache_ttl_limit</a>
<a href="http://www.postfix.org/postconf.5.html#smtp_connection_cache_time_limit" title="http://www.postfix.org/postconf.5.html#smtp_connection_cache_time_limit">http://www.postfix.org/postconf.5.html#smtp_connection_cache_time_limit</a>

then any cached connections would be closed before it is time to send
another message. Generally, with serialized deliveries, you should not
cache connections. Keeping idle connections open is anti-social, you're
consuming remote resources.

A transport with a destination rate delay will not do demand caching,
which IIRC requires either concurrent or closely spaced deliveries
before it is enabled.

Bottom line, with rate delays, each delivery would be expected to use a new
connection. On demand connection re-use is not compatible with rate delays.

Re: Best practice - concurrent delivery to remote sites

By Andrey Repin at 12/06/2018 - 16:15

Greetings, Viktor Dukhovni!

Thank you for your assistance.
I see how lingering connections could be a worse problem than multiple
simultaneous connections.
I'll have to think this issue through again.
And probably start a new topic for it.

Re: Best practice - concurrent delivery to remote sites

By Andrey Repin at 12/06/2018 - 13:19

Greetings, Wietse Venema!

Assuming his use case (hundreds of mails could be generated per minute to the
same destination), this seems appropriate.
I'm considering doing the same on my relay systems, to limit the rate at which
they talk to the smarthost.
My use case is not "hundreds", but I'd rather have this level of throttling,
than leaving a wide open gap for brute force attacks.
(As I'm not a huge fan of fail2ban. I prefer more direct approaches.)

Re: Best practice - concurrent delivery to remote sites

By Andrey Repin at 12/06/2018 - 07:57

Greetings, Stefan Bauer!

For queue management, look at <a href="http://www.postfix.org/qmgr.8.html" title="http://www.postfix.org/qmgr.8.html">http://www.postfix.org/qmgr.8.html</a>
I can't provide exact solutions, as I'm solving a similar problem myself ATM.

Sure they are. To each their own.