DevHeads.net

SMTP_HELO_NAME can cause Blacklist triggers

I learned the hard way that if you don't set $myhostname to a FQDN you can quickly end up on a black list despite having valid SPF records.
The documentation is IMO insufficiently clear that $myhostname MUST be fully qualified and that Postfix will NOT tack on $mydomain if no 'dots' are detected.

Sure, this could be chalked up to "stupid admin error" but doesn't it make sense to either warn about a short $myhostname during server startup and/or add code to smtp_proto.c before calling smtp_chat_cmd(session, "EHLO %s", var_smtp_helo_name) that if 2 dots are not found in $myhostname to automatically tack on $mydomain?

I think all/most other cases of a short $myhostname are unlikely to cause mayhem, but during HELO it is major.

I thought the valid_hostname() check in mail_params_init() would fault but it doesn't. Is the best solution then to extend valid_hostname() to actually check for at least 2 dots?
I've generated a proposed patch.
<a href="https://github.com/tb3088/postfix/commit/171c01607d9c4a0b819b5507502ca075ea0ae636" title="https://github.com/tb3088/postfix/commit/171c01607d9c4a0b819b5507502ca075ea0ae636">https://github.com/tb3088/postfix/commit/171c01607d9c4a0b819b5507502ca07...</a>

Aside, isn't the sense of this test backwards?
<a href="https://github.com/vdukhovni/postfix/blob/eb73f242bb5ecd3ddf70142835e349c10a71c078/postfix/src/util/valid_hostname.c#L149" title="https://github.com/vdukhovni/postfix/blob/eb73f242bb5ecd3ddf70142835e349c10a71c078/postfix/src/util/valid_hostname.c#L149">https://github.com/vdukhovni/postfix/blob/eb73f242bb5ecd3ddf70142835e349...</a>

Comments

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Wietse Venema at 02/05/2019 - 16:50

Patton, Matthew [Contractor]:
If someone registers the domain 'foo', then that is a valid name,
and they have right to use "helo foo", "mail from:user@foo", and
so on.

The problem is not sending helo without a dot, the problem is sending
helo with a name that does not exist.

Wietse

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Viktor Dukhovni at 02/05/2019 - 16:59

Returning to the OP's question, Postfix does append $mydomain to
the automatically derived value of $myhostname when the latter
is not explicitly set in main.cf and is not fully qualified.

The OP probably has an explicit unqualified setting of myhostname
in main.cf (perhaps via Debian's: myhostname = /etc/mailname or
similar) and it is not up to Postfix to second-guess such an
explicit setting. If you want Postfix to derive the FQDN,
do not set myhostname at all, set just mydomain, and Postfix
will append that to the system hostname if not an FQDN.

The above assumes that the system hostname is stable, and not
derived via DHCP. In the latter case do set an explicit stable
FQDN for $myhostname.

RE: SMTP_HELO_NAME can cause Blacklist triggers

By Patton, Matthew... at 02/05/2019 - 22:42

Except that it doesn't. (or I misunderstood what you wrote)
I set $myhostname = 'smtp'.
$mydomain was also set.

I had to set both since gethostbyname() would have returned a value of 'ip-XXXXXX.aws.internal'.

My HELO string was verifiably just $myhostname. The naked 'smtp' was an instant blacklist largely as a result of millions of vulnerable Microtek home routers which have been exploited.

If Postfix had instead used $myhostname.$mydomain IFF $myhostname is not FQDN (has no dots at all) then 'smtp'.$mydomain would have been perfectly fine since there was an A record for it for quite some time.

Fair enough. But I still think that at the very least the docs should be a little more explicit, and furthermore a warning is merited during valid_hostname() and with SLOPPY_VALID_HOSTNAME we can continue without error.

If nothing else HELO should really scream if there are no dots or should be revised to always default to doing the reasonable thing. On the Internet it can't be valid not to have at least 2 dots in $myhostname.$mydomain or 1 dot in $mydomain. RBL are getting increasingly aggressive (or stupid, depending on viewpoint) such that any non-dot value of HELO could easily wind up (if it isn't already) a criteria by which to blacklist an IP.

Whatever shenanigans are permitted internal to an organization is not important but as "good netizens" shouldn't the program at least complain even if it allows you to still shoot oneself in the foot? It's also a good idea not to surprise software users with unexpected outcomes. I had every expectation that setting both $myhostname and $mydomain was sufficient for proper behavior. I had to go read the code to discover that such was not the case.

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Matus UHLAR - f... at 02/07/2019 - 15:09

On 06.02.19 02:42, Patton, Matthew [Contractor] wrote:
any evidence about this?

what led you to the conclusion that your non-fqdn hostname caused RBL
listing?

I know servers that refuse non-FQDN helo.
I've seen servers that refuse invalid or generic DNS names
(ip-XXXXXX.aws.internal is both).
But I don't remember a RBL that would immediately list such hosts.

Precisely as documented: $smtp_helo_name defaults to $myhostname
<a href="http://www.postfix.org/postconf.5.html#smtp_helo_name" title="http://www.postfix.org/postconf.5.html#smtp_helo_name">http://www.postfix.org/postconf.5.html#smtp_helo_name</a>

again, how do you know it got to any blacklist?

well, since the $mydomain is by default taken from $myhostname, it should be
obvious you must set $myhostname to a FQDN.

yes, apparently some of the docs could be little more explicit about
$hostname or $smtp_helo_name should be a FQDN.

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Bill Cole at 02/07/2019 - 18:52

I don't believe any DNSBL will list on a generic "non-FQDN helo" basis,
however there are idiosyncratic non-FQDN helo names and patterns which
are good enough indicators of membership in specific botnets to block or
even list on a DNSBL. I believe that the CBL has at times used such
names as a listing trigger, but it is important to add that it is my
understanding that the CBL also has strong criteria for *where* and
*how* a sending machine must display app-layer fingerprint behaviors for
a listing, i.e. you can't get listed for sending entirely legitimate
non-bulk email to working addresses of living humans using an otherwise
botnet-only HELO name.

PSBL and NiX Spam *MAY* also use such "fingerprint" HELO names and I'm
not as confident that their safeguards are as strong as CBL's.

But your core point is valid: mailing from an AWS instance (or from
anywhere on an IP with a programmatically derived PTR) in general is
going to work poorly. There is too little accountability for abuse from
the AWS IP pool for it to merit a default level of trust.

RE: SMTP_HELO_NAME can cause Blacklist triggers

By Patton, Matthew... at 02/08/2019 - 10:42

The host has both forward and reverse registered. It was in SPF. It's been sending mail just fine for months.
Previously the server had been HELO using an invalid FQDN configured via $myhostname. The hostname was bogus, as in could not be resolved, though the domain did exist. Furthermore the domain it was identifying as didn't match the PTR nor SPF. If anything, the host should have been blackballed from day one.

I changed the HELO to 'smtp' and $mydomain to match the A/PTR and the CBL got triggered within a very short period of time. Likely because of a hosted email provider like Microsoft (outlook.com) since we send a lot of messages their way. The other BL were clear/green.

The CBL remediation page was explicit about the lone 'smtp' being a trigger word thanks to Microtek routers, and it refused to clear me (hit count kept going up) until I had a FQDN or perhaps a hostname that wasn't a trigger word - I didn't have the luxury of time to screw around testing hypotheses. As soon as I changed my HELO to the FQDN the reputation started to improve until it rolled off.

If the HELO is simply 'smtp' its apparently good enough for the CBL.

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Matus UHLAR - f... at 02/11/2019 - 11:13

On 08.02.19 14:42, Patton, Matthew [Contractor] wrote:
It is not. As you said, your reverse DNS was "ip-XXXXXX.aws.internal", and
thus generic name. the HELO name was just "smtp".

They were two different reasons leading to CBL listing.

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Alice Wonder at 02/07/2019 - 19:29

On 2/7/19 2:52 PM, Bill Cole wrote:
*snip*
Even with a proper reverse DNS if you use a VPS hosting service you have
to deal with being blacklisted. Doesn't matter how long you have has the
IP either.

Someone spins up a VPS on same subnet and starts spamming and the entire
subnet gets put on the blacklist because of showshoe spammers.

What you can do if you don't have the cash to buy your own subnet and
use a VPS service, get three of them at different hosting locations and
when one gets blacklisted, forward to one of the others to relay until
you are removed from the blacklist (usually takes 24 hours because of
cached DNS results)

Re: SMTP_HELO_NAME can cause Blacklist triggers

By John Stoffel at 02/11/2019 - 17:46

Alice> On 2/7/19 2:52 PM, Bill Cole wrote:
Alice> *snip*
Alice> Even with a proper reverse DNS if you use a VPS hosting service
Alice> you have to deal with being blacklisted. Doesn't matter how
Alice> long you have has the IP either.

Amen! I'm in this boat right now, where Charter Spectrum in the US is
blocking all Digital Ocean subnet blocks. Jerks.

Alice> Someone spins up a VPS on same subnet and starts spamming and
Alice> the entire subnet gets put on the blacklist because of showshoe
Alice> spammers.

Or worse... *some* lazy companies just block entire blocks of
addresses.

Alice> What you can do if you don't have the cash to buy your own
Alice> subnet and use a VPS service, get three of them at different
Alice> hosting locations and when one gets blacklisted, forward to one
Alice> of the others to relay until you are removed from the blacklist
Alice> (usually takes 24 hours because of cached DNS results)

I'm starting to think I need to do this, but that would triple monthly
cost.

What I'm working on now is to just relay all email for @charter.net
(and some other domains they run) via my charter.net account, since I
use the jerks for my only possible internet at home.

John

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Viktor Dukhovni at 02/05/2019 - 23:11

Actually, it does, under the stated condition. You have an explicit
setting, which is honoured. Postfix does not second-guess explicit
settings, the administrator knows best.

Which is discussed in the second part of my post.

If that's what you want, and you're setting myhostname explicitly,
then it is your responsibility to do that. This allows users who
do want dotless hostnames to have those if that's right for them.

Which is apparently not your case. :-(

The documentation explains how the default value is constructed.
As with all the other parameters, if you choose a non-default value,
then that's what you get:

<a href="http://www.postfix.org/postconf.5.html#myhostname" title="http://www.postfix.org/postconf.5.html#myhostname">http://www.postfix.org/postconf.5.html#myhostname</a>

There are legitimate use-cases on private networks where a short
HELO is perfectly fine, and may be preferred. Postfix defaults to
an FQDN for myhostname, but will bend to your will if you override
that default.

You misunderstood the documentation, the domain is only appended
when computing the *default* value, when the name returned by
gethostname() is dotless. Read with a bit of care, the documentation
promises exactly that.

The internet hostname of this mail system. The default is to
That is mydomain is only appended to the result of gethostname()
when constructing the default value, it plays no role in explicit
settings by the administrator, and this is correct behaviour.

Warnings about unqualified hostnames don't belong in the running
Postfix system, they could be part of some option postconf(1)
"lint-mode", if someone had cycles to contribute the requisite code.

RE: SMTP_HELO_NAME can cause Blacklist triggers

By Patton, Matthew... at 02/06/2019 - 00:36

In Internet-connected SMTP (which is something like 99.99999% of installations) if $myhostname is not a FQDN or at least a properly registered domain in DNS as it concerns HELO, bad things are *likely* to happen. The number of people who run mail servers isolated from the Internet is vanishingly small. Why cater to the oddball environment where anything goes? So what if they have to suffer the ignominy of a warning message about a arguably mis-configured server?

No I didn't misunderstand the documentation. I provided both pieces of information via main.cf and I damn well expected Postfix to do the *self-evidently correct* thing. Much to my annoyance It did not.

Technically that is a violation. The hostname is and always has been defined as being UPTO the first dot. So Postfix is re-defining the term 'hostname' contrary to 40 years of Unix tradition if not specification? Legions of sysadmins set the hostname as a singleton and domain name is derived from other sources at the OS level. 'hostname -s' and 'hostname -f' etc. do the right thing. Sure Linux and friends have been known to play fast and loose with the terminology (eg. Redhat's HOSTNAME=FQDN in /etc/sysconfig/network) but that doesn't make it right.

One could read <a href="https://www.ietf.org/rfc/rfc2821.txt" title="https://www.ietf.org/rfc/rfc2821.txt">https://www.ietf.org/rfc/rfc2821.txt</a> sections 3.2 and 4.1.1.1, the latter of which *CLEARLY* establish that the proper string to send is the FQDN if at all possible. By setting $mydomain and $myhostname I had provided that information. Therefore by any reasonable reading Postfix should foremost define 'smtp_helo_name=$myhostname.$mydomain' (or if you rather '$fqdn') and amend that to just $myhostname should it have dots. Or if that's too big a change, then if there is no dot in $myhostname, tacked on $mydomain if it was available. Both behaviors can be overridden should the admin explicitly set 'smtp_helo_name'. (aside, I didn't even know about 'smtp_helo_name' before this incident.)

Even if we don't update Postfix to "second guess the all-knowing admin insistent on running an invalid environment" (I mean seriously, even Windows AD forces you to behave - this isn't the 80's), then it should have at least issued a warning during configuration load. No, I'm not about to suggest that every time a HELO command is issued, a warning message get emitted.

I won't belabor the point any further. Clearly this is a case of WONTFIX and if the admin is foolish enough to override a default then he had better have read the source code first. I don't consider that a defensible position. But oh well...

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Peter at 02/06/2019 - 05:29

On 06/02/19 17:36, Patton, Matthew [Contractor] wrote:
...

This is absolutely incorrect. A hostname is an identifier for a machine
or node on a network. When that network is a local network the hostname
absolutely can be a single word. When that network is the internet then
a fully qualified domain name is *required* to identify a host.
considering that above you make it blatantly obvious that you are
referring to usage on the internet at large and not usage on a local
network then it is absolutely appropriate for the referenced hostname to
be a FQDN in this context.

<a href="https://en.wikipedia.org/wiki/Hostname" title="https://en.wikipedia.org/wiki/Hostname">https://en.wikipedia.org/wiki/Hostname</a>
RFC952

Peter

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Viktor Dukhovni at 02/06/2019 - 00:47

I repeat, you misunderstood the documentation. Postfix computes its
best guess at the FQDN when you DO NOT *explicitly* set myhostname, in
main.cf. If you DO *explicitly* set myhostname in main.cf, then it is
*your* choice as whether you want to append $mydomain, something else
or nothing. The *default* behaviour is as you'd expect.

Postfix has been around for 22 years, and this has not been an issue for
the vast majority of users. Your misunderstanding may be annoying, but
we can't prevent configuration mistakes. What is a mistake for some, is
a valid choice for others. What we can do is provide sensible defaults
and then take the user's explicit configuration settings to be a valid
expression of intent.

RE: SMTP_HELO_NAME can cause Blacklist triggers

By Patton, Matthew... at 02/06/2019 - 01:50

The issue is NOT that I wanted Postfix to willy-nilly mangle $myhostname into a FQDN on my behalf. If there were a private keyword of $fqdn that was supported in the configuration, then yes, I would fully expect Postfix to intelligently determine if $fqdn should equal $myhostname or $myhostname.$mydomain modulo the presence of dots.

The issue is narrowly concerning HELO only! It's the only important case where stuff can and does go badly if it's malformed. By perverting the very definition of what is a hostname, Postfix sets the user up for unexpected problems. If Postfix is going to deviate from accepted norms, then the least it could do is alert the admin that "Oh by the way since our defaults assume you subscribe to our redefining of the term 'hostname' to actually mean 'FQDN' (or be prepared for bad things), we see that your hostname isn't a FQDN, and you might want to change that or update 'smtp_helo_name' and some other settings (TBD) before you get blackballed and wonder what the hell just happened."

(aside: I would also suggest $myorigin should likewise always default to $fqdn, and $mydestination have both short and fully-qualified hostnames as standard).

Other pieces of software are cognizant enough to look for questionable values and to gripe accordingly. At the very least I was asking for a big ol' gripe, or logic on this one code-path that tries to do the right thing by default, or a 1-liner in the docs making it damn obvious that if $myhostname wasn't a FQDN you had better be running off-net or strongly urged to validate 'smtp_helo_name' and other potentially very important settings. If the Postfix philosophy is to assume admins by definition will have read the source code so they can accurately intuit every knock-on effect of one of their choices, then we might as well go back to nasty ol' Sendmail. Surprising users with non-intuitive behavior is not nice. To paper it over with "you changed a default, you should have knowed" is similarly unhelpful.

Who luckily never change defaults or just go along with Postfix' in-house redefining of the term 'hostname'. Not to mention RBL were few and far between if even present for much of that 22 years. Now everyone and their dog runs hosted email of one kind or another and the rules are massively more strict. SPF records are a new invention, and automated blacklisting instigated by massive bot farms is new as well.

Anyhow, peace out. I forked the code and will update the logic as IMO it should have been all along.

Re: SMTP_HELO_NAME can cause Blacklist triggers

By Viktor Dukhovni at 02/06/2019 - 02:35

Such a belligerent, indignant tone never endears an OP to the
project maintainers, and creates a strong disincentive to making
any effort to address the issue. It would be prudent to lose a
bunch of the pejorative adjectives in the future.

You are free to switch to using those other pieces of software.