DevHeads.net

It is possible for Postfix logging to bypass journald?

We recently switched our Postfix mail servers to Ubuntu Server 18, which
uses journald for logging. Since we have monitoring systems that parse
/var/log/maillog, we enabled rsyslog with imuxsock so we still can parse
the log like we did before journald. But, it's unreliable.

Our monitoring systems are reporting failed deliveries of messages
because of missing log lines in /var/log/maillog. When using journalctl
to query the journal, the missing lines can be found, but these queries
are too CPU intensive.

We also see that journald is occasionally logging messages such as this:

Jan 08 20:55:16 host123 systemd-journald[11136]: Forwarding to syslog
missed 2 messages.

Since this message doesn't provide any information as to why the
messages were missed, I have to wonder if it's related to this warning
message on the rsyslog site:

"Note: It must be noted, however, that the journal tends to drop
messages when it becomes busy instead of forwarding them to the system
log socket. This is because the journal uses an async log socket
interface for forwarding instead of the traditional synchronous one."

See:
<a href="https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#imuxsock-systemd-details-label" title="https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.html#imuxsock-systemd-details-label">https://www.rsyslog.com/doc/v8-stable/configuration/modules/imuxsock.htm...</a>

I'm aware we could switch to using imjournal, which might solve the
issue since it reads the journal directly (which does seem to contain
the missing messages), but I have to imagine that it would come at a
very high CPU cost.

See:
<a href="https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html" title="https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html">https://www.rsyslog.com/doc/v8-stable/configuration/modules/imjournal.html</a>

So, I'm trying to figure out if it would be possible to get Postfix to
use an alternate logging mechanism that would completely bypass journald
so that we can have reliable loggging in a manner that is less CPU
intensive than journald/imjournal.

Ideas?

Thanks,

Curtis

Comments

Re: It is possible for Postfix logging to bypass journald?

By Robert L Mathews at 01/09/2019 - 20:55

On 1/9/19 4:05 PM, Curtis wrote:
We had this problem. It was fixed by putting this in
/etc/systemd/journald.conf:

# allow for busy mail logs; allows 1000 per second
RateLimitInterval=5s
RateLimitBurst=5000

And/or by putting this into /etc/rsyslog.conf:

$SystemLogRateLimitInterval 0

(The latter is supposedly no longer necessary, but it used to be, and
does not appear to be harmful.)

Re: It is possible for Postfix logging to bypass journald?

By Curtis at 01/28/2019 - 22:46

On 1/9/2019 5:55 PM, Robert L Mathews wrote:
Thanks for your input. Unfortunately, even after playing with these
settings, we see no improvement. When you run "systemctl status
systemd-journald" do you see any messages like this?

Jan 28 18:16:01 [somehost] systemd-journald[25662]: Forwarding to syslog
missed 6 messages.

If others are not seeing this issue, then I am wondering if it has
something to do our setup being inside of an LXC container.

For now, we ended up fixing our log parsing script to make journalctl
calls so that no lines are missed. Overall, journald seems like a huge
downgrade for us... I get the impression it was designed for desktop
users, not for servers.

That said, I noticed in another thread that Wietse announced that
Postfix has an option to log to a file now...

<a href="ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/postfix-3.4-20190127-nonprod-logger.RELEASE_NOTES" title="ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/postfix-3.4-20190127-nonprod-logger.RELEASE_NOTES">ftp://ftp.porcupine.org/mirrors/postfix-release/experimental/postfix-3.4...</a>

We're not ready to upgrade right now, but I'm looking forward to trying
this option out in the future.

Re: It is possible for Postfix logging to bypass journald?

By Wietse Venema at 01/29/2019 - 08:24

Postfix 3.4 will have support for logging to file and to stdout.

Otherwise, Postfix uses the syslog function, part of the system
library; Postfix has no control over where that library sends its
data. You may be able to tell systemd to keep its hands off the
syslog socket, in which case rsyslogd can do it job.

Wietse

Re: It is possible for Postfix logging to bypass journald?

By Wietse Venema at 01/09/2019 - 20:38

I recall that system-effing-d has a rare-limiting feature that very
helpfully drops Postfix logging.

Here's one search result with suggestions for systemd.
<a href="https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/" title="https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/">https://www.rootusers.com/how-to-change-log-rate-limiting-in-linux/</a>

Another search result: systemd and rsyslog both have rate limits.
<a href="https://support.asperasoft.com/hc/en-us/articles/216128628-How-to-disable-rsyslog-rate-limiting" title="https://support.asperasoft.com/hc/en-us/articles/216128628-How-to-disable-rsyslog-rate-limiting">https://support.asperasoft.com/hc/en-us/articles/216128628-How-to-disabl...</a>

It is time to update the Postfix page on LINUX logging brain damage.

Wietse

Re: It is possible for Postfix logging to bypass journald?

By Matus UHLAR - f... at 01/10/2019 - 15:22

On 09.01.19 19:38, Wietse Venema wrote:
oh, please... systemd and rsyslog. I use sysvinit+syslog-ng wherever
possible, on linux