smtpd_delay_reject with rspamd milter

I'm having trouble with access_maps kicking in after an upgrade from a
Postfix 2.something to Postfix 3.1. on Ubuntu 14.06 and using postscreen
and rspamd milter.

After some testing I'm not sure yet, but it looks like the recommended
smtpd_delay_reject = yes in connection with having the access_map checks
listed in the smtpd_recipient_restrictions is the cause of this. It allows
the milter to kick in, so that the access_check (almost?) never happens.

After changing to
smtpd_delay_reject = no
and moving the client access_maps to smtpd_client_restrictions they seem
to be working again.

rspamd is used as a milter (as specified in the rspamd docs) and not as a
policy service (which would allow putting it at the end of restrictions).

# rspamd
smtpd_milters = inet:localhost:11332
milter_protocol = 6
milter_mail_macros = i {mail_addr} {client_addr} {client_name}
milter_default_action = accept

Is this a known problem? Other workarounds? Can I specify to use the
milter only after the restrictions? (Maybe doesn't make sense?)
Or am I misinterpreting something?