DevHeads.net

authselect: what to do with systemd and nss-mdns that modify nsswith.conf

Hello,

systemd and nss-mdns packages modifies nsswitch.conf in their %post
scriptlets which creates conflicts with authselect on systems that are
configured by authselect. This needs to be solved somehow.

Originally, I wanted to create an authselect command that can be used by
packages on systems that are configured by authselect and on systems
that are configured with different ways [1]. But it turned out that
there are only two packages that touches nsswitch.conf so I do not think
this is necessary.

I prepared initial design document with multiple solutions that came to
my mind [2] and I would like to discuss this with the community to see
what is the correct and desired way to solve this.

Thank you,
Pavel.

[1]
<a href="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/message/4YMQOZCV32272UIGZM45NVJLVZY2UOZH/" title="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/message/4YMQOZCV32272UIGZM45NVJLVZY2UOZH/">https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject....</a>

[2]
<a href="https://fedoraproject.org/wiki/User:Pbrezina/Authselect_and_packages_that_modifies_nsswitch" title="https://fedoraproject.org/wiki/User:Pbrezina/Authselect_and_packages_that_modifies_nsswitch">https://fedoraproject.org/wiki/User:Pbrezina/Authselect_and_packages_tha...</a>

Comments

Re: authselect: what to do with systemd and nss-mdns that modify

By =?UTF-8?Q?Pavel... at 01/04/2019 - 03:50

On 12/6/18 12:20 PM, Pavel Březina wrote:
Reading the whole discussion here, I think it would be best for the
moment to make systemd and nss-mdns scriptlets compatible also with
authselect configuration and let systemd maintainers to get their
modules in glibc if/when possible. Once it is in glibc default
nsswitch.conf it will get picked up by authselect.

Re: authselect: what to do with systemd and nss-mdns that modify

By Lennart Poettering at 12/06/2018 - 08:36

Heya!

nss-systemd should be in nsswitch.conf by default. It's required for
systemd's DynamicUser=1 option to work correctly, and that's core
service functionality. Hence, given that systemd is Fedora's PID 1,
nss-systemd should also be in nsswitch.conf unconditionally (in the
'passwd' and 'group' lines). A system where nss-systemd is not enabled
is simply broken right now.

nss-myhostname should be in nsswitch.conf by default too. It's very
minimal, and just makes sure the local hostname remains resolvable all
the time. By enabling this, installers and image generators don't have
to patch /etc/hosts anymore like they traditionally did, in fact they
can remove it altogether and just leave resolution of the local
hostname to the module, and it will magically follow whatever is
currently set via sethostname(). This module should be in the 'hosts'
line.

Then there is nss-mymachines. It's primarily useful if
systemd-machined or systemd-nspawn is used. Given that those are now
part of the 'systemd-container' RPM it would be OK to also add
nss-mymachines to nsswitch.conf only when the RPM is installed, if
there's a concept for that. That said, in order to simplify things,
and given that systemd is a very core part of the OS I'd personally
just put it statically in nsswitch.conf too by default. After all a
missing NSS module listed in nsswitch.conf is just skipped, hence this
should not matter. This module should be in the 'passwd', 'group' and
'hosts' lines.

Finally, there's nss-resolve. It's the client side to
systemd-resolved. It's the client side to systemd-resolved's
DNS/mDNS/LLMNR/DoT/DNSSEC stack. systemd-resolved is not default in
Fedora right now. Quite frankly I think it should be, but that's another
political discussion (and I am not sure I am ready to have it right
now). The module is benign though: if resolved is not running it
doesn't do anything. It only does its thing if resolved is
running. Thus I'd also suggest to just enable it by default, and
simplify things because then people can use resolved just by doing
"systemctl enable systemd-resolved" and don't need to do anything
else. This module should be in the 'hosts' line.

Summary: I'd make things simple, and enable all four unconditionally
and by default without any dynamic infrastructure, without postinst
scripts or anything. If that's not acceptable, then at least two of the
four should be unconditionally enabled.

Lennart

Re: authselect: what to do with systemd and nss-mdns that modify

By Zbigniew =?utf-... at 12/06/2018 - 11:00

On Thu, Dec 06, 2018 at 02:36:36PM +0100, Lennart Poettering wrote:
I also think we should "enable" them unconditionally, in the sense that
they should be configured in the default /etc/nsswitch.conf configurations.
As Lennart wrote, if the module is not installed in the filesystem, that
configuration has no effect. And if it is installed, but talks to an
inactive service, it has almost no effect too (apart from the linkage
and small runtime overhead). "Enabling" them in one consistent way in
the normal distro configuration seems much better to me then patching
it in, in a slightly different way on each installation.

In QA the subject of reducing the test matrix often comes up. This is
a nice opportunity: let's avoid having a nsswitch line that depends on
the installation order.

Re: authselect: what to do with systemd and nss-mdns that modify

By Simo Sorce at 12/06/2018 - 11:25

On Thu, 2018-12-06 at 16:00 +0000, Zbigniew Jędrzejewski-Szmek wrote:
I wonder if we should think of a tighter system integration and subsume
the tasks of nss_machines into SSSD.
It would allow for detection and logging of UID conflicts should they
happen in a live system with the ability, for the admin to better
choose which of the pools should have priority in case of conflict ...

Simo.

Re: authselect: what to do with systemd and nss-mdns that modify

By Lennart Poettering at 12/06/2018 - 11:49

No disrespect, but think this would mean the layering is the wrong way
round: sssd is an add-on currently, a layer above systemd. systemd as
the lower layer really shouldn't call into higher layer stuff for
registering UIDs/GIDs. I mean, conceptually, higher level stuff should
call into lower level stuff but not the other way round. (And there's
also the problem that sssd is hardly universally accepted. systemd
probably runs on more systems not using sssd than on those using
sssd.)

Currently Linux has no nice protocol for registering UIDs or UID
ranges. In systemd we tried to make the best from what we got, and
what the layers underneath permit us (i.e. glibc NSS):

1. we always register the IDs we take possession of in NSS, to make
sure that it can be used as a comprehensive database of UID
registrations. (That's what the nss modules mentioned in my earlier
mail are about after all).

2. When picking a UID we check with NSS first if the UID is already
taken.

3. When picking a UID we also check if any IPC object is already owned
by the UID.

4. When checking whether UID is used we take the lckpwdf() lock first
(if we can, since its in /etc, i.e. potentially read-only), in
order to serialize registration. This sucks a bit admittedly, since
strictly speaking that lock is just about the file-based database
(i.e. /etc/passwd and friends). But it's the best there is, and it
is official glibc API.

We also documented in a lot of detail the UID/GID range assumptions we
make:

<a href="https://systemd.io/UIDS-GIDS" title="https://systemd.io/UIDS-GIDS">https://systemd.io/UIDS-GIDS</a>

I think it would be great if we could come to a better protocol for
taking possession of UID ranges in the long run. But I still think the
best database for such allocation should be NSS itself, right
now. What would be great if we'd at least agree on a better lock file
though, maybe in /run. And maybe some rules that when you allocate
long UID ranges at once (like container managers do), that you only
have to NSS check key UIDs from it.

Lennart

Re: authselect: what to do with systemd and nss-mdns that modify

By Simo Sorce at 12/12/2018 - 10:10

On Thu, 2018-12-06 at 17:49 +0100, Lennart Poettering wrote:
Which layer is above or below and handles this is not set in stone
either way. And we have many systems that call "above" (like kernel
calling to user-space), so I do not see this as a problem.

Acceptance is also not a problem, the point is integrating when and
where it makes sense on Fedora.

This is fine.

This is where we may want to integrate. The problem is that systems
like AD and FreeIPA are represented in Linux (via SSSD or Windbind) by
allocating a range where new users will be mapped, however those UIDs
cannot be represented in NSS as they do not exist yet in the central
system. So having a place where you can ask "is this range free?/this
range is taken" sounds like is needed.

This is fine for cases where all users can be taken at once, but see
above.

It does fall short, most enterprise installations use LDAP or other
external databases, and that lock does nothing. And in general won't be
able to do anything because the single system has no authority to grab
ranges. However the daemons that provide users to NSS have a little bit
of knowledge of what are the ranges that will be likely/certainly used
by those external services, and should be consulted when there is a
need to allocate a range, because in general the external system is the
authoritative one.
In case of conflict, it doesn't matter how painful on a single system,
it's the single system that has to change as the external system is
shared among many and is generally always considered authoritative.

+1

Except NSS can't do reservations, and those are critical.

What's the point of a lockfile ?
What we really need is a way to reserve a range, then you can do what
you need within that range in your nss module, without any lock needed
(as long as other modules respect the range assignments).

I am not sure I grok this last sentence, but yeah, the whole point is
about allocating ranges, we should come up with a way to do that
system-wide where all providers can participate.
We also need to figure out how to handle potential conflicts after
administrative changes are pushed around from either a dynamic central
system (AD/FreeIPA/LDAP) or a config managment system that only has a
view on centrally managed resources.
It doesn't have to be anything magic, but it needs to be something the
admin can act on reasonably.

Simo.

Re: authselect: what to do with systemd and nss-mdns that modify

By Lennart Poettering at 12/12/2018 - 10:41

Well, systemd is not a Fedora-only project. It's in fact an excercise
in unification across distros, and hence interfacing with upper layer
stuff that isn't accepted standard outside of Fedora is always a
problem.

My recommendation would be to simply synthesize "reservation" users on
these UIDs as long as you are not assigning them to a real user, that
makes sure noone else can take them. i.e. call them "sssd-rsrvd0",
"sssd-rsrvd1", … or so, and return a synthetic "struct passwd" for
them, that contains generic fallback data. That's easy to do in an NSS
module, and the records can be safely synthesized on-the-fly. With
that you basically take ownership of an UID range for sssd fully, and
can be sure that nothing else will invade it. Neither systemd, nor
adduser, nor anything else.

When a user is actually allocated to one of those UIDs you then
replace the reserved entry with a real one. (And flush nscd, if you
care about nscd. Ideally of course nscd wouldn't require that and
accept a TTL value from NSS modules the same way it accepts them for
host lookups, so that the synthetic records could carry a TTL of zero,
so that there's no need to every flush any caches explicitly).

Well, it could, you just need to take the UIDs, see above.

Well, sure, it would be great if we could assign ranges for exclusive
ownership to specific programs, but even then it would be wise to be
defensive, and check NSS before we take a UID — and to do that
race-freely there kinda needs to be a lock file.

There's also the big problem that UID space is scarce. In containers
you usually don't get more than 64K UIDs (so that you can have 2^16
for each container, and 2^16-1 at max, so that you use the full 2^32
UID space). And in those 64K space is scarce, as 1000…60000 is already
all taken by classic adduser. Which only leaves 60000…65533 for
particular applications. 5.5K is not that much left.

With systemd's dynamic users we took "soft" possession of the range
61184…65519. "Soft" because we are defensive if somebody else
allocated from the same range, as we cannot claim sole ownership of
this. And from reports we got that's kinda necessary as UNIX has no
rules in this area and hence existing bigger university installations
used the full 64K range basically without any reservations.

Hence, yes, I'd love if we'd could assign exclusive owenership of
specific UID ranges to specific apps, but that's probably too late
now...

But anyway, I figure you can avoid taking any externally visible locks
if you don#t want to, as long as you reserve the full UID space you
want exclusive ownership for with the suggested synthetic reservation
user entries. As long as all UIDs you took possesion of never show up
"un-unowned" there's no need to take a lock anywhere, as long as you
internally instantly switch over from "reserved" to "taken" and back.

So, the problem with using NSS as reservation database for large UID
ranges is efficiency: if you want to allocate 64K in one go, you'd in
theory have to do getpwuid() on every single UID in it before deciding
to own it. Hence, what I was looking for was that we can say that one
would only have to check all UID where ((uid & 0xFFFF) in [0 … 65535])
or so, i.e. only the first and last UID in each 64K subrange of the
range allocate...

For both dynamic users and nspawn user ranges we so far were defensive
and ordered us to the end of the NSS module list, leaving other
modules precedence.

Lennart

Re: authselect: what to do with systemd and nss-mdns that modify

By Zbigniew =?utf-... at 12/06/2018 - 11:34

On Thu, Dec 06, 2018 at 11:25:04AM -0500, Simo Sorce wrote:
Integration with sssd could be useful, dunno. But nss modules only report
existing usage of uids on the system. So by the time the nss modules are
invoked, it's already too late, in the sense that two completely unrelated
entities are sharing the user, possibly leading to unintended privilege
augmentation or information leakage. Nss modules are not useful to "choose"
anything.

Zbyszek

Re: authselect: what to do with systemd and nss-mdns that modify

By Lennart Poettering at 12/06/2018 - 11:55

Yes, I agree fully. Announcing allocated users with NSS is one thing,
it's something we always should do, unconditionally. It's only
reasonably way to announce you took possession of a range. Actually
allocating ranges is a different discussion. It's a discussion worth
having, but is unrelated from the NSS discussion I think.

Lennart

Re: authselect: what to do with systemd and nss-mdns that modify

By Florian Weimer at 12/06/2018 - 09:10

* Lennart Poettering:

The way myhostname is currently implemented, it is rather problematic.
I think it has largely stopped stomping over the DNS namespace (with the
_gateway change), so it could be moved to the front, eliminating all
those problems.

The other issue is that the systemd NSS modules are quite heavy and pull
in other DSOs and may cause single-thread optimizations to be disabled
because they load libpthread.

Thanks,
Florian

Re: authselect: what to do with systemd and nss-mdns that modify

By Dominik 'Rathan... at 12/07/2018 - 07:11

On Thursday, 06 December 2018 at 15:10, Florian Weimer wrote:
[...]
That's news to me, what is this "_gateway" change? I noticed that in
a couple of traceroutes, but assumed it was an ISP thing. Are you telling
me this is a Fedora "feature"? Is this documented somewhere?

Regards,
Dominik

Re: authselect: what to do with systemd and nss-mdns that modify

By Tomasz Torcz at 12/07/2018 - 10:46

On Fri, Dec 07, 2018 at 01:11:00PM +0100, Dominik 'Rathann' Mierzejewski wrote:
This is systemd feature. Looking into its NEWS file, gateway resolving
was added in v218, released 4 years ago, in December 2014.
It was changed to _gateway in v235, released in October 2017.

Re: authselect: what to do with systemd and nss-mdns that modify

By Dominik 'Rathan... at 12/07/2018 - 12:58

On Friday, 07 December 2018 at 16:46, Tomasz Torcz wrote:
Ok, I read man nss-myhostname and I do like the hostname and localhost
resolveability feature. I don't like the _gateway part though. Can I
have the former without the latter?

Regards,
Dominik

Re: authselect: what to do with systemd and nss-mdns that modify

By Zbigniew =?utf-... at 12/07/2018 - 13:19

On Fri, Dec 07, 2018 at 06:58:31PM +0100, Dominik 'Rathann' Mierzejewski wrote:
Not without patching systemd :]

Zbyszek

Re: authselect: what to do with systemd and nss-mdns that modify

By Zbigniew =?utf-... at 12/07/2018 - 08:09

On Fri, Dec 07, 2018 at 01:11:00PM +0100, Dominik 'Rathann' Mierzejewski wrote:
man nss-myhostname(8).

Zbyszek

Re: authselect: what to do with systemd and nss-mdns that modify

By =?UTF-8?Q?Pavel... at 12/06/2018 - 08:58

On 12/6/18 2:36 PM, Lennart Poettering wrote:
Reading <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1284325" title="https://bugzilla.redhat.com/show_bug.cgi?id=1284325">https://bugzilla.redhat.com/show_bug.cgi?id=1284325</a> there is can
happen some ID overlaps with FreeIPA/Samba which is undesirable. I would
say that this must be solves if this module is enabled by default. Was
there any progress in this area?

Re: authselect: what to do with systemd and nss-mdns that modify

By Lennart Poettering at 12/06/2018 - 10:30

I think that's a misunderstanding of what the module does. At the
point the module announces those uid/gid ranges they are already
reserved, hence the conflict is already there. nss-mymachines is hence
only the messanger, not the culprit. Moreover, I think that
registering all taken users in NSS is really key to minimize such
conflicts. Hence, I am very strongly of the opinion that any component
taking possession off a user or a range of users it *must* show up in
NSS too, so that other components know.

I commented on the bug too.

Lennart

Re: authselect: what to do with systemd and nss-mdns that modify

By Florian Weimer at 12/06/2018 - 13:42

* Lennart Poettering:

I don't think we enforce that reservation system-wide. Do we filter out
those accounts when they come in over LDAP? Can users add them locally
using adduser?

None of the NSS modules in glibc provide such filtering.

Thanks,
Florian

Re: authselect: what to do with systemd and nss-mdns that modify

By Lennart Poettering at 12/06/2018 - 13:58

The UID/GID allocation in systemd itself (for DynamicUser=1) and in
systemd-nspawn (for --private-users=) both check NSS before they take
a UID/GID. Hence, if LDAP users live in the same range we use it makes
the space scarcer, but it shouldn't cause conflicts — as long as
everything is properly registered in NSS.

"adduser" registers from the range 1000…60000 on Fedora by
default. DynamicUser=1 uses the range 61184…65519. systemd-nspawn uses
524288…1879048191. So these at least do not overlap.

Lennart

Re: authselect: what to do with systemd and nss-mdns that modify

By Tom Hughes at 12/06/2018 - 08:53

Based on my experimentation with an F29 live image last week both
nss-systemd and nss-myhostname are in the default configuration.

Equally, neither nss-mymachines or nss-resolve appear to be in
the default configuration on an F29 image.

Tom

Re: authselect: what to do with systemd and nss-mdns that modify

By Florian Weimer at 12/06/2018 - 09:11

* Tom Hughes:

Not in the file shipped by the glibc package.

Florian

Re: authselect: what to do with systemd and nss-mdns that modify

By Tom Hughes at 12/06/2018 - 09:27

On 06/12/2018 14:11, Florian Weimer wrote:
Well something that has been installed as part of the live
image has enabled them then...

Tom

Re: authselect: what to do with systemd and nss-mdns that modify

By =?UTF-8?Q?Pavel... at 12/07/2018 - 05:20

On 12/6/18 3:27 PM, Tom Hughes wrote:
systemd scriptlet enabled them in nsswitch.conf.

Additionally, if you have configured your system with authselect, there
is systemd in all profiles shipped with the package in passwd and
groups. But modules in hosts are still enabled via systemd scriptlet.

Re: authselect: what to do with systemd and nss-mdns that modify

By Florian Weimer at 12/06/2018 - 07:31

* Pavel Březina:

Other packagers (notably sssd) have made the required changes by
requesting them form glibc developers.

Would this help to solve the problem here?

Thanks,
Florian

Re: authselect: what to do with systemd and nss-mdns that modify

By =?UTF-8?Q?Pavel... at 12/06/2018 - 08:55

On 12/6/18 1:31 PM, Florian Weimer wrote:
If all those packages should be enabled by default, this probably the
best solution.