DevHeads.net

Downgrading glibc from Rawhide removed /bin/sh (!)

$ sudo dnf install glibc-headers.i686
Last metadata expiration check: 0:53:05 ago on Thu 07 Mar 2019 09:42:26 GMT.
Dependencies resolved.
================================================================================
Package Architecture Version Repository Size
================================================================================
Installing:
glibc-headers i686 2.29-7.fc30 rawhide 475 k
Downgrading:
glibc i686 2.29-7.fc30 rawhide 3.7 M
glibc x86_64 2.29-7.fc30 rawhide 3.9 M
glibc-all-langpacks x86_64 2.29-7.fc30 rawhide 25 M
glibc-common x86_64 2.29-7.fc30 rawhide 822 k
glibc-devel i686 2.29-7.fc30 rawhide 1.0 M
glibc-devel x86_64 2.29-7.fc30 rawhide 1.0 M
glibc-headers x86_64 2.29-7.fc30 rawhide 474 k
glibc-langpack-en x86_64 2.29-7.fc30 rawhide 820 k
glibc-static x86_64 2.29-7.fc30 rawhide 1.9 M

Transaction Summary
================================================================================
Install 1 Package
Downgrade 9 Packages

Total download size: 39 M
Is this ok [y/N]: y
Downloading Packages:
(1/10): glibc-2.29-7.fc30.x86_64.rpm 661 kB/s | 3.9 MB 00:06
(2/10): glibc-2.29-7.fc30.i686.rpm 610 kB/s | 3.7 MB 00:06
(3/10): glibc-common-2.29-7.fc30.x86_64.rpm 737 kB/s | 822 kB 00:01
(4/10): glibc-devel-2.29-7.fc30.i686.rpm 675 kB/s | 1.0 MB 00:01
(5/10): glibc-devel-2.29-7.fc30.x86_64.rpm 926 kB/s | 1.0 MB 00:01
(6/10): glibc-headers-2.29-7.fc30.x86_64.rpm 609 kB/s | 474 kB 00:00
(7/10): glibc-langpack-en-2.29-7.fc30.x86_64.rp 968 kB/s | 820 kB 00:00
(8/10): glibc-headers-2.29-7.fc30.i686.rpm 934 kB/s | 475 kB 00:00
(9/10): glibc-static-2.29-7.fc30.x86_64.rpm 882 kB/s | 1.9 MB 00:02
(10/10): glibc-all-langpacks-2.29-7.fc30.x86_64 1.3 MB/s | 25 MB 00:18
Error in <unknown> scriptlet in rpm package glibc-common
error: failed to exec scriptlet interpreter /bin/sh: No such file or directory
warning: %triggerpostun(glibc-common-2.29-7.fc30.x86_64) scriptlet failed, exit status 127

Error in <unknown> scriptlet in rpm package glibc-common
error: failed to exec scriptlet interpreter /bin/sh: No such file or directory
warning: %triggerpostun(info-6.5-12.fc30.x86_64) scriptlet failed, exit status 127

Error in <unknown> scriptlet in rpm package glibc-common
Running scriptlet: glibc-common-2.29-7.fc30.x86_64 19/19
error: failed to exec scriptlet interpreter /bin/sh: No such file or directory
warning: %triggerin(glibc-common-2.29-7.fc30.x86_64) scriptlet failed, exit status 127

Error in <unknown> scriptlet in rpm package glibc-common
Verifying : glibc-2.29-7.fc30.i686 1/19
Verifying : glibc-2.29.9000-4.fc31.i686 2/19
Verifying : glibc-2.29-7.fc30.x86_64 3/19
Verifying : glibc-2.29.9000-4.fc31.x86_64 4/19
Verifying : glibc-all-langpacks-2.29-7.fc30.x86_64 5/19
Verifying : glibc-all-langpacks-2.29.9000-4.fc31.x86_64 6/19
Verifying : glibc-common-2.29-7.fc30.x86_64 7/19
Verifying : glibc-common-2.29.9000-4.fc31.x86_64 8/19
Verifying : glibc-devel-2.29-7.fc30.i686 9/19
Verifying : glibc-devel-2.29.9000-4.fc31.i686 10/19
Verifying : glibc-devel-2.29-7.fc30.x86_64 11/19
Verifying : glibc-devel-2.29.9000-4.fc31.x86_64 12/19
Verifying : glibc-headers-2.29-7.fc30.x86_64 13/19
Verifying : glibc-headers-2.29.9000-4.fc31.x86_64 14/19
Verifying : glibc-langpack-en-2.29-7.fc30.x86_64 15/19
Verifying : glibc-langpack-en-2.29.9000-4.fc31.x86_64 16/19
Verifying : glibc-static-2.29-7.fc30.x86_64 17/19
Verifying : glibc-static-2.29.9000-4.fc31.x86_64 18/19
Verifying : glibc-headers-2.29-7.fc30.i686 19/19

Downgraded:
glibc-2.29-7.fc30.i686 glibc-2.29-7.fc30.x86_64
glibc-all-langpacks-2.29-7.fc30.x86_64 glibc-common-2.29-7.fc30.x86_64
glibc-devel-2.29-7.fc30.i686 glibc-devel-2.29-7.fc30.x86_64
glibc-headers-2.29-7.fc30.x86_64 glibc-langpack-en-2.29-7.fc30.x86_64
glibc-static-2.29-7.fc30.x86_64

Installed:
glibc-headers-2.29-7.fc30.i686

Complete!

$ ll /bin/sh
-bash: /usr/bin/ls: No such file or directory
$ sudo dnf install /bin/sh
-bash: /usr/bin/sudo: No such file or directory

Rich.

Comments

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Florian Weimer at 03/07/2019 - 07:13

* Richard W. M. Jones:

That looks like a bug in itself.

The last time I looked at something similar, I saw this: RPM would not
adjust a pre-existing symbolic link to a new target very late in the
transaction. Like deleting old files which are gone in an update or
downgrade, this does *not* happen when the unpacking of the replacement
package happens, but towards the conclusion of the transaction. In the
meantime, scriptlets run with the broken file system.

In your case, maybe one of the scriptlet errors prevented the final step
with the adjustment of the symbolic link by RPM.

(Just to be clear, the symbolic link is regularly packaged, it's not
something that we manage using scripts.)

Thanks,
Florian

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Panu Matilainen at 03/07/2019 - 08:41

On 3/7/19 1:13 PM, Florian Weimer wrote:
IIRC the issue is that at when ldconfig runs from the package scripts,
on downgrade the newer file is still on disk and thus ldconfig leaves
the link the way it is, but at the end of transaction it'll be gone and
symlinks can be broken.

$ rpm -q --filetriggers glibc-common
transfiletriggerin scriptlet (using /bin/sh) -- /lib, /lib64, /usr/lib,
/usr/lib64
/sbin/ldconfig
transfiletriggerpostun scriptlet (using /bin/sh) -- /lib, /lib64,
/usr/lib, /usr/lib64
/sbin/ldconfig

The %transfiletriggerpostun would've probably fixed it if it used -p
<lua> instead of shell.

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Florian Weimer at 03/07/2019 - 11:52

* Panu Matilainen:

Is there a chance that RPM will be changed to run more scriptlets with
the final disk contents?

We switched to the shell for the benefit of rpm-ostree.

Thanks,
Florian

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Colin Walters at 03/08/2019 - 09:41

On Thu, Mar 7, 2019, at 10:53 AM, Florian Weimer wrote:
Short answer: <a href="https://github.com/projectatomic/rpm-ostree/issues/749#issuecomment-470926180" title="https://github.com/projectatomic/rpm-ostree/issues/749#issuecomment-470926180">https://github.com/projectatomic/rpm-ostree/issues/749#issuecomment-4709...</a>
(Let me know if that would help something, here or in the issue; it wouldn't be hard for us to implement)

Slightly longer answer: There are *multiple* layers of defenses in rpm-ostree against exactly this sort of problem. The reason we don't implement Lua is (as the later linked issue spells out in more detail) is to ensure we can reliably construct a *new* root filesystem in a transactional model without having any potential effects on your running filesystem.

The fact that with rpm-ostree you always have a "base image" that is constructed on the server side is the first primary line of defense; we won't ship you a new image (ostree commit) that isn't known good.

The second line of defense is that *client side* rpm-ostree always runs a "sanity check" in the new deployment:
<a href="https://github.com/projectatomic/rpm-ostree/pull/892" title="https://github.com/projectatomic/rpm-ostree/pull/892">https://github.com/projectatomic/rpm-ostree/pull/892</a>

The third line of defense is that somehow even if you do get a broken tree, you always have the previous one in the bootloader.

But we do support both layering and overrides - you can engage the package system side.

Now personally if I wanted to test a new glibc I'd first do it in a disposable clone of my development podman container, rather than replacing the one on the root filesystem of my host. But let's assume for whatever reason we do need to test an updated glibc on the host system; rpm-ostree supports that too:

[root@localhost ~]# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-atomic:fedora/29/x86_64/atomic-host
Version: 29.20190306.2 (2019-03-06T23:32:07Z)
Commit: 57297da7779ed8b7b7b9a0f39f6f12a703000a40cf451770fe23749a5558f60d
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4

ostree://fedora-atomic:fedora/29/x86_64/atomic-host
Version: 29.20190219.0 (2019-02-19T04:52:26Z)
Commit: d00adf110907f93f6cdd05deda0e2878c9bd71c74e0c4c2e9a5250d2f4cc8868
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
[root@localhost ~]# rpm -q glibc
glibc-2.28-26.fc29.x86_64

(Here I browse to <a href="https://koji.fedoraproject.org/koji/packageinfo?packageID=57" title="https://koji.fedoraproject.org/koji/packageinfo?packageID=57">https://koji.fedoraproject.org/koji/packageinfo?packageID=57</a> and find the previous glibc built for f29):

[root@localhost ~]# rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/glibc/2.28/25.fc29/x86_64/glibc-{common-,}2.28-25.fc29.x86_64.rpm
Downloading 'https://kojipkgs.fedoraproject.org//packages/glibc/2.28/25.fc29/x86_64/glibc-common-2.28-25.fc29.x86_64.rpm'... done!
Downloading 'https://kojipkgs.fedoraproject.org//packages/glibc/2.28/25.fc29/x86_64/glibc-2.28-25.fc29.x86_64.rpm'... done!
Checking out tree 57297da... done
Enabled rpm-md repositories: updates fedora fahc
rpm-md repo 'updates' (cached); generated: 2019-03-07T20:40:34Z
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'fahc' (cached); generated: 2019-03-07T05:41:31Z
Importing rpm-md... done
Resolving dependencies... done
error: Could not depsolve transaction; 2 problems detected:
0. nothing provides glibc-langpack = 2.28-25.fc29 needed by glibc-2.28-25.fc29.x86_64
1. nothing provides glibc-langpack = 2.28-25.fc29 needed by glibc-2.28-25.fc29.x86_64

Hmm. Oh right, this is libsolv's way of telling me I forgot a package probably...
Ah, yep, I need to also take in glibc-all-langpacks:

[root@localhost ~]# rpm-ostree override replace https://kojipkgs.fedoraproject.org//packages/glibc/2.28/25.fc29/x86_64/glibc-{common-,all-langpacks-,}2.28-25.fc29.x86_64.rpm
Downloading 'https://kojipkgs.fedoraproject.org//packages/glibc/2.28/25.fc29/x86_64/glibc-common-2.28-25.fc29.x86_64.rpm'... done!
Downloading 'https://kojipkgs.fedoraproject.org//packages/glibc/2.28/25.fc29/x86_64/glibc-all-langpacks-2.28-25.fc29.x86_64.rpm'... done!
Downloading 'https://kojipkgs.fedoraproject.org//packages/glibc/2.28/25.fc29/x86_64/glibc-2.28-25.fc29.x86_64.rpm'... done!
Checking out tree 57297da... done
...
Applying 3 overrides
Processing packages... done
Running pre scripts... done
Running post scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 227.0 MB (pkgcache branches: 3)
Downgraded:
glibc 2.28-26.fc29 -> 2.28-25.fc29
glibc-all-langpacks 2.28-26.fc29 -> 2.28-25.fc29
glibc-common 2.28-26.fc29 -> 2.28-25.fc29
Run "systemctl reboot" to start a reboot
[root@localhost ~]# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
ostree://fedora-atomic:fedora/29/x86_64/atomic-host
Version: 29.20190306.2 (2019-03-06T23:32:07Z)
BaseCommit: 57297da7779ed8b7b7b9a0f39f6f12a703000a40cf451770fe23749a5558f60d
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
ReplacedBasePackages: glibc-all-langpacks glibc-common glibc 2.28-26.fc29 -> 2.28-25.fc29

● ostree://fedora-atomic:fedora/29/x86_64/atomic-host
Version: 29.20190306.2 (2019-03-06T23:32:07Z)
Commit: 57297da7779ed8b7b7b9a0f39f6f12a703000a40cf451770fe23749a5558f60d
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4

ostree://fedora-atomic:fedora/29/x86_64/atomic-host
Version: 29.20190219.0 (2019-02-19T04:52:26Z)
Commit: d00adf110907f93f6cdd05deda0e2878c9bd71c74e0c4c2e9a5250d2f4cc8868
GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
[root@localhost ~]#

Now, I also can see we executed the transfiletriggerin in question:

[root@localhost ~]# journalctl -b -r -u rpm-ostreed | grep transfiletriggerin | head -1
Mar 08 13:36:37 localhost.localdomain rpm-ostree[1753]: Executed %transfiletriggerin(glibc-common) for lib, lib64, usr/lib, usr/lib64 in 203ms; 24842 matched files

So yes, I would need to reboot to test it now, but on the flip side rpm-ostree carefully maintained the integrity of my system, and didn't touch my booted deployment. This property relies on not executing arbitrary lua code in the context of the daemon.

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Panu Matilainen at 03/08/2019 - 04:20

On 3/7/19 5:52 PM, Florian Weimer wrote:
There's %transfiletriggerin, %transfiletriggerpostun and %posttrans that
run after the entire transaction, and then the individual postun-type
scripts/triggers. What is it that you're missing?

Sigh...

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Florian Weimer at 03/08/2019 - 07:54

* Panu Matilainen:

Correct symbolic links.

If the symbolic links are not installed as they are in the packaged
contents before running scriptlets (any scriptlets), we need to bring
back the ldconfig scriptlets. This is not just a problem for glibc.

Thanks,
Florian

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Panu Matilainen at 03/08/2019 - 08:29

On 3/8/19 1:54 PM, Florian Weimer wrote:
Ehm? Symlinks are installed just like any other content, and rpm has no
reason to mess with them. Like I said above, the problem is almost
certainly a mistimed ldconfig causing the links changed back to the
version that is about to get removed.

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Panu Matilainen at 03/08/2019 - 08:52

On 3/8/19 2:29 PM, Panu Matilainen wrote:
Hmm, except that in this particular there shouldn't be any mistimed
ldconfigs present. Perhaps I should take a closer look.

Note that the scenario with ldconfig is very real though, as long as
*any* package runs ldconfig in middle of a transaction involving
downgrades, the links can get misadjusted to version(s) that will be
removed, and unless ldconfig is run again after that removal, things
will be broken. Back when each and every library package ran ldconfig in
its post and postun, things also got fixed right away. Now that some do
and others dont, its more of a mixed bag. Either ldconfig should only
run once at the very end, or it should run after each library package.

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Panu Matilainen at 03/08/2019 - 10:06

On 3/8/19 2:52 PM, Panu Matilainen wrote:
It's glibc's own %post own scripts that are somehow breaking it. I've a
minimal reproducer here with glibc 2.29 in /srv/root chroot. The bash
version is just to show whether bash is alive or not:

[root@sopuli glibc]# chroot /srv/test bash --version|head -1
GNU bash, version 4.4.23(1)-release (x86_64-redhat-linux-gnu)
[root@sopuli glibc]# rpm -Uv --oldpackage --root /srv/test
glibc-2.28-26.fc29.x86_64.rpm glibc-common-2.28-26.fc29.x86_64.rpm
glibc-minimal-langpack-2.28-26.fc29.x86_64.rpm
Verifying packages...
Preparing packages...
glibc-common-2.28-26.fc29.x86_64
glibc-minimal-langpack-2.28-26.fc29.x86_64
glibc-2.28-26.fc29.x86_64
glibc-2.29-8.fc30.x86_64
glibc-minimal-langpack-2.29-8.fc30.x86_64
glibc-common-2.29-8.fc30.x86_64
error: failed to exec scriptlet interpreter /bin/sh: No such file or
directory
warning: %triggerpostun(glibc-common-2.28-26.fc29.x86_64) scriptlet
failed, exit status 127
error: failed to exec scriptlet interpreter /bin/sh: No such file or
directory
warning: %triggerin(glibc-common-2.28-26.fc29.x86_64) scriptlet failed,
exit status 127
[root@sopuli glibc]# chroot /srv/test bash --version|head -1
chroot: failed to run command ‘bash’: No such file or directory

So, exactly the same as the original posting. Now, lets go back to the
original scenario, and run the same downgrade with --nopost:

[root@sopuli glibc]# rpm -U --root /srv/test
glibc-2.29-8.fc30.x86_64.rpm glibc-common-2.29-8.fc30.x86_64.rpm
glibc-minimal-langpack-2.29-8.fc30.x86_64.rpm
warning: glibc-2.29-8.fc30.x86_64.rpm: Header V3 RSA/SHA256 Signature,
key ID cfc659b9: NOKEY
[root@sopuli glibc]# chroot /srv/test bash --version|head -1
GNU bash, version 4.4.23(1)-release (x86_64-redhat-linux-gnu)
[root@sopuli glibc]# rpm -Uv --oldpackage --nopost --root /srv/test
glibc-2.28-26.fc29.x86_64.rpm glibc-common-2.28-26.fc29.x86_64.rpm
glibc-minimal-langpack-2.28-26.fc29.x86_64.rpm
Verifying packages...
Preparing packages...
glibc-common-2.28-26.fc29.x86_64
glibc-minimal-langpack-2.28-26.fc29.x86_64
glibc-2.28-26.fc29.x86_64
glibc-2.29-8.fc30.x86_64
glibc-minimal-langpack-2.29-8.fc30.x86_64
glibc-common-2.29-8.fc30.x86_64
[root@sopuli glibc]# chroot /srv/test bash --version|head -1
GNU bash, version 4.4.23(1)-release (x86_64-redhat-linux-gnu)
[root@sopuli glibc]#

glibc's %post is /usr/sbin/glibc_post_upgrade.<arch>, so that's what's
doing something bad here. When straced through forks and all, guess
what? It's running ldconfig:

22611 execve("/usr/sbin/glibc_post_upgrade.x86_64",
["/usr/sbin/glibc_post_upgrade.x86"...], 0x7ffc249bfc00 /* 50 vars */) = 0
22611 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffd33d5ed40) = -1 EINVAL
(Invalid argument)
22611 brk(NULL) = 0x7f42ba930000
22611 brk(0x7f42ba9311c0) = 0x7f42ba9311c0
22611 arch_prctl(ARCH_SET_FS, 0x7f42ba930880) = 0
22611 uname({sysname="Linux", nodename="sopuli.koti.laiskiainen.org",
...}) = 0
22611 readlink("/proc/self/exe", 0x7ffd33d5dc50, 4096) = -1 ENOENT (No
such file or directory)
22611 brk(0x7f42ba9521c0) = 0x7f42ba9521c0
22611 brk(0x7f42ba953000) = 0x7f42ba953000
22611 openat(AT_FDCWD, "/etc/ld.so.conf", O_RDONLY) = 3
22611 fstat(3, {st_mode=S_IFREG|0644, st_size=28, ...}) = 0
22611 read(3, "include ld.so.conf.d/*.conf\n", 28) = 28
22611 close(3) = 0
22611 access("/sbin/ldconfig", X_OK) = 0
22611 vfork( <unfinished ...>
22612 execve("/sbin/ldconfig", ["/sbin/ldconfig"], 0x7ffd33d5ee08 /* 50
vars */ <unfinished ...>

So it's exactly the situtation I explained where a mistimed ldconfig
breaks things. No mystery there.

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Florian Weimer at 03/11/2019 - 06:29

* Panu Matilainen:

Yes, you are right, I had actually looked at this failure a few weeks
ago and reached similar conclusions (my memory is failing).

Right, we have to do that unfortunately.

So the problem is not the symbolic link handling, but the fact that the
scriptlets run while old files still stick around, before RPM deletes
them closer to the end of the transaction. And *this* happens because
we use symbolic links to the actual implementation. If we didn't do
that, RPM would have no choice and would have to replace the files on
disk, so that the old version is gone.

We actually have compensating code in glibc_post_upgrade for these
lingering files, deleting files early that would break scriptlets which
run before they are deleted by RPM.

The question is whether we want to extend this code to cover more cases.
This is quite hard to do however, especially for downgrades, because we
do not know which files to delete in the %post scriptlet of the old
version (which cannot foretell the changes the newer glibc version that
is being removed brought in). Presumably, we could look at the file
list in the RPM database and delete anything that's not part of the
current package version (the one whose %post is running).

I had already raised the issue with the symbolic links upstream (as I
said, my memory is failing), and feedback was not exactly positive:

<https://sourceware.org/ml/libc-alpha/2018-11/msg00612.html>

In fact, Siddhesh suggested using *more* symbolic links:

<https://sourceware.org/ml/libc-alpha/2018-12/msg00348.html>

What's RPM's justification for deleting files so late in the
transaction? Can this be changed at the RPM level? I'm sure we aren't
the only ones affected by this.

Thanks,
Florian

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Panu Matilainen at 03/12/2019 - 06:53

On 3/11/19 12:29 PM, Florian Weimer wrote:
The answer is basically twofold:

1) Rpm cannot do removals as long as there are dependencies on the older
versions, eg on library soname changes an early erasure could cause
scriptlet failures from dependent packages.

2) Hardcoded removals after install has the virtue of simplicity,
both in terms of implementation and being rather idiot-proof: installs
and erasures could be safely interleaved IFF the dependency data from
packages is perfect. For install-scripts it usually is close enough, but
much less so for erasure. Given the potentially dramatic outcome of
early erasure gone wrong, simple-and-stupid doesn't seem so stupid.

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Florian Weimer at 03/13/2019 - 08:57

* Panu Matilainen:

You mean the %postun fails for a dependency because it runs after the
upgrade to the new soname, and the new soname is gone at that point?

(This obviously does not apply to glibc anytime soon.)

I don't understand this point. It is possible interleave file system
changes arbitrarily until you run scriptlets. But RPM does not do this
because it runs non-trigger scriptlets immediately, so it rarely
provides a consistent execution environment for scriptlets.

Thanks,
Florian

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By King InuYasha at 03/08/2019 - 05:02

On Fri, Mar 8, 2019 at 3:21 AM Panu Matilainen < ... at redhat dot com> wrote:
Will this be the motivator for rpm-ostree to finally support Lua scriptlets[1]?

[1]: <a href="https://github.com/projectatomic/rpm-ostree/issues/749" title="https://github.com/projectatomic/rpm-ostree/issues/749">https://github.com/projectatomic/rpm-ostree/issues/749</a>

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Richard W.M. Jones at 03/07/2019 - 07:39

On Thu, Mar 07, 2019 at 12:13:22PM +0100, Florian Weimer wrote:
I fixed this by typing ‘ldconfig’ (thankfully I had a root shell open
on the IPMI console ...)

Rich.

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Richard W.M. Jones at 03/07/2019 - 06:45

Actually it's more subtle. It didn't remove the files, but it did
break something really fundamental, perhaps execv? Perhaps new
binaries cannot link with the slightly older glibc?

$ echo /usr/bin/ls
/usr/bin/ls
$ /usr/bin/ls
-bash: /usr/bin/ls: No such file or directory

Rich.

Re: Downgrading glibc from Rawhide removed /bin/sh (!)

By Hans de Goede at 03/07/2019 - 06:57

Hi,

On 07-03-19 11:45, Richard W.M. Jones wrote:
In my experience with breaking systems through glibc changes,
the "No such file or directory" error is not about ls, it
is about /lib64/ld-linux.so.2 missing.

On my system I have:

[hans@shalem ~]$ ls -l /lib/ld-linux.so.2
lrwxrwxrwx. 1 root root 10 14 dec 01:46 /lib/ld-linux.so.2 -> ld-2.28.so

I suspect the symlink somehow ended up broken or missing
due to the downgrade.

Regards,

Hans