DevHeads.net

/lib/firmware/microcode.dat update on CentOS 6

Dear All,

An update just brought on my CenOS 6 boxes updated microcode.dat files:

/lib/firmware/microcode.dat

Does anybody know off hand what (how critical) is that, as, if it is
related to most famous these days trouble with CPU hardware, I will need
to reboot relevant boxes to have new microcode loaded. But if it is not
that critical, it can wait till next reboot.

Thanks a lot and apologies for laziness (not looking into details of
this particular update myself).

Valeri

++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++

Comments

Re: /lib/firmware/microcode.dat update on CentOS 6

By Johnny Hughes at 01/17/2018 - 18:22

On 01/17/2018 03:24 PM, Valeri Galtsev wrote:
As Phil said, this basically REMOVES the new microcode.dat file and
reverts to the last stable one before the spectre / meltdown update set.

Please see my tweet about this:

<a href="https://twitter.com/JohnnyCentOS/status/953734648764477440" title="https://twitter.com/JohnnyCentOS/status/953734648764477440">https://twitter.com/JohnnyCentOS/status/953734648764477440</a>

For those of you without twitter (WHAT .. who doesn't have twitter :D )

Look at:

<a href="https://t.co/6fT61xgtGH" title="https://t.co/6fT61xgtGH">https://t.co/6fT61xgtGH</a>

Get the latest microcode.dat file from here:

<a href="https://t.co/zPwagbeJFY" title="https://t.co/zPwagbeJFY">https://t.co/zPwagbeJFY</a>

See how to update the microcode from the links at the bottom of this page:

<a href="https://t.co/EOgclWdHCw" title="https://t.co/EOgclWdHCw">https://t.co/EOgclWdHCw</a>

An before anyone asks .. I have no idea why Red Hat chose this path,
they did. It doesn't matter if I (or anyone else) agrees with the
decision. It is what it is.

Thanks.
Johnny Hughes

Re: /lib/firmware/microcode.dat update on CentOS 6

By Pete Biggs at 01/18/2018 - 05:41

But can I just clarify. We have to *manually* install the microcode
update an EL7 in order to be protected against Spectre? EL6 as well?

Presumably this is to remove RH from the loop and to stop people
blaming them - i.e. this is between Intel and the customer, it's
nothing to do with them.

What about future microcode updates? They come out reasonably regularly
(2 or 3 times a year) - are RH going to absolve themselves from all
future updates because presumably the next update will also contain the
Spectre fixes?

So, before I re-invent the wheel, does anyone have automation scripts
to do the microcode update? I don't relish the prospect of doing this
manually on a couple of hundred machines. Is it reasonable to grab the
microcode_ctl SRPM and create my own updated RPM to do it?

P.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Valeri Galtsev at 01/18/2018 - 11:42

On 01/18/18 03:41, Pete Biggs wrote:
I bet you are right. I was going to rant about that... then it occurred
to me that class action against Intel (didn't hear about AMD though) is
quite likely, so, indeed, RedHat does not want to be even mentioned in
it, which will be unfair, especially after RedHat putting effort into
fixing somebody's else crap.

Valeri

Re: /lib/firmware/microcode.dat update on CentOS 6

By Johnny Hughes at 01/22/2018 - 11:08

On 01/18/2018 09:42 AM, Valeri Galtsev wrote:
It isn't about washing hands, lawsuits, or soeoen else's stuff. It is
about broken microcode updates.

The code from intel was broken .. causing several CPUs not to boot after
update. That (and only that) is why they were pulled.

Users MUST individually QA the microcode for their individual CPUs, OEM
Frimware, chipset etc.

The issue here is that the microcode broke peoples machines .. therefore
it had to be rolled back. All the other discussion is full and total BS.

It is likely ONCE all the microcode updates are tested and completely
working that Red Hat will again include it in the microcode_ctl RPM ..
but that can't put stuff in there that is breaking machines.

While things are beuing released as QA quality, they are going to have
to be done individually by admins .. that's just how it is.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Valeri Galtsev at 01/22/2018 - 12:06

On 01/22/18 09:08, Johnny Hughes wrote:
Thanks Johnny, for correcting my wild guess which was wrong, and the
fact that my guess was wrong I realized after reading your other post!

As with everything else the end user pays one way or another. :-(

Valeri

++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++

Re: /lib/firmware/microcode.dat update on CentOS 6

By Johnny Hughes at 01/23/2018 - 07:26

On 01/22/2018 10:06 AM, Valeri Galtsev wrote:
Here are a couple of posts for our reading pleasure:

Intel recommends not installing the microcode now:
<a href="http://intel.ly/2DsL9qz" title="http://intel.ly/2DsL9qz">http://intel.ly/2DsL9qz</a>

Linus Torvalds agrees:
<a href="http://tcrn.ch/2n2mEcA" title="http://tcrn.ch/2n2mEcA">http://tcrn.ch/2n2mEcA</a>

So, wait on those microcode updates for a while.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Chris Murphy at 01/24/2018 - 14:06

Except this doesn't mention microcode at all. I can't even tell WTF
they're recommending not doing in this doc, it's that badly written.
You have to infer, by reading two prior docs, that they're referring
to microcode. And then you have to assume that's still what they're
referring to when they say:

"We recommend that OEMs, cloud service providers, system
manufacturers, software vendors and end users stop deployment of
current versions." Current versions of what? Microcode?

But yes, indeed they appear to have pulled the 20180108 microcode,
which was previously set to latest at this link, and it is now
reverted to the 20171117 microcode.

<a href="https://downloadcenter.intel.com/download/27337/Linux-Processor-Microcode-Data-File?v=t" title="https://downloadcenter.intel.com/download/27337/Linux-Processor-Microcode-Data-File?v=t">https://downloadcenter.intel.com/download/27337/Linux-Processor-Microcod...</a>

What these means for people who have CPUs which were not crashing
(rebooting being a new euphemism for crashing) , but saw variant 2
Spectre mitigation with the 20180108 microcode, will lose full
mitigation until Intel gets its ducks into a row.

*eye roll*

His comments aren't about microcode though. And it also looks like he
got IBRS and IBPB confused. The better post on this front is

<a href="https://lkml.org/lkml/2018/1/22/598" title="https://lkml.org/lkml/2018/1/22/598">https://lkml.org/lkml/2018/1/22/598</a>

As far as I know, there still is no mitigation for Spectre variant 1.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Chris Adams at 01/24/2018 - 17:22

Once upon a time, Chris Murphy < ... at colorremedies dot com> said:
Well, that's the only thing Intel provides for CPUs, so that's all it
can be.

Lots of people weren't seeing issues, but that's in part because Intel's
updated microcode release only actually updated microcode for recent
CPUs. I have many servers that aren't crashing, but that's because
Intel hasn't actually even tried to fix the microcode for their CPUs
yet.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Leon Fauster at 01/24/2018 - 20:41

Comparing microcode-20171117 with microcode-20180108 shows that
from the 94 ucode files only 19 where updated

$ diff -r --brief microcode-20171117 microcode-20180108
Files microcode-20171117/intel-ucode/06-3c-03 and microcode-20180108/intel-ucode/06-3c-03 differ
Files microcode-20171117/intel-ucode/06-3d-04 and microcode-20180108/intel-ucode/06-3d-04 differ
Files microcode-20171117/intel-ucode/06-3e-04 and microcode-20180108/intel-ucode/06-3e-04 differ
Files microcode-20171117/intel-ucode/06-3f-02 and microcode-20180108/intel-ucode/06-3f-02 differ
Files microcode-20171117/intel-ucode/06-3f-04 and microcode-20180108/intel-ucode/06-3f-04 differ
Files microcode-20171117/intel-ucode/06-45-01 and microcode-20180108/intel-ucode/06-45-01 differ
Files microcode-20171117/intel-ucode/06-46-01 and microcode-20180108/intel-ucode/06-46-01 differ
Files microcode-20171117/intel-ucode/06-47-01 and microcode-20180108/intel-ucode/06-47-01 differ
Files microcode-20171117/intel-ucode/06-4e-03 and microcode-20180108/intel-ucode/06-4e-03 differ
Files microcode-20171117/intel-ucode/06-55-04 and microcode-20180108/intel-ucode/06-55-04 differ
Files microcode-20171117/intel-ucode/06-56-02 and microcode-20180108/intel-ucode/06-56-02 differ
Files microcode-20171117/intel-ucode/06-56-03 and microcode-20180108/intel-ucode/06-56-03 differ
Files microcode-20171117/intel-ucode/06-5e-03 and microcode-20180108/intel-ucode/06-5e-03 differ
Files microcode-20171117/intel-ucode/06-7a-01 and microcode-20180108/intel-ucode/06-7a-01 differ
Files microcode-20171117/intel-ucode/06-8e-09 and microcode-20180108/intel-ucode/06-8e-09 differ
Files microcode-20171117/intel-ucode/06-8e-0a and microcode-20180108/intel-ucode/06-8e-0a differ
Files microcode-20171117/intel-ucode/06-9e-09 and microcode-20180108/intel-ucode/06-9e-09 differ
Files microcode-20171117/intel-ucode/06-9e-0a and microcode-20180108/intel-ucode/06-9e-0a differ
Files microcode-20171117/intel-ucode/06-9e-0b and microcode-20180108/intel-ucode/06-9e-0b differ
Files microcode-20171117/microcode.dat and microcode-20180108/microcode.dat differ
Files microcode-20171117/releasenote and microcode-20180108/releasenote differ

Microcode ID?

$ awk '/cpu family/||/model\t/||/stepping/' /proc/cpuinfo |sort |uniq

and convert it into hex

Re: /lib/firmware/microcode.dat update on CentOS 6

By Robert Arkiletian at 01/25/2018 - 13:16

On Wed, Jan 24, 2018 at 4:41 PM, Leon Fauster
< ... at googlemail dot com> wrote:
...
Thanks for this info Leon. Very helpful. I was trying to figure this
out. Intel should make this clear on their microcode download page.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Robert Arkiletian at 04/05/2018 - 15:50

There is a new guidance document from Intel here

<a href="https://newsroom.intel.com/wp-content/uploads/sites/11/2018/04/microcode-update-guidance.pdf" title="https://newsroom.intel.com/wp-content/uploads/sites/11/2018/04/microcode-update-guidance.pdf">https://newsroom.intel.com/wp-content/uploads/sites/11/2018/04/microcode...</a>

Changes since the previous release are highlighted in yellow.
Unfortunately, the latest microcode here

<a href="https://downloadcenter.intel.com/download/27591/Linux-Processor-Microcode-Data-File?product=873" title="https://downloadcenter.intel.com/download/27591/Linux-Processor-Microcode-Data-File?product=873">https://downloadcenter.intel.com/download/27591/Linux-Processor-Microcod...</a>

does not seem to contain all the yellow updates (at least for my
hardware). The document is from April 2 and the latest microcode
update is from March 12.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Leroy Tennison at 01/24/2018 - 14:38

What's amazing to me is, after "Intel Inside - don't divide" (their 486 debacle), they didn't learn and have a better plan for addressing these kinds of things.

Except this doesn't mention microcode at all. I can't even tell WTF
they're recommending not doing in this doc, it's that badly written.
You have to infer, by reading two prior docs, that they're referring
to microcode. And then you have to assume that's still what they're
referring to when they say:

"We recommend that OEMs, cloud service providers, system
manufacturers, software vendors and end users stop deployment of
current versions." Current versions of what? Microcode?

But yes, indeed they appear to have pulled the 20180108 microcode,
which was previously set to latest at this link, and it is now
reverted to the 20171117 microcode.

<a href="https://downloadcenter.intel.com/download/27337/Linux-Processor-Microcode-Data-File?v=t" title="https://downloadcenter.intel.com/download/27337/Linux-Processor-Microcode-Data-File?v=t">https://downloadcenter.intel.com/download/27337/Linux-Processor-Microcod...</a>

What these means for people who have CPUs which were not crashing
(rebooting being a new euphemism for crashing) , but saw variant 2
Spectre mitigation with the 20180108 microcode, will lose full
mitigation until Intel gets its ducks into a row.

*eye roll*

His comments aren't about microcode though. And it also looks like he
got IBRS and IBPB confused. The better post on this front is

<a href="https://lkml.org/lkml/2018/1/22/598" title="https://lkml.org/lkml/2018/1/22/598">https://lkml.org/lkml/2018/1/22/598</a>

As far as I know, there still is no mitigation for Spectre variant 1.

Re: /lib/firmware/microcode.dat update on CentOS 6

By m.roth at 01/24/2018 - 16:20

Leroy Tennison wrote:
mark

Re: /lib/firmware/microcode.dat update on CentOS 6

By Valeri Galtsev at 01/24/2018 - 15:02

On Wed, January 24, 2018 12:38 pm, Leroy Tennison wrote:
It probably would be fair to conclude that they didn't plan to address
these things at all ;-)

Valeri

++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++

Re: /lib/firmware/microcode.dat update on CentOS 6

By Johnny Hughes at 01/18/2018 - 06:03

On 01/18/2018 03:41 AM, Pete Biggs wrote:
No, this is because at least one major CPU (Intel type 79) is completely
broken by the Intel Microcode Update. Those machines can't boot after
the microcode rpm is installed. It impacts at least these processors:

Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz

There may be others.

<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1532283" title="https://bugzilla.redhat.com/show_bug.cgi?id=1532283">https://bugzilla.redhat.com/show_bug.cgi?id=1532283</a>

That means, it is NOT full-proof install and likely leaves many
servers/machines broken. I suppose that the decision is, go pack to
what works for all machines (the last known good install) and let admins
work with their hardware vendor because the alternative is breaking things.

This also needs to be fixed with a combination of firmware updates AND
microcode updates. All of that is outside the expertise of a Linux
vendor and is unique for each processor, chipset and firmware combination.

How they handle it in the future I have no way of knowing, but if you
had 20,000 servers with the impacted CPU and you updated and could not
reboot, I would assume that you did not appreciate it.

That is what I have found so far with a bit of research.

This is NOTHING about who to blame and everything about stable, working
updates .. at least it seems so to me.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Peter Kjellstrom at 01/18/2018 - 12:27

On Thu, 18 Jan 2018 04:03:48 -0600

As a data point, we have the updated microcode running on 600+ Haswell
servers and so far no indication of problems.

We'll keep the ibrs/spectre mitigation this gives us and not revert
(unless it turns out it does cause problems).

/Peter

Re: /lib/firmware/microcode.dat update on CentOS 6

By Phelps, Matt at 01/18/2018 - 09:51

Also, do we know if the updated CentOS microcode RPM reverted the microcode
for *all* Intel CPUs, or just the ones that had issues? In other words, if
I apply the latest microcode update to our 100+ machines (which all have
the previous update, and are OK) will they revert to a vulnerable state?

Re: /lib/firmware/microcode.dat update on CentOS 6

By Johnny Hughes at 01/18/2018 - 10:01

On 01/18/2018 07:51 AM, Phelps, Matthew wrote:
It reverted for all .. but, your machines may or may not be protected as
only a subset of machines were updated with the original microcode from
Intel.

It is your call as to what you install .. but the correct method is to
install the current microcode_ctl .. and then research your specific
machine, its CPU, chipset, firmware .. go to the vendor and make sure
you get all the things necessary to mitigate the issues. It will be
different for each CPU vendor (Intel or AMD), each CPU / Chipset combo,
and even each vendor (Dell may have new firmware for x and y but not z
models, etc.)

There is no one size fits all update for this issue.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Pete Geenhuizen at 01/18/2018 - 12:01

On 01/18/18 09:01, Johnny Hughes wrote:
Do we update the microcode now or do we wait until the latest
microcode_ctl rpm is available and then tackle this issue?

Re: /lib/firmware/microcode.dat update on CentOS 6

By isdtor at 01/18/2018 - 12:37

The message is: stay away from microcode updates because they're broken right now. Intel may or may not release fixes next week to be tested by OEMs.

Once working updates are available, OEMs will integrate them into their firmware/BIOS releases. That is one method to avail of microcode updates. The other method is loading during OS boot (via udev rule), with codes provided by the microcode_ctl rpm. It looks like Red Hat are now staying away from that; in any case, their previous rpm only included ucodes for three cpus. I did not check if the microcode.dat included more updates than that.

Method number 2b is to download the firmware from Intel directly and provide it in the locations defined by the microcode_ctl rpm. Then it's up to you to do the QA.

If your RHEL/CentOS is fully up to date, you're protected against variant 1/Spectre and Meltdown. Red Hat have done a pretty good job to backport those patches from upstream. GKH's blog is worth a read.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Matthew Miller at 01/18/2018 - 12:31

On Thu, Jan 18, 2018 at 11:01:18AM -0500, Pete Geenhuizen wrote:
Check with your hardware vendor for BIOS/EFI firmware updates. Apply
those.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Pete Geenhuizen at 01/18/2018 - 13:44

On 01/18/18 11:31, Matthew Miller wrote:
Pete

Re: /lib/firmware/microcode.dat update on CentOS 6

By Pete Geenhuizen at 01/18/2018 - 12:45

On 01/18/18 11:31, Matthew Miller wrote:

Re: /lib/firmware/microcode.dat update on CentOS 6

By Matthew Miller at 01/18/2018 - 14:55

On Thu, Jan 18, 2018 at 11:45:42AM -0500, Pete Geenhuizen wrote:
It does not matter. The microcode_ctl package contains CPU firmware
that is loaded at by the kernel early in the boot process if it's newer
than the one provided by the system firmware/BIOS. It is never
permanently stored in NVRAM or anything — it's loaded at each boot.

You should get a BIOS/EFI firmware update from your hardware vendor
which includes updated microcode. Then, you'll get the IBRS-capable
microcode at boot, every boot. This makes microcode_ctl moot.

Read more about this here: <a href="https://access.redhat.com/solutions/3315431" title="https://access.redhat.com/solutions/3315431">https://access.redhat.com/solutions/3315431</a>

Re: /lib/firmware/microcode.dat update on CentOS 6

By Valeri Galtsev at 01/18/2018 - 17:34

On 01/18/18 12:55, Matthew Miller wrote:
Thanks, Johnny, Matthew, Peter, ... everybody for your insights!

Valeri

++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++

Re: /lib/firmware/microcode.dat update on CentOS 6

By Phil Perry at 01/18/2018 - 16:27

On 18/01/18 18:55, Matthew Miller wrote:
Hence, by my understanding, there should not be any permanent damage
should you get a 'bad' microcode update, either from Intel or Red Hat,
that prevents the system from booting. Presumably one should still
always be able to boot the machine from a rescue disk, mount the fs and
either delete the offending microcode or uninstall the microcode_ctl
package to allow the system to boot again. This should not result in a
'bricked' permanently unrecoverable system.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Phelps, Matt at 01/18/2018 - 10:29

Thank you. We all appreciate your efforts, and your guidance.

Re: /lib/firmware/microcode.dat update on CentOS 6

By Leon Fauster at 01/17/2018 - 19:56

BTW: Will the microcode loaded from microcode.dat or from the intel-ucode directory (for EL6)?

Re: /lib/firmware/microcode.dat update on CentOS 6

By Phil Perry at 01/17/2018 - 17:53

On 17/01/18 21:24, Valeri Galtsev wrote:
See:

<a href="https://access.redhat.com/errata/RHSA-2018:0093" title="https://access.redhat.com/errata/RHSA-2018:0093">https://access.redhat.com/errata/RHSA-2018:0093</a>

Red Hat have rolled back the recent microcode updates for Spectre as
they were causing instabilities in some systems.

Updated microcode was only made available for a relatively small number
of CPUs so it might be the case that the your microcode was never
actually updated, hence there is nothing to roll back in the latest
release, so no need to panic about rebooting. Checking /var/log/messages
should give you more clues when your microcode was actually last updated
and allow you to determine if it was before or after the recent Spectre
fiasco.