Get the most out of your Centmin Mod LEMP stack
Become a Member

Security AMD Zenbleed Zero Day Speculative Security Vulnerability On AMD Zen 2 Processors CVE-2023-20593

Discussion in 'System Administration' started by eva2000, Jul 25, 2023.

  1. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    A new zero day security vulnerability has been announced for AMD Zen 2 family processors called Zendbleed https://lock.cmpxchg8b.com/zenbleed.html
    From AMD 'Zenbleed' bug lets hackers steal data from Ryzen CPUs | PCWorld


     
  2. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Lucky for me my Linode KVM VPS are AMD EPYC Milan 3rd generation 7xx3 = AMD EPYC 7713
    Code (Text):
    lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                4
    On-line CPU(s) list:   0-3
    Thread(s) per core:    1
    Core(s) per socket:    4
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             AuthenticAMD
    CPU family:            25
    Model:                 1
    Model name:            AMD EPYC 7713 64-Core Processor
    Stepping:              1
    CPU MHz:               1999.989
    BogoMIPS:              4001.64
    Hypervisor vendor:     KVM
    Virtualization type:   full
    L1d cache:             64K
    L1i cache:             64K
    L2 cache:              512K
    L3 cache:              16384K
    NUMA node0 CPU(s):     0-3
    

    Upcloud VPS users will need to contact support I'd imagine as they use AMD EPYC Rome 2nd gen = AMD EPYC 7542 Rome
    Code (Text):
    lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                1
    On-line CPU(s) list:   0
    Thread(s) per core:    1
    Core(s) per socket:    1
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             AuthenticAMD
    CPU family:            23
    Model:                 49
    Model name:            AMD EPYC 7542 32-Core Processor
    Stepping:              0
    CPU MHz:               2894.560
    BogoMIPS:              5789.12
    Hypervisor vendor:     KVM
    Virtualization type:   full
    L1d cache:             64K
    L1i cache:             64K
    L2 cache:              512K
    L3 cache:              16384K
    NUMA node0 CPU(s):     0
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm art rep_good nopl extd_apicid eagerfpu pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd rsb_ctxsw ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 retpoline_amd arat umip spec_ctrl intel_stibp
    
     
  3. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    https://www.amd.com/en/resources/product-security/bulletin/amd-sb-7008.html

    For data center CPUs
     
  4. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    AlmaLinux has test patches for Zenbleed AlmaLinux OS - Forever-Free Enterprise-Grade Operating System

     
  5. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Cloudflare Blog regarding AMD Zenbleed https://blog.cloudflare.com/zenbleed-vulnerability/

     
  6. buik

    buik “The best traveler is one without a camera.”

    1,982
    517
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,644
    Local Time:
    10:07 PM
    Don't like this at all.
    There is a very obvious reason that Red Hat, tests updates thoroughly. Linux firmware is the package required for partial or full functionality of certain hardware devices. You definitely don't want 0 day code in there. Unless there really is no other way, but there is no reason for that at the moment.

    CVE-2023-20593 is classed as medium. and since it is not (mis)used, totally no redeb to panic.

    I was already worried that AlmaLinux would go this route, after the announcement towards non 1:1.
     
  7. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Usually I'd agree. Though Cloudflare themselves already took the very same patches and applied it to their entire AMD EPYC Zen2 fleet of servers. They wouldn't do that if their folks with knowledge deemed it OK to do so - same as AlmaLinux folks. Though in AlmaLinux's case they're only releasing it for testing. Of course side effects depending on hardware used aren't truly known yet
     
  8. buik

    buik “The best traveler is one without a camera.”

    1,982
    517
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,644
    Local Time:
    10:07 PM

    But here is a big difference. Cloudflare obviously knows its own "custom" server fleet like the best.
    Complete data cages or even datacenters with the same servers using the exact same components.
    Of course, they have a few different server models. It could hardly be otherwise. But that seems to me to be limited to the minimum. Then it's quite easy to make a patch, stress it on test servers first, beforce implementing it production wise.
    And Cloudflare as a company providing Internet security services, you have to.

    A generic distro like AlmaLinux is a big different, in my opinion.
    Which should run on any computer built with all the components and thus combinations from the Hardware - Red Hat Ecosystem Catalog.

    Should it be a critical CVE, as a temporary, patch, ok as long as it is later replaced by RHEL upstream code.
    And that doesn't happen. The example I cited the other day is Grafana.

    AlmaLinux used Stream code to fix Grafana's critical CVE.
    The code is not replaced with RHEL upstream code and it is still using Stream code.
    Despite the fact that RHEL has long since released an update.

    As can also be seen in AlmaLinux's Grafana software with Stream code and the version of Rocky Linux, which is a 1:1 on RHEL.
     
  9. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Isn't that expected given AlmaLinux is no longer 1:1 clone of RHEL? AlmaLinux used Grafana's direct upstream source patch https://git.almalinux.org/rpms/grafana/commit/0ee7429587298d987cde5406df203320de227083
    That seems normal to me. My PHP-FPM EOL 5.6, 7.0-7.4 backported security patches are also from official php.net backported patches too :)

    Seems the actual backport patch is from Grafana upstream source while the unit test patched (0011-fix-alert-test.patch) is from CentOS Stream?
    Code (Text):
    # https://gitlab.com/redhat/centos-stream/rpms/grafana/-/blob/3731c12a8956d50514a58d9aa2f2d330d4ee32b6/0007-fix-alert-test.patch
    Patch11: 0011-fix-alert-test.patch
    # Patch was taken from grafana github sources and backported for 9.0.9
    # git diff v9.2.19 v9.2.20 -- pkg/
    Patch12: 0012-CVE-2023-3128.patch
    
     
  10. buik

    buik “The best traveler is one without a camera.”

    1,982
    517
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,644
    Local Time:
    10:07 PM
    Not logically or by definition.
    None 1:1 copy is in my opinion whatever Oracle does. Adding own software, because customers are likely to ask for it.

    Supporting multiple PHP versions (also with backport patches if needed in case of EOL), just like what you are doing.
    Offering MySQL software, supporting multiple additional Linux Kernels etc.

    So now AlmaLinux is fixing CVEs with code other than upstream. Totally unnecessary.

    Thus, the distro unnecessarily drifts away from upstream. Time and energy better used elsewhere because you can easily use upstream patches. For example, use the time for an Alma-homemade kernel. CloudLinux like multiple PHP versions etc.
     
  11. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  12. buik

    buik “The best traveler is one without a camera.”

    1,982
    517
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,644
    Local Time:
    10:07 PM
    Exactly and so that's the problem. Apparently, they feel they have to sit in Red Hat's chair.
    Same patch, different source, given the minor changes.
    With a release way too fast, so proper testing is almost impossible.

    This in combination with mixed Stream code, as you point out yourself. Gives a strange codebase mix.
    Same as this AMD bug, lots of effort to do, then Red Hat arrives with a similar patch a few days later.

    Why? I see absolutely no use case for a Stream mixed distro with fixes very similar to upstream.
     
  13. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Personally, I don't see the problem. You do what needs to be done to solve a problem :)

    AlmaLinux are releasing the AMD Zenbleed patch on July 27th it seems https://almalinux.discourse.group/t/zenbleed-patch-release-7am-eastern-us-time-7-27-23/2802

    They're also inviting for user feedback on how they should be handling these in future :)
     
  14. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    In contrast Rocky Linux is contining 1:1 clone RHEL compatibility so they decide to wait on RHEL's response https://forums.rockylinux.org/t/zen...s-rocky-linux-working-on-it/10812/2?u=eva2000. But RHEL are confusingly saying 'not affected' ? Which is true for Kernel package 2217845 – (CVE-2023-20593, zenbleed) CVE-2023-20593 hw: amd: Cross-Process Information Leak

    Edit: looks like they updated cve-details to list as Affected.

    So will be while to wait on RHEL and Rocky Linux to address AMD Zenbleed. Yeah takes time to test patches so sort of expected.

    From
    2217845 – (CVE-2023-20593, zenbleed) CVE-2023-20593 hw: amd: Cross-Process Information Leak

     
  15. buik

    buik “The best traveler is one without a camera.”

    1,982
    517
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,644
    Local Time:
    10:07 PM
    Because "ABI compatible with RHEL" is far too vaguely defined in terms of compatibility.
    That they are no longer go 1 on 1, oke fine no problem. But then at the very least take 100% compatibility just like Oracle.
    Or specify exactly what... as ABI is an incredibly broad term. And without extensive specification, it is nothing at all.
    And since only a few lines of code differ now, you don't notice much of it. We'll see in a while. After they really start changing a lot of code.

    As I suspected earlier, AlmaLinux want's in that Red Hat's chair,
    and via GitLab and others is actively fixing, or trying to fix Red Hat's software and sending in patches.

    AlmaLinux argues that: “We can fix this now. We don’t have to wait.”
    That's exactly the same as: I know I'm going to die someday, I better die now. I don't have to wait.

    You cannot solve a problem that does not exist (software was rated as not affected on that time). So without a specific use case, there is absolutely no reason to go correcting or (trying) directing Red Hat's software.

    There is a reason Red Hat is waiting to update software and/or release a potential update.
    It is complex matter and needs to work perfectly with its large compatibility hardware index before they come out with an update. And since the impact is changed from not affected to only moderate, I don't understand the panic of some administrators.

    But that, of course, is my personal opinion. Just like everything I write here.

     
    Last edited: Jul 28, 2023
  16. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah that would be expected if AlmaLinux is not pursuing 1:1 RHEL clone state. When I refer to doing what you need to do to solve a problem, I was thinking back when I inspected some of RHEL's .spec file source RPM files and patches, they sometimes take upstream open source updates/backports and alter the code so it works with RHEL's package/version dependencies = RHEL does whatever it needs to do to make it work with what they have.

    In AlmaLinux's defence, Redhat later changed AMD Zendbled to affected for all but RHEL 6 - so RHEL 7, 8, 9 are all affected cve-details. Just Redhat was either slow off the mark or incorrectly diagnosed AMD Zenbleed from the initial bug report discussion at least 2217845 – (CVE-2023-20593, zenbleed) CVE-2023-20593 hw: amd: Cross-Process Information Leak

    But yes the microcode/linux firmware fix is for small set of AMD Zen2 CPUs, rest would need vendor BIOS updates from motherboard manufacturers which for mobile/desktop AMD Zen2 aren't scheduled until Oct-Dec 2023! For some folks impacted, would be quicker to switch/upgrade the CPU to AMD Zen3 or AMD Zen4 based :D
     
  17. buik

    buik “The best traveler is one without a camera.”

    1,982
    517
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,644
    Local Time:
    10:07 PM
    Let me be clear:)
    I am not defending anyone.
    I disagree with the new path of both AlamaLinux, and Red Hat.
    It's just my opinion.
    I don't think Red Hat was slow or incorrect.
    At that time, the CVE was not final yet (still-unrated CVE at that time).
    You have to be ansolutely sure. You can't start intergrating all kinds of security related code without an established CVE.

    And since Red Hat uses the principle of backport.
    They have to do more work because custom code, can be of great influence.
    This obviously has to be done carefully and that takes time.
     
  18. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Definitely understand that :)
    Yeah a fine balance between adequate testing and urgency for zero day security vulnerabilities :)
     
  19. buik

    buik “The best traveler is one without a camera.”

    1,982
    517
    113
    Apr 29, 2016
    Flanders
    Ratings:
    +1,644
    Local Time:
    10:07 PM
    With a moderate CVE, there is no urgency and no reason to speed-up, the process at all.
    If critical in case of vulnerabilities however, known software houses almost always get the code at least a week before publishing the CVE formally.

    Then company's like Red Hat releases a patch update on day 1.
    But of course well tested because it has received the code times before.
    I keep repeating it. I don't understand this fuss about a moderate CVE at all.
    Perhaps because I am clearly trained in stressful situations.

    Where step 1 is, to keep your head cool.
    The latter is not to be tough, i am not mr big star. I'm off the street and Straight edge. I just mean that more colleagues should take this training. Just be cool. Nothing to worry.
     
  20. eva2000

    eva2000 Administrator Staff Member

    52,729
    12,075
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,602
    Local Time:
    6:07 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Just did a yum update on my AlmaLinux 8 and AlmaLinux 9 Centmin Mod servers and see the linux-firmware and microcode_ctl updates for Intel Downfall and AMD Zenbleed security vulnerabilities now now out of test repo into main repo AlmaLinux OS - Forever-Free Enterprise-Grade Operating System

    On AlmaLinux 8 got these updates
    Code (Text):
                                   
      linux-firmware-20230404-114.git2e92a49f.el8_8.alma.1.noarch                               
      microcode_ctl-4:20220809-2.20230808.1.el8_8.alma.x86_64
    

    On AlmaLinux 9 got these updates
    Code (Text):
      linux-firmware-whence-20230310-134.el9_2.alma.1.noarch                                   
      microcode_ctl-4:20220809-2.20230808.1.el9_2.alma.noarch
    

    From AlmaLinux instructions after update, to update the CPU microcode run the following or reboot the computer:
    Code (Text):
    echo 1 > /sys/devices/system/cpu/microcode/reload
    

    using microcode reload echo command on AlmaLinux 8
    Code (Text):
    dmesg | egrep -iv 'firewall' | grep -C2 microcode
    [12204821.506073] i2c i2c-2: sendbytes: NAK bailout.
    [12214335.393522] i2c i2c-2: sendbytes: NAK bailout.
    [12217429.885519] microcode: updated to revision 0xf4, date = 2023-02-23
    [12217429.885566] x86/CPU: CPU features have changed after loading microcode, but might not take effect.
    [12217429.885567] x86/CPU: Please consider either early loading through initrd/built-in or a potential BIOS update.
    [12217429.885568] microcode: Reload completed, microcode revision: 0xf4
    

    AlmaLinux 9 system didn't show anything in dmesg but does see auto applied microcode updates via command
    Code (Text):
    grep -i microcode /var/log/messages