Security August 2018: Kernel Security Update: L1TF - L1 Terminal Fault Attack CVE-2018-3620 & CVE-2018-3646

eva2000 · Aug 15, 2018

Redhat and CentOS Linux Kernel and microcode security updates are coming out for L1TF - L1 Terminal Fault Attack CVE-2018-3620 & CVE-2018-3646. Second post below has performance impact for the Kernel/microcode updates and other outlined mitigation suggestions.

All Linux users including Centmin Mod users will need to update their Linux kernels for L1TF attack vulnerability fix. This is only part of the fix especially for VPS users as bios/microcode updates at your VPS provider host node level also are required which may require server reboots at both web host and your end.

Yum Updates For CVE-2018-3620 & CVE-2018-3646

Keep an eye on updates to L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646 - Red Hat Customer Portal for news of Redhat/CentOS 6/7 Linux Kernel updates which you would find listing Kernel updates if available via command
Code (Text):
yum clean all
yum list updates
To update kernel if listed and answer yes = y when prompted
Code (Text):
yum update
You will need to reboot server after Kernel updates are made. According to Redhat if you're on VPS, VPS host nodes will need to be restarted by web host too

after updating firmware and hypervisors, every system/virtual machine will need to power off and restart to recognize a new hardware type.
Click to expand...

Example of listed updates for CentOS 7 64bit Red Hat Customer Portal
Code (Text):
yum list updates -q
Updated                   Packages
kernel.x86_64             3.10.0-862.11.6.el7  updates
kernel-devel.x86_64       3.10.0-862.11.6.el7  updates
kernel-headers.x86_64     3.10.0-862.11.6.el7  updates
kernel-tools.x86_64       3.10.0-862.11.6.el7  updates
kernel-tools-libs.x86_64  3.10.0-862.11.6.el7  updates
perf.x86_64               3.10.0-862.11.6.el7  updates
python-perf.x86_64        3.10.0-862.11.6.el7  updates
Example of listed updates for CentOS 6 64bit Red Hat Customer Portal
Code (Text):
yum list updates -q
Updated                 Packages
kernel.x86_64           2.6.32-754.3.5.el6  updates
kernel-devel.x86_64     2.6.32-754.3.5.el6  updates
kernel-firmware.noarch  2.6.32-754.3.5.el6  updates
kernel-headers.x86_64   2.6.32-754.3.5.el6  updates
For Linode VPS users on custom Linode 4.1* Kernels, there's an updated Kernel = 4.17.15-x86_64-linode115 for this which also requires a server reboot.

Once you have updated your Linux Kernels as per 1st post instructions (only applies to non-OpenVZ systems like Xen/KVM/dedicated) and have rebooted server, you will have access to /sys system level verification of type of Spectre/Meltdown class Kernel fixes that are in place on your server - including L1TF via
/sys/devices/system/cpu/vulnerabilities/l1tf. Which you can see via cat command the contents of
/sys/devices/system/cpu/vulnerabilities/l1tf
Code (Text):
cat /sys/devices/system/cpu/vulnerabilities/l1tf
There are others for meltdown, spectre v1/v2 and speculative store bypass too. Note Speculative store bypass security fix is disabled by default as vulnerability requires local user access first and lets end users determine if their usage scenarios require the fix to be enabled as it has up to 8% performance impact when enabled.
Code (Text):
ls -lah /sys/devices/system/cpu/vulnerabilities/
total 0
drwxr-xr-x 2 root root    0 Aug 17 00:26 .
drwxr-xr-x 9 root root    0 Aug 17 00:26 ..
-r--r--r-- 1 root root 4.0K Aug 17 00:26 l1tf
-r--r--r-- 1 root root 4.0K Aug 17 00:26 meltdown
-r--r--r-- 1 root root 4.0K Aug 17 00:26 spec_store_bypass
-r--r--r-- 1 root root 4.0K Aug 17 00:26 spectre_v1
On updated Linux Kernel systems you can use this command to use cat command to spit out the contents of each of these files located in /sys/devices/system/cpu/vulnerabilities/
Code (Text):
find /sys/devices/system/cpu/vulnerabilities/ -mindepth 1 -type f | while read f; do echo -e "\n$f"; cat $f; done
Below are example outputs from this command for updated Linode custom kernel, CentOS 7 and 6 default distro provided kernels.

Linode
Code (Text):
uname -r
4.17.15-x86_64-linode115
Code (Text):
find /sys/devices/system/cpu/vulnerabilities/ -mindepth 1 -type f | while read f; do echo -e "\n$f"; cat $f; done

/sys/devices/system/cpu/vulnerabilities/spectre_v2
Mitigation: Full generic retpoline

/sys/devices/system/cpu/vulnerabilities/l1tf
Mitigation: PTE Inversion

/sys/devices/system/cpu/vulnerabilities/spec_store_bypass
Vulnerable

/sys/devices/system/cpu/vulnerabilities/spectre_v1
Mitigation: __user pointer sanitization

/sys/devices/system/cpu/vulnerabilities/meltdown
Mitigation: PTI
CentOS 7
Code (Text):
uname -r
3.10.0-862.11.6.el7.x86_64
Code (Text):
find /sys/devices/system/cpu/vulnerabilities/ -mindepth 1 -type f | while read f; do echo -e "\n$f"; cat $f; done

/sys/devices/system/cpu/vulnerabilities/l1tf
Mitigation: PTE Inversion

/sys/devices/system/cpu/vulnerabilities/spectre_v1
Mitigation: Load fences, __user pointer sanitization

/sys/devices/system/cpu/vulnerabilities/spectre_v2
Vulnerable: Retpoline without IBPB

/sys/devices/system/cpu/vulnerabilities/spec_store_bypass
Vulnerable

/sys/devices/system/cpu/vulnerabilities/meltdown
Mitigation: PTI
CentOS 6
Code (Text):
uname -r
2.6.32-754.3.5.el6.x86_64
Code (Text):
find /sys/devices/system/cpu/vulnerabilities/ -mindepth 1 -type f | while read f; do echo -e "\n$f"; cat $f; done

/sys/devices/system/cpu/vulnerabilities/meltdown
Mitigation: PTI

/sys/devices/system/cpu/vulnerabilities/spectre_v1
Mitigation: Load fences

/sys/devices/system/cpu/vulnerabilities/spectre_v2
Mitigation: Full retpoline

/sys/devices/system/cpu/vulnerabilities/spec_store_bypass
Vulnerable

/sys/devices/system/cpu/vulnerabilities/l1tf
Mitigation: PTE Inversion
For OpenVZ VPS systems, they use Host node level Kernel so your web host needs to update host node level Kernel as you have no control over updating Kernels.

L1TF detection script

L1TF detection script from L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646 - Red Hat Customer Portal. Actual script https://access.redhat.com/sites/default/files/cve-2018-3620--2018-08-14-1255.sh

On L1TF vulnerable system
Code (Text):
./cve-2018-3620--2018-08-14-1255.sh

CVE-2018-3620 Detection Script Ver. 1.2
This script is primarily designed to detect CVE-2018-3620 on supported
Red Hat Enterprise Linux systems and kernel packages.
Result may be inaccurate for other RPM based systems.

CPU vendor: Intel
Running kernel: 3.10.0-862.3.3.el7.x86_64
Virtualization: None
SMT status: On

This system is vulnerable for the following reasons:
* Kernel is not updated
* Mitigation is not active

Red Hat recommends that you:
* Update your kernel and reboot the system.
* Optionally ask your HW vendor for CPU microcode update if available.

Note about microcode update
Customers desiring performance improvement will need to ask their HW vendor for CPU microcode update
in order to use the mitigation implemented in the microcode.

Note about Hyper-Threading (SMT)
Customers desiring to completely mitigate this issue will need to consider disabling SMT.
For details how to disable SMT see:
https://access.redhat.com/solutions/352663

For more information about the vulnerability see:
https://access.redhat.com/security/vulnerabilities/l1tf

For more information about possible performance impact see:
https://access.redhat.com/security/vulnerabilities/l1tf-perf
On L1TF Kernel updated + rebooted system
Code (Text):
./cve-2018-3620--2018-08-14-1255.sh

CVE-2018-3620 Detection Script Ver. 1.2
This script is primarily designed to detect CVE-2018-3620 on supported
Red Hat Enterprise Linux systems and kernel packages.
Result may be inaccurate for other RPM based systems.

CPU vendor: Intel
Running kernel: 3.10.0-862.11.6.el7.x86_64
Virtualization: kvm
SMT status: Off
Mitigation: Mitigation: PTE Inversion

This system is not vulnerable, because it has correct mitigation applied.

Note about microcode update
Customers desiring performance improvement will need to ask their HW vendor for CPU microcode update
in order to use the mitigation implemented in the microcode.

Note about virtualization
CVE-2018-3646 is the CVE identifier assigned to the virtualization escape aspect of the
flaw. In virtualized environment, there are more steps to mitigate the issue, including:
* Host needs to have updated kernel and CPU microcode
* Host needs to have updated virtualization software
* Guest needs to have updated kernel
* Hypervisor needs to propagate new CPU features correctly
For more details about mitigations in virtualized environment see the vulnerability article.

For more information about the vulnerability see:
https://access.redhat.com/security/vulnerabilities/l1tf

For more information about possible performance impact see:
https://access.redhat.com/security/vulnerabilities/l1tf-perf
L1TF - L1 Terminal Fault Attack CVE-2018-3620 & CVE-2018-3646 Information

Red Hat has been made aware of a new computer microprocessor hardware implementation (microarchitecture) issue similar to Spectre and Meltdown which has been reported to affect x86 microprocessors manufactured by Intel. Unprivileged attackers can use this flaw to bypass memory security restrictions in order to gain access to data stored in memory that would otherwise be inaccessible. There are three pieces to this vulnerability. The first affects only Intel “SGX” secure enclaves and is mitigated through microcode updates independently of the operating system. The other two pieces require software-level mitigations performed by operating systems and hypervisors. Full mitigation from potential attack by untrusted guest virtual machines in an environment using virtualization will require specific action by a system administrator.

CVE-2018-3620 is the CVE identifier assigned to the operating system vulnerability for this issue. CVE-2018-3646 is the CVE identifier assigned to the virtualization aspect of the flaw. This issue is referred to as L1 Terminal Fault (L1TF) by the larger industry and as “Foreshadow” by the security researcher.

The L1 Terminal Fault vulnerability allows a malicious actor to bypass memory access security controls ordinarily imposed and managed by the operating system or hypervisor. An attacker can use this vulnerability to read any physical memory location that is cached in the L1 data cacheof the processor. Normally, operating system and hypervisor managed “page tables” provide information to the processor about which memory locations should be accessible to an application, the operating system kernel itself, and guest virtual machine instances. These page tables are formed from page table entries (PTEs) that include a “present” bit indicating validity. In exploiting L1TF, an attacker abuses the Intel processor logic that recognizes valid PTEs.
Click to expand...

More info

L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646 - Red Hat Customer Portal

INTEL-SA-00161

L1 Terminal Fault - The Latest Speculative Execution Side Channel Attack - Phoronix

While Meltdown breaks the user space to kernel space protection, L1TF allows to attack any physical memory address in the system and the attack works across all protection domains. It allows an attack of SGX and also works from inside virtual machines because the speculation bypasses the extended page table (EPT) protection mechanism.

Ouch, it's going to be a hot mess in the public cloud/virtualization space. Any exposure by bare-metal users appears minimal to unaffected. The initial Linux kernel mitigations:
Click to expand...

Indeed a hot mess.. if there's any reason to force me off Cloud VPS onto bare metal dedicated hosting, this would probably be it.

Summary:

Security researchers have identified a speculative execution side-channel method called L1 Terminal Fault (L1TF). This method impacts select microprocessor products supporting Intel® Software Guard Extensions (Intel® SGX). Further investigation by Intel has identified two related applications of L1TF with the potential to impact additional microprocessors, operating systems, system management mode, and virtualization software. If used for malicious purposes, this class of vulnerability has the potential to improperly infer data values from multiple types of computing devices.

Intel is committed to product and customer security and to coordinated disclosure. We worked closely with other technology companies, operating system, and hypervisor software vendors, developing an industry-wide approach to mitigate these issues promptly and constructively.
For facts about these new exploits, technical resources, and steps you can take to help protect systems and information please visit: Side Channel Methods – Analysis, News and Updates.

Description:

CVE-2018-3615 - L1 Terminal Fault: SGX

Systems with microprocessors utilizing speculative execution and Intel® software guard extensions (Intel® SGX) may allow unauthorized disclosure of information residing in the L1 data cache from an enclave to an attacker with local user access via a side-channel analysis.

7.9 High CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N

CVE-2018-3620 - L1 Terminal Fault: OS/SMM

Systems with microprocessors utilizing speculative execution and address translations may allow unauthorized disclosure of information residing in the L1 data cache to an attacker with local user access via a terminal page fault and a side-channel analysis.

7.1 High CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:H/I:N/A:N

CVE-2018-3646 - L1 Terminal Fault: VMM

Systems with microprocessors utilizing speculative execution and address translations may allow unauthorized disclosure of information residing in the L1 data cache to an attacker with local user access with guest OS privilege via a terminal page fault and a side-channel analysis.

7.1 High CVSS:3.0/AV:L/AC:L/PR:N/UI:N/S:C/C:H/I:N/A:N

Affected products:

The following Intel-based platforms are potentially impacted by these issues. Intel may modify this list at a later time.

Intel® Core™ i3 processor (45nm and 32nm)
Intel® Core™ i5 processor (45nm and 32nm)
Intel® Core™ i7 processor (45nm and 32nm)
Intel® Core™ M processor family (45nm and 32nm)
2nd generation Intel® Core™ processors
3rd generation Intel® Core™ processors
4th generation Intel® Core™ processors
5th generation Intel® Core™ processors
6th generation Intel® Core™ processors **
7th generation Intel® Core™ processors **
8th generation Intel® Core™ processors **
Intel® Core™ X-series Processor Family for Intel® X99 platforms
Intel® Core™ X-series Processor Family for Intel® X299 platforms
Intel® Xeon® processor 3400 series
Intel® Xeon® processor 3600 series
Intel® Xeon® processor 5500 series
Intel® Xeon® processor 5600 series
Intel® Xeon® processor 6500 series
Intel® Xeon® processor 7500 series
Intel® Xeon® Processor E3 Family
Intel® Xeon® Processor E3 v2 Family
Intel® Xeon® Processor E3 v3 Family
Intel® Xeon® Processor E3 v4 Family
Intel® Xeon® Processor E3 v5 Family **
Intel® Xeon® Processor E3 v6 Family **
Intel® Xeon® Processor E5 Family
Intel® Xeon® Processor E5 v2 Family
Intel® Xeon® Processor E5 v3 Family
Intel® Xeon® Processor E5 v4 Family
Intel® Xeon® Processor E7 Family
Intel® Xeon® Processor E7 v2 Family
Intel® Xeon® Processor E7 v3 Family
Intel® Xeon® Processor E7 v4 Family
Intel® Xeon® Processor Scalable Family
Intel® Xeon® Processor D (1500, 2100)
** indicates Intel microprocessors affected by CVE-2018-3615 - L1 Terminal Fault: SGX

Please check with your system manufacturer for more information regarding updates for your system.

Recommendations:

Intel has worked with operating system vendors, equipment manufacturers, and other ecosystem partners to develop platform firmware and software updates that can help protect systems from these methods.

This includes the release of updated Intel microprocessor microcode to our customers and partners. This microcode was previously released as part of INTEL-SA-00115.

Status of available microcode can be found here.

End users and systems administrators should check with their system manufacturers and system software vendors and apply any available updates as soon as practical.
Click to expand...

Intel CPUs are affected but AMD CPUs, Intel Xeon Phi, and older Intel CPUs are among those not believed to be affected. Intel did post this video to YouTube when trying to dig up more on L1 Terminal Fault:
Click to expand...

eva2000 · Aug 16, 2018

From Impact tab L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646 - Red Hat Customer Portal

Attack Description and Impact
Traditional Host Attack Vector:
When a processor supports speculative execution of instructions, a speculative load from a virtual address which cannot be resolved to a physical address leads to a page-fault exception during its translation process. Before this page-fault exception is delivered, speculative execution of the load instruction uses the physical address from the not present (P flag = 0) OR reserved bits set paging structure entry to access the physical memory. If the physical memory location was cached in the L1 data cache, the speculative load reads data from an inappropriate physical memory location. As a result, subsequent speculative operations use this data and can have measurable impact upon the caches that an attacker can use to read unauthorized data.

An unprivileged system user or process could exploit the L1TF vulnerability to read data from arbitrary physical memory locations of the kernel and/or other processes running on the system provided that the data is first loaded into the L1 data cache. Since loads into the L1 data cache are an intrinsic component of the processor, a malicious attacker only has to wait for a program containing secrets of interest to load those secrets while performing normal operations. This can include cryptographic keys, and other data.

Virtualization Attack Vector:
Virtualized guest environments compound the impact from exploiting this vulnerability. In virtualized environments, guests often run an operating system which manages memory as if it were running on a bare metal host machine. What the guest operating system perceives as physical memory is in reality a virtual address space on the host, created by the Virtual Machine Monitor (VMM, e.g. KVM), also known as a hypervisor. The hypervisor introduces a layer of memory virtualization to ensure that it has control over the true host physical memory and so that it can provide isolated virtual address spaces to numerous guests while managing their access to the host physical memory. This means that virtualized environments have two levels of memory translations.

To facilitate efficient translation of the Guest physical addresses to Host physical addresses, hardware processors have introduced the Extended Page Table (EPT) feature.

EPT is similar to the hierarchical Paging structures used by the host operating system to translate virtual addresses to physical memory addresses. While translating Guest physical addresses to Host physical addresses, just like page-fault exceptions on the host, EPT Violationexceptions cause VM exits, indicating that the given guest physical address translation can not resolve to a host physical address OR the guest does not have permissions to access the given host memory.

Similar to the host, speculative execution of instructions by a guest user can leverage those EPT violations-induced VM exits to read data from host physical memory via cache side-channel attacks. Since the guest operating system can control guest physical address bits in the EPT paging entry, a guest user can target specific host physical addresses and read data via the L1 data cache, including arbitrary host kernel memory and/or memory of the other guests/processes running on the host.
Click to expand...

Kernel fix has a performance impact for VPS it seems and may even need disabling hyperthreading !!!!

Performance impact
Mitigating L1TF in bare metal environments that do not use virtualization has negligible performance impact and does not require specific action beyond installing the updates.

Mitigating L1TF in virtualized environments requires that the hypervisor check whether to flush the L1 data cache (L1D) of potential secrets from other virtual machines (or the host) whenever it begins running virtual machine code. Depending upon the workload, it may also be necessary to disable Intel Hyper-threading, in order to prevent a malicious virtual machine running on one hyperthread from attacking another on its peer thread.

The L1 data cache makes data available to the CPU much faster than the main memory and (almost) all data loaded from memory must pass through it. Once the L1 data cache is flushed, the CPU needs to spend hundreds of cycles to repopulate those entries from lower level caches. This is expensive in terms of time spent on data access. In terms of relative efficiency, L1 data cache takes 1x CPU cycles, while the same data access from lower level caches could take 4x CPU cycles. Data access from main memory could take 60-70x or more cycles. Since data contained within the L1 is also contained within the higher level caches, it will typically be refilled from the L2 at only modest impact to performance. This overhead also increases external interrupt and other event latency.

Similarly, Hyper-Threading (an SMT implementation) simultaneously executes more than one instruction stream to gain system performance. Disabling these threads would drop the number of logical CPUs seen by the system by half, thus significantly reducing the overall compute throughput of a system and its performance.

Note the actual performance impact of these measures is subject to the workload and can vary from one workload to another. More details about the workload specific performance impact can be seen here: Performance considerations for L1 Terminal Fault
Click to expand...

From Performance considerations for L1 Terminal Fault - Red Hat Customer Portal

The rest of this document describes L1TF and the performance impact for mitigating its three components.
Details for L1TF and the controls to mitigate it can be found in the following article:
L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646

It is advised to refer to the "Mitigation/Remediation" subsection of the above document as a reference for the various mitigation measures documented here. Testing was conducted with a view to each mitigation measure, and how they impact performance when enabled. The three mitigation techniques and their performance impact is discussed below.
Click to expand...

First: PTE Inversion
PTE Inversion fixes the L1TF vulnerability for processes running on the host. It is permanently enabled and does not impact performance.
Click to expand...

Second: L1D cache flushing on vmentry
To ensure that a guest cannot attack data which is present in the L1D cache, the hypervisor flushes the L1D before entering the guest. The kernel provides three controllable L1D flush modes, including “never”, “cond” for conditional, and “always”.
It’s difficult to predict which workloads L1D flushing will impact, because so many variables are involved as to which important cache line data resides in the L1D cache when it gets flushed.

Setting “never” indicates to the hypervisor that it should never flush the L1D on a vmentry. It has no performance impact.

Setting “cond”, the default value, causes the L1D to be flushed for a limited number of vmentry code paths.

Most workloads we tested saw no measurable impact.

A small number of workloads exhibited performance losses of upwards of 3-4%. Though seeing such a loss on one test configuration didn’t always mean that loss would be seen on another test configuration running the same workload.

Setting “always” causes an L1D flush for every vmentry.

The majority of the workloads showed little to no impact.

Several large multithreaded applications, like SPECjbb, showed an up to 4% slowdown.

Asynchronous file I/O benchmarks, which trigger high rates of vmexits, did have up to a 14% impact in one hardware configuration, and only 0-5% in another configuration.

This was also the setup where we saw the biggest performance impact from not having updated microcode. The above 14% impact (with microcode) slowed down to a 27% performance impact (without microcode). In most cases the hit from not having updated microcode was much smaller, in the 0-2% range. Thus, updated microcode can help with performance, especially if the L1D flushing is set to “always”.

Custom microbenchmarks (that tend to not represent real workloads), with tight loops that triggered high rates of vmexits did show slowdowns well into the double digit ranges.

Click to expand...

Third: Disabling Hyper-Threading
The performance impact from disabling Hyper-Threading (HT) varies widely. Disabling HT reduces the number of available vCPUs by 50%, which reduces the scale of virtualized environments. The Red Hat products that use KVM (Red Hat Virtualization and Red Hat OpenStack) are affected, as are OpenShift-on-OpenStack environments.

Red Hat’s performance testing on Intel hardware is consistent with Intel’s published data on Hyper-Threading. The performance impact when HT is disabled is dependent on many factors. Measured impact ranges from a +30% gain, to -50% loss and beyond. Most HT testing, however, showed losses in the 0-30% range.

Disabling HT in environments with less than ~50% cpu utilization usually showed no performance impact. Disabling HT in environments with higher cpu utilization yields a serious performance drop. The amount of loss was usually dependent on the cpu utilization before HT was disabled.

When HT is disabled in VM guests whose cpu and memory usage fit within one numa node, the performance impact we measured with database, network, and SPEC benchmarks, frequently fell into the -10-30% range. However on larger VMs (where cpu+memory usage spanned multiple numa nodes) the impact tended to be less, usually in the -10% range.

We’ve measured certain applications that ran faster when HT was disabled. Some examples include:

A highly optimized and tightly scheduled linpack benchmark within a VM that spans multiple numa nodes. In that scenario, the performance of this benchmark improved up to a 30% gain with HT disabled.

OLTP benchmark (based on TPC-C) whose guest size fit on one numa node ran faster with HT enabled. However larger OLTP guests that need the memory and cpu resources from multiple numa nodes ran faster with HT disabled.

The impact for clouds hosting VMs may be significant. Disabling HT has the perceived effect of 50% fewer vCPU for scheduling purposes. As a result, OpenStack and Red Hat Virtualization will have a perceived performance impact of up to 50% less capacity. Tools that monitor utilization will indeed view a NON-HT cloud to be up to 50% smaller than with HT enabled. Cloud providers, (e.g. Amazon Web Services, Google Cloud and Microsoft Azure) may have mitigations within their public clouds such that Red Hat Enterprise Linux VM’s in their clouds are not vulnerable to this CVE. It is recommended to verify susceptibility and mitigations measures with public cloud vendors in which Red Hat software is resident.

Click to expand...

Dam disabling Intel Hyperthreading would be huge impact on performance a Xeon E3-12xx with 4 core + 4 HT threads = 8 cpu threads would be reduced to just 4 cores/4 threads !

eva2000 · Aug 16, 2018

Foreshadow Flaw Targets Intel SGX and Virtual Machines

As the industry has patched for Spectre and Meltdown, we now have Foreshadow which targets Intel secure enclaves (SGX), virtual machines, and hyper-threading through L1 Terminal Fault attack (L1TF.) These speculative execution exploits are difficult to understand so we are going to try distilling everything down to the key points.
Click to expand...

Largely left out of the Spectre and Meltdown discussion was Intel SGX. Intel SGX creates a secure enclave to store sensitive information that even an operating system is not supposed to see. With Foreshadow, researchers have been able to peek inside SGX and get to extremely sensitive data. If you want to see this in action, there is now a video of the attack available:
Click to expand...

Due to the fact that both threads on an Intel Hyper-Threading core are executing using the same cache resources, this mainstay of enhancing performance also means that there is a shared level 1 cache. On the desktop side, this is less of an issue since OS vendors can patch the OS and most desktop users are not running untrusted virtual machine code on their systems.

Where this gets slightly scarier is with Intel virtual machines. If a bad actor creates a corrupt extended page table in their virtual machine, they can trick the hypervisor. The hypervisor can then reveal information to other virtual machines. Hyper-Threading in the multi-tenant cloud means that VMs from two different users can be running on the same physical core as two threads. One VM can use L1TF to then try gaining information about the other. This is a big deal for cloud providers since the cloud is multi-tenant native. For enterprises that have virtualization clusters or private clouds running untrusted VMs, this can be equally concerning.
Click to expand...

Foreshadow Mitigation
This vulnerability was disclosed to Intel and other parties months ago. In the meantime, patches have been released. Those that we have spoken to say that the performance impact for bare metal servers is negligible.

In virtualized multi-tenant systems there is a major implication. Although there is a L1 cache flush patch available for hypervisors, it also means that a major mitigation step is to turn off Intel Hyper-Threading. We regularly see a 30% improvement on many of our workloads in the STH lab with Hyper-Threading so turning off the feature can have an enormous negative impact on performance.
Click to expand...

eva2000 · Aug 16, 2018

Tweet from CentOS Karanbir Singh on Twitter

'nosmt' is the kernel option needed to disable Hyper Threading. If you need to, for L1TF
Click to expand...

eva2000 · Aug 16, 2018

Understanding L1 Terminal Fault aka Foreshadow: What you need to know

Hyper-Threads run simultaneously (the “S” in “SMT”), and as a result, it is possible that one thread is running Hypervisor code or another virtual machine instance, while simultaneously the peer thread is executing a malicious guest. When entering the malicious guest on the peer thread, the L1 data cache will be flushed, but unfortunately, it is not possible to prevent it from subsequently observing the cache loads performed by its peer thread. Thus, if two different virtual machines are running on the same core at the same time, it is difficult to guarantee that they cannot perform an L1TF exploit against one another and steal secrets.

The precise impact of L1TF to Hyper-Threading depends upon the specific use case and the virtualization environment being used. In some cases, it may be possible for public cloud vendors (who have often built special purpose hardware to assist in isolation) to take steps to render Hyper-Threading safe. In other cases, such as in a traditional enterprise environment featuring untrusted guest virtual machines, it may be necessary to disable Intel Hyper-Threading. Since this varies from one use case to another, and from one environment to another, Red Hat and our peers are not disabling Intel Hyper-Threading by default. Customers should instead consult our Knowledge Base article and make the appropriate determination for their own situation.

To help facilitate control over Intel Hyper-Threading, Red Hat is shipping updated kernels that include a new interface through which customers can disable Hyper-Threading at boot time. Consult the updated kernel documentation and Knowledgebase for further information.

Wrapping up
The L1TF (L1 Terminal Fault) Intel processor vulnerability is complex and in some cases requires specific actions by customers to effect a complete mitigation. Red Hat and our partners have been working to prepare for the public coordinated disclosure, and to prepare patches, documentation, training, and other materials necessary to help keep our customers and their data safe. We recommend that you always follow best security practices, including deploying the updates for the L1TF vulnerability as quickly as possible.

For further information, please consult the Red Hat Knowledgebase article on L1TF.

Jon Masters is chief ARM architect at Red Hat.
Click to expand...

and A deeper look at L1 Terminal Fault aka Foreshadow

eva2000 · Aug 16, 2018

DigitalOcean twitter response DigitalOcean on Twitter

In light of the L1TF security vulnerability we are working with Intel to ensure the protection of our customers and their data. For more information: https://do.co/l1tf
Click to expand...

Blog A Message about Intel’s L1TF Security Vulnerability

Today, Intel released a statement regarding L1 Terminal Fault (L1TF), a severe security vulnerability that affects many multi-tenant environments running virtual machines, including DigitalOcean. This vulnerability exposes data to any guest running on the same processor core.

In DigitalOcean’s environment, this means an attacker could theoretically use one Droplet to view another Droplet’s memory. However, they should have no ability to target a specific Droplet or user.

The security implications of this vulnerability are significant and require us to move rapidly to ensure our platform remains protected. In the wake of previous vulnerabilities, Intel has improved their communications flow with us and shared more information sooner, which enabled us to start our mitigation efforts yesterday. However, due to the condensed timeline, unforeseen issues may arise during these efforts. We will continue to work with Intel to enhance their multi-party vulnerability disclosure process so we can improve our agility and efficiency in the future, and better address these types of issues.

Remediation efforts will be completed within a few weeks, and during this time we will take all possible steps to ensure customer Droplets and data remain safe. We do not anticipate any downtime for our users as a result of our mitigation efforts.

We are closely monitoring this situation, and we will update this blog post as more information becomes available. We will notify customers directly should there be any action required of them, or any action taken that may impact their DigitalOcean account.

You can read Intel’s initial statement here.
Click to expand...

eva2000 · Aug 16, 2018

AWS already has implemented its own protections at host level it seems. But at guest level you'd still need to do updates L1 Terminal Fault Speculative Execution Issue

L1 Terminal Fault Speculative Execution Issue
August 14, 2018 11:00 AM PDT

CVE Identifiers: CVE-2018-3620, CVE-2018-3646
Intel has published a security advisory (INTEL-SA-00161) regarding a new side-channel analysis method concerning their processors called "L1 Terminal Fault" (L1TF). AWS has designed and implemented its infrastructure with protections against these types of attacks, and has also deployed additional protections for L1TF. All EC2 host infrastructure has been updated with these new protections, and no customer action is required at the infrastructure level.

Updated kernels for Amazon Linux AMI 2017.09 (ALAS-2018-1058), Amazon Linux AMI 2018.03 (ALAS-2018-1058), and Amazon Linux 2 (ALAS-2018-1058) are available in the respective repositories. As a general security best practice, we recommend that customers patch their operating systems or software as relevant patches become available to address emerging side-channel issues.

New AMIs with the updated kernels are being prepared. This bulletin will be updated after these AMIs are available.
Meanwhile, we suggest using the stronger security and isolation properties of EC2 instances rather relying on operating system process boundaries or containers when workloads execute with different security privileges.
Click to expand...

eva2000 · Aug 16, 2018

Foreshadow: Breaking the Virtual Memory Abstraction with Transient Out-of-Order Execution

Foreshadow is a speculative execution attack on Intel processors which allows an attacker to steal sensitive information stored inside personal computers or third party clouds. Foreshadow has two versions, the original attack designed to extract data from SGX enclaves and a Next-Generation version which affects Virtual Machines (VMs), hypervisors (VMM), operating system (OS) kernel memory, and System Management Mode (SMM) memory.
Click to expand...

What about other processors (AMD/ARM)?

The original Foreshadow attack affects most SGX-enabled Intel processors. As SGX is currently present only in Intel CPUs, we are unaware of Foreshadow affecting other CPU vendors. To the best of our understanding, Foreshadow-NG only affects Intel processors. However, we are still working to better understand the implications of Foreshadow-NG and this answer might change as the situation develops.
Click to expand...

Are there mitigations against Foreshadow?

Foreshadow and Foreshadow-NG require mitigations at the both software and microcode level. This includes updates to most operating systems, hypervisors as well as CPU microcode updates. See Intel's security advisory and our technical report for additional details.
Click to expand...

Do Spectre and Meltdown mitigations such as Retpoline, KPTI, IBRS, STIBP, or IBPB mitigate Foreshadow?

No. The mitigations against Meltdown and Spectre are not effective against Foreshadow and Foreshadow-NG.
Click to expand...

How is Foreshadow different from Meltdown?

Foreshadow is different from Meltdown as it targets virtual machines and SGX in addition to data stored in the operating system's kernel (which was targeted by Meltdown). While harder to exploit, Foreshadow is effective against systems deploying Kernel Page Table Isolation (KPTI), which stops Meltdown attacks.
Click to expand...

How is Foreshadow different from prior Spectre-like attacks on SGX?

Concurrent speculative execution attacks on SGX, e.g., SGXPectre, rely on the code of the affected enclave to contain gadgets vulnerable to Spectre. Thus, such attacks can be theoretically mitigated by removing these gadgets from the affected enclave. In contrast, Foreshadow extracts enclave memory without relying on any code vulnerability in the victim enclave or even without requiring the affected enclave to execute. As such, Foreshadow can be used even in the case where the code of the affected enclave is "perfect", i.e., does not contain any side-channel vulnerability.
Click to expand...

What CPUs are affected by Foreshadow and Foreshadow-NG?

Intel confirmed that Foreshadow affects all SGX-enabled Core processors (Skylake and Kaby Lake), while Atom family processors with SGX support remain unaffected. Intel confirmed that Foreshadow-NG affects the following processes:

Intel Core™ i3/i5/i7/M processor (45nm and 32nm)

2nd/3rd/4th/5th/6th/7th/8th generation Intel Core processors

Intel Core X-series Processor Family for Intel X99 and X299 platforms

Intel Xeon processor 3400/3600/5500/5600/6500/7500 series

Intel Xeon Processor E3 v1/v2/v3/v4/v5/v6 Family

Intel® Xeon® Processor E5 v1/v2/v3/v4 Family

Intel® Xeon® Processor E7 v1/v2/v3/v4 Family

Intel® Xeon® Processor Scalable Family

Intel® Xeon® Processor D (1500, 2100)

See Intel's security advisory for additional details.
Click to expand...

eva2000 · Aug 16, 2018

AMD EPYC not vulnerable AMD Processor Security | AMD

Foreshadow
8/14/18 – Updated

As in the case with Meltdown, we believe our processors are not susceptible to these new speculative execution attack variants: L1 Terminal Fault – SGX (also known as Foreshadow) CVE 2018-3615, L1 Terminal Fault – OS/SMM (also known as Foreshadow-NG) CVE 2018-3620, and L1 Terminal Fault – VMM (also known as Foreshadow-NG) CVE 2018-3646, due to our hardware paging architecture protections. We are advising customers running AMD EPYC™ processors in their data centers, including in virtualized environments, to not implement Foreshadow-related software mitigations for their AMD platforms.
Click to expand...

eva2000 · Aug 16, 2018

Early benchmark numbers for performance impact for L1TF fixes An Early Look At The L1 Terminal Fault "L1TF" Performance Impact On Virtual Machines - Phoronix

With having started the kernel build and testing quickly upon finding out about the L1 Terminal Fault, my testing began with an Intel Core i7 8086K given it was in a rack under my desk and available for testing. With L1TF's exposure really being to virtual machines, testing Intel Xeon obviously makes more sense and those are among the systems now being tested under Linux with the latest mitigation techniques. But I continued on testing the i7-8086K anyhow from the position as what home/workstation users might see who are firing up their own virtual machines.
Click to expand...

eva2000 · Aug 16, 2018

Google Cloud's host infrastructure has deployed mitigations for L1TF but guest OS side still needs taking care of Protecting against the new “L1TF” speculative vulnerabilities | Google Cloud Blog

Directly exploiting these vulnerabilities requires control of hardware resources that are accessible only with operating system level control of the underlying physical or virtual processors. Unpatched operating systems may also permit indirect exploitation, dependent on their handling of operations that manipulate memory mappings.

We have deployed mitigations to Google’s infrastructure, including the infrastructure that underpins Google Cloud, which prevent the creation of vulnerable page-table entries within our host OS. In this blog, we’ll go into detail about these vulnerabilities, how we mitigate guest-controlled entries not controlled by the host OS, and what steps customers may need to take to ensure they are fully protected.
Click to expand...

Google Cloud mitigations against L1TF

Google Compute Engine employs host isolation features which ensure that an individual core is never concurrently shared between distinct virtual machines. This isolation also ensures that, in the case that different virtual machines are scheduled sequentially, the L1 data cache is completely flushed to ensure that no vulnerable state remains. In addition, we have also developed and deployed infrastructure that allows us to monitor our hosts for certain classes of these attacks.
Click to expand...

and Product Status: L1 Terminal Fault - Google Help

Google Compute Engine
The infrastructure that runs Compute Engine and isolates customer workloads from each other is protected against known attacks.

Compute Engine customers running multi-tenant workloads on a single VM may be vulnerable. In this environment, it may be possible for one tenant to observe the memory of other tenants or the guest operating system itself.

Compute Engine customers running multi-tenant workloads on the same VM should prioritize patching those environments, but as always, all Compute Engine customers are encouraged to follow security best practices when it comes to keeping runtime environments patched and protected against known security vulnerabilities.

See the Compute Engine security bulletin page for guidance on updating your Compute Engine VMs, the list of patched image versions, and links to additional information from operating system providers.
Click to expand...

Google Kubernetes Engine
The infrastructure that runs Kubernetes Engine and isolates customer Clusters and Nodes from each other is protected against known attacks.

Kubernetes Engine customers running containers from different customers on the same GKE Node are vulnerable. In this environment, it may be possible for one container to observe the memory of other containers on the same Node or the Node environment itself.
Click to expand...

eva2000 · Aug 16, 2018

According to Oracle and Intel a single standalone mitigation method isn't enough to protect you Intel Processor L1TF vulnerabilities: CVE-2018-3615, CVE-2018-3620, CVE-2018-3646

The technical steps Intel recommends to mitigate L1TF vulnerabilities on affected systems include:

Ensuring that affected Intel processors are running the latest Intel processor microcode. Intel reports that the microcode update it has released for the Spectre 3a (CVE-2018-3640) and Spectre 4 (CVE-2018-3639) vulnerabilitiesalso contains the microcode instructions which can be used to mitigate the L1TF vulnerabilities. Updated microcode by itself is not sufficient to protect against L1TF.

Applying the necessary OS and virtualization software patches against affected systems. To be effective, OS patches will require the presence of the updated Intel processor microcode. This is because updated microcode by itself is not sufficient to protect against L1TF. Corresponding OS and virtualization software updates are also required to mitigate the L1TF vulnerabilities present in Intel processors.

Disabling Intel Hyper-Threading technology in some situations. Disabling HT alone is not sufficient for mitigating L1TF vulnerabilities. Disabling HT will result in significant performance degradation.

Click to expand...

eva2000 · Aug 17, 2018

Linode forum has an L1TF security fix update with blog post to follow soon How is Linode handling L1TF/Foreshadow? | Linode Questions. However, there suggestion to update kernels isn't enough according to Intel here. Seems Linode has also fixed their custom kernel 4.1* issues as I tried and it worked, rebooted Linode VPS and showed the correct networking so that's good news

You can check kernel version via uname -r command i.e. on Linode after reboot
Code (Text):
uname -r
4.17.15-x86_64-linode115
Note newer Linode VPS might had defaulted to distro provided Linux Kernels due to previous custom kernel 4.1* issues.

Linode is currently investigating the impact of the recently disclosed L1TF vulnerability (also known as "Foreshadow").

What can I do right now?
If you use the Linode kernel, make sure you're set to "Latest", and reboot your Linode – we've just pushed 4.17.15, which contains patches for L1TF/Foreshadow.

If you're using a distro kernel, update with your distro's package manager, and reboot.

Where can I find more info?
We'll have a post on the Linode Blog later today (16 August) with general information, and we'll update this question with a link once that blog post has been created.

More specific information about this vulnerability is available at the following links:

Foreshadow: Breaking the Virtual Memory Abstraction with Transient Out-of-Order Execution
INTEL-SA-00161

We'll continue to update this post with additional resources on L1TF/Foreshadow. For general updates, stay tuned to our blog. If you have specific questions about L1TF/Foreshadow, please feel free to ask in the thread below.
Click to expand...

Blog post is up Intel’s L1TF CPU Vulnerabilities & Linode

Earlier this week Intel publicly disclosed a new class of processor vulnerabilities known as L1 Terminal Fault (L1TF). Variants of L1TF affect many single and multi-tenant environments, including some of Linode’s infrastructure and Linodes themselves.

We have begun mitigation efforts and anticipate full mitigation of our fleet within the next few weeks. We believe we can achieve this without any interruption to your running systems, and without requiring any coordination on your part. However, this is still evolving and we’ll know more as we go. Early results of our mitigations have been encouraging.

While that protects our side of things, you should make sure you’re running a Linux kernel with the mitigations in place. Check out our guide on upgrading your kernel.

As we move forward with our mitigation efforts over the next few weeks, we will continue providing more information here on our blog. Stay tuned!
Click to expand...

eva2000 · Aug 17, 2018

Once you have updated your Linux Kernels as per 1st post instructions (only applies to non-OpenVZ systems like Xen/KVM/dedicated) and have rebooted server, you will have access to /sys system level verification of type of Spectre/Meltdown class Kernel fixes that are in place on your server - including L1TF via
/sys/devices/system/cpu/vulnerabilities/l1tf. Which you can see via cat command the contents of
/sys/devices/system/cpu/vulnerabilities/l1tf
Code (Text):
cat /sys/devices/system/cpu/vulnerabilities/l1tf
There are others for meltdown, spectre v1/v2 and speculative store bypass too. Note Speculative store bypass security fix is disabled by default as vulnerability requires local user access first and lets end users determine if their usage scenarios require the fix to be enabled as it has up to 8% performance impact when enabled.
Code (Text):
ls -lah /sys/devices/system/cpu/vulnerabilities/
total 0
drwxr-xr-x 2 root root    0 Aug 17 00:26 .
drwxr-xr-x 9 root root    0 Aug 17 00:26 ..
-r--r--r-- 1 root root 4.0K Aug 17 00:26 l1tf
-r--r--r-- 1 root root 4.0K Aug 17 00:26 meltdown
-r--r--r-- 1 root root 4.0K Aug 17 00:26 spec_store_bypass
-r--r--r-- 1 root root 4.0K Aug 17 00:26 spectre_v1
On updated Linux Kernel systems you can use this command to use cat command to spit out the contents of each of these files located in /sys/devices/system/cpu/vulnerabilities/
Code (Text):
find /sys/devices/system/cpu/vulnerabilities/ -mindepth 1 -type f | while read f; do echo -e "\n$f"; cat $f; done
Below are example outputs from this command for updated Linode custom kernel, CentOS 7 and 6 default distro provided kernels.

Linode
Code (Text):
uname -r
4.17.15-x86_64-linode115
Code (Text):
find /sys/devices/system/cpu/vulnerabilities/ -mindepth 1 -type f | while read f; do echo -e "\n$f"; cat $f; done

/sys/devices/system/cpu/vulnerabilities/spectre_v2
Mitigation: Full generic retpoline

/sys/devices/system/cpu/vulnerabilities/l1tf
Mitigation: PTE Inversion

/sys/devices/system/cpu/vulnerabilities/spec_store_bypass
Vulnerable

/sys/devices/system/cpu/vulnerabilities/spectre_v1
Mitigation: __user pointer sanitization

/sys/devices/system/cpu/vulnerabilities/meltdown
Mitigation: PTI
CentOS 7
Code (Text):
uname -r
3.10.0-862.11.6.el7.x86_64
Code (Text):
find /sys/devices/system/cpu/vulnerabilities/ -mindepth 1 -type f | while read f; do echo -e "\n$f"; cat $f; done

/sys/devices/system/cpu/vulnerabilities/l1tf
Mitigation: PTE Inversion

/sys/devices/system/cpu/vulnerabilities/spectre_v1
Mitigation: Load fences, __user pointer sanitization

/sys/devices/system/cpu/vulnerabilities/spectre_v2
Vulnerable: Retpoline without IBPB

/sys/devices/system/cpu/vulnerabilities/spec_store_bypass
Vulnerable

/sys/devices/system/cpu/vulnerabilities/meltdown
Mitigation: PTI
CentOS 6
Code (Text):
uname -r
2.6.32-754.3.5.el6.x86_64
Code (Text):
find /sys/devices/system/cpu/vulnerabilities/ -mindepth 1 -type f | while read f; do echo -e "\n$f"; cat $f; done

/sys/devices/system/cpu/vulnerabilities/meltdown
Mitigation: PTI

/sys/devices/system/cpu/vulnerabilities/spectre_v1
Mitigation: Load fences

/sys/devices/system/cpu/vulnerabilities/spectre_v2
Mitigation: Full retpoline

/sys/devices/system/cpu/vulnerabilities/spec_store_bypass
Vulnerable

/sys/devices/system/cpu/vulnerabilities/l1tf
Mitigation: PTE Inversion
For OpenVZ VPS systems, they use Host node level Kernel so your web host needs to update host node level Kernel as you have no control over updating Kernels.

eva2000 · Aug 17, 2018

L1TF detection script from L1TF - L1 Terminal Fault Attack - CVE-2018-3620 & CVE-2018-3646 - Red Hat Customer Portal. Actual script https://access.redhat.com/sites/default/files/cve-2018-3620--2018-08-14-1255.sh

On L1TF vulnerable system

Code (Text):

./cve-2018-3620--2018-08-14-1255.sh

CVE-2018-3620 Detection Script Ver. 1.2
This script is primarily designed to detect CVE-2018-3620 on supported
Red Hat Enterprise Linux systems and kernel packages.
Result may be inaccurate for other RPM based systems.

CPU vendor: Intel
Running kernel: 3.10.0-862.3.3.el7.x86_64
Virtualization: None
SMT status: On

This system is vulnerable for the following reasons:
* Kernel is not updated
* Mitigation is not active

Red Hat recommends that you:
* Update your kernel and reboot the system.
* Optionally ask your HW vendor for CPU microcode update if available.

Note about microcode update
Customers desiring performance improvement will need to ask their HW vendor for CPU microcode update
in order to use the mitigation implemented in the microcode.

Note about Hyper-Threading (SMT)
Customers desiring to completely mitigate this issue will need to consider disabling SMT.
For details how to disable SMT see:
https://access.redhat.com/solutions/352663

For more information about the vulnerability see:
https://access.redhat.com/security/vulnerabilities/l1tf

For more information about possible performance impact see:
https://access.redhat.com/security/vulnerabilities/l1tf-perf

On L1TF Kernel updated + rebooted system

Code (Text):

./cve-2018-3620--2018-08-14-1255.sh

CVE-2018-3620 Detection Script Ver. 1.2
This script is primarily designed to detect CVE-2018-3620 on supported
Red Hat Enterprise Linux systems and kernel packages.
Result may be inaccurate for other RPM based systems.

CPU vendor: Intel
Running kernel: 3.10.0-862.11.6.el7.x86_64
Virtualization: kvm
SMT status: Off
Mitigation: Mitigation: PTE Inversion

This system is not vulnerable, because it has correct mitigation applied.

Note about microcode update
Customers desiring performance improvement will need to ask their HW vendor for CPU microcode update
in order to use the mitigation implemented in the microcode.

Note about virtualization
CVE-2018-3646 is the CVE identifier assigned to the virtualization escape aspect of the
flaw. In virtualized environment, there are more steps to mitigate the issue, including:
* Host needs to have updated kernel and CPU microcode
* Host needs to have updated virtualization software
* Guest needs to have updated kernel
* Hypervisor needs to propagate new CPU features correctly
For more details about mitigations in virtualized environment see the vulnerability article.

For more information about the vulnerability see:
https://access.redhat.com/security/vulnerabilities/l1tf

For more information about possible performance impact see:
https://access.redhat.com/security/vulnerabilities/l1tf-perf

eva2000 · Aug 18, 2018

More info at L1TF - L1 Terminal Fault — The Linux Kernel documentation

L1 Terminal Fault is a hardware vulnerability which allows unprivileged speculative access to data which is available in the Level 1 Data Cache when the page table entry controlling the virtual address, which is used for the access, has the Present bit cleared or other reserved bits set.

Affected processors
This vulnerability affects a wide range of Intel processors. The vulnerability is not present on:

Processors from AMD, Centaur and other non Intel vendors

Older processor models, where the CPU family is < 6

A range of Intel ATOM processors (Cedarview, Cloverview, Lincroft, Penwell, Pineview, Silvermont, Airmont, Merrifield)

The Intel XEON PHI family

Intel processors which have the ARCH_CAP_RDCL_NO bit set in the IA32_ARCH_CAPABILITIES MSR. If the bit is set the CPU is not affected by the Meltdown vulnerability either. These CPUs should become available by end of 2018.

Whether a processor is affected or not can be read out from the L1TF vulnerability file in sysfs. See L1TF system information.

Related CVEs
The following CVE entries are related to the L1TF vulnerability:

CVE-2018-3615 L1 Terminal Fault SGX related aspects
CVE-2018-3620 L1 Terminal Fault OS, SMM related aspects
CVE-2018-3646 L1 Terminal Fault Virtualization related aspects
Click to expand...

and L1TF - L1 Terminal Fault — The Linux Kernel documentation

Mitigation control on the kernel command line
The kernel command line allows to control the L1TF mitigations at boot time with the option “l1tf=”. The valid arguments for this option are:

full - Provides all available mitigations for the L1TF vulnerability. Disables SMT and enables all mitigations in the hypervisors, i.e. unconditional L1D flushing. SMT control and L1D flush control via the sysfs interface is still possible after boot. Hypervisors will issue a warning when the first VM is started in a potentially insecure configuration, i.e. SMT enabled or L1D flush disabled.

full,force - Same as ‘full’, but disables SMT and L1D flush runtime control. Implies the ‘nosmt=force’ command line option. (i.e. sysfs control of SMT is disabled.)

flush - Leaves SMT enabled and enables the default hypervisor mitigation, i.e. conditional L1D flushing. SMT control and L1D flush control via the sysfs interface is still possible after boot. Hypervisors will issue a warning when the first VM is started in a potentially insecure configuration, i.e. SMT enabled or L1D flush disabled.

flush,nosmt - Disables SMT and enables the default hypervisor mitigation, i.e. conditional L1D flushing. SMT control and L1D flush control via the sysfs interface is still possible after boot. Hypervisors will issue a warning when the first VM is started in a potentially insecure configuration, i.e. SMT enabled or L1D flush disabled.

flush,nowarn - Same as ‘flush’, but hypervisors will not warn when a VM is started in a potentially insecure configuration.

off - Disables hypervisor mitigations and doesn’t emit any warnings.

The default is ‘flush’. For details about L1D flushing see 1. L1D flush on VMENTER.
Click to expand...

Mitigation selection guide
1. No virtualization in use
The system is protected by the kernel unconditionally and no further action is required.

2. Virtualization with trusted guests
If the guest comes from a trusted source and the guest OS kernel is guaranteed to have the L1TF mitigations in place the system is fully protected against L1TF and no further action is required.

To avoid the overhead of the default L1D flushing on VMENTER the administrator can disable the flushing via the kernel command line and sysfs control files. See Mitigation control on the kernel command line and Mitigation control for KVM - module parameter.

3. Virtualization with untrusted guests
3.1. SMT not supported or disabled
If SMT is not supported by the processor or disabled in the BIOS or by the kernel, it’s only required to enforce L1D flushing on VMENTER.

Conditional L1D flushing is the default behaviour and can be tuned. See Mitigation control on the kernel command line and Mitigation control for KVM - module parameter.

3.2. EPT not supported or disabled
If EPT is not supported by the processor or disabled in the hypervisor, the system is fully protected. SMT can stay enabled and L1D flushing on VMENTER is not required.

EPT can be disabled in the hypervisor via the ‘kvm-intel.ept’ parameter.

3.3. SMT and EPT supported and active
If SMT and EPT are supported and active then various degrees of mitigations can be employed:

L1D flushing on VMENTER:

L1D flushing on VMENTER is the minimal protection requirement, but it is only potent in combination with other mitigation methods.

Conditional L1D flushing is the default behaviour and can be tuned. See Mitigation control on the kernel command line and Mitigation control for KVM - module parameter.

Guest confinement:

Confinement of guests to a single or a group of physical cores which are not running any other processes, can reduce the attack surface significantly, but interrupts, soft interrupts and kernel threads can still expose valuable data to a potential attacker. See 2. Guest VCPU confinement to dedicated physical cores.

Interrupt isolation:

Isolating the guest CPUs from interrupts can reduce the attack surface further, but still allows a malicious guest to explore a limited amount of host physical memory. This can at least be used to gain knowledge about the host address space layout. The interrupts which have a fixed affinity to the CPUs which run the untrusted guests can depending on the scenario still trigger soft interrupts and schedule kernel threads which might expose valuable information. See 3. Interrupt affinity.

The above three mitigation methods combined can provide protection to a certain degree, but the risk of the remaining attack surface has to be carefully analyzed. For full protection the following methods are available:

Disabling SMT:

Disabling SMT and enforcing the L1D flushing provides the maximum amount of protection. This mitigation is not depending on any of the above mitigation methods.

SMT control and L1D flushing can be tuned by the command line parameters ‘nosmt’, ‘l1tf’, ‘kvm-intel.vmentry_l1d_flush’ and at run time with the matching sysfs control files. See 4. SMT control, Mitigation control on the kernel command line and Mitigation control for KVM - module parameter.

Disabling EPT:

Disabling EPT provides the maximum amount of protection as well. It is not depending on any of the above mitigation methods. SMT can stay enabled and L1D flushing is not required, but the performance impact is significant.

EPT can be disabled in the hypervisor via the ‘kvm-intel.ept’ parameter.

Click to expand...

Default mitigations
The kernel default mitigations for vulnerable processors are:

PTE inversion to protect against malicious user space. This is done unconditionally and cannot be controlled.

L1D conditional flushing on VMENTER when EPT is enabled for a guest.

The kernel does not by default enforce the disabling of SMT, which leaves SMT systems vulnerable when running untrusted guests with EPT enabled.

The rationale for this choice is:

Force disabling SMT can break existing setups, especially with unattended updates.

If regular users run untrusted guests on their machine, then L1TF is just an add on to other malware which might be embedded in an untrusted guest, e.g. spam-bots or attacks on the local network.

There is no technical way to prevent a user from running untrusted code on their machines blindly.

It’s technically extremely unlikely and from today’s knowledge even impossible that L1TF can be exploited via the most popular attack mechanisms like JavaScript because these mechanisms have no way to control PTEs. If this would be possible and not other mitigation would be possible, then the default might be different.

The administrators of cloud and hosting setups have to carefully analyze the risk for their scenarios and make the appropriate mitigation choices, which might even vary across their deployed machines and also result in other changes of their overall setup. There is no way for the kernel to provide a sensible default for this kind of scenarios.

Click to expand...

eva2000 · Aug 21, 2018

Yup less performance with Intel Xeon Scalable cpus and L1TF mitigation fixes in place! The Performance Hit For A Xeon-Backed Ubuntu Linux VM With L1TF / Foreshadow Patches - Phoronix

For getting a better idea of the performance impact of mitigating L1TF/Foreshadow vulnerabilities I tested the Ubuntu patched kernel in a variety of configurations. First was the unmitigated Ubuntu 18.04 kernel, then Ubuntu 18.04 with the default out-of-the-box mitigation on the host and guest kernels, then having the host booted with the kernel parameter to force an L1D cache flush on every VMENTER rather than the default behavior of the conditional flushing, and then again when booting with l1tf=full for the full mitigation, which in the process also disables SMT/HT support.

The system used for this round of testing was a Tyan 1U server with two Intel Xeon Gold 6138 processors yielding a combined total of 40 cores / 80 threads, 128GB of RAM, and a Samsung 850 EVO SATA 3.0 SSD for storage. The KVM guest was the only VM running on the system during testing and was allowed access to 80% of the system's cores/threads (64 threads), 48GB of RAM, and 160GB of storage. Ubuntu 18.04 LTS was running on both the guest and host. The other default Spectre/Meltdown mitigations were present in their default behavior during testing including KPTI, __user pointer sanitization, and full generic IBPB IBRS_FW.
Click to expand...

Wow Apache took a much bigger hit in performance with L1TF fixes in place than Nginx did !

eva2000 · Aug 21, 2018

From Protecting Our Customers through the Lifecycle of Security Threats | Intel Newsroom - Intel hardware level (not software) protections aren't due until next generation Intel Xeon Scalable Cascade Lake

L1TF is also addressed by changes we are already making at the hardware level. As we announced in March, these changes begin with our next-generation Intel® Xeon® Scalable processors (code-named Cascade Lake), as well as new client processors expected to launch later this year.
Click to expand...

eva2000 · Aug 21, 2018

Intel Publishes L1TF and Foreshadow Performance Impacts

Intel L1TF Foreshadow Performance Impact Data
Most teams we speak to say that bare metal mitigations are a relatively low-performance impact. Intel’s data generally shows no more than a 1% performance impact if any at all. What was more interesting was what happens when you cannot mitigate simply through bare metal patches. In cases such as cloud providers, VPS providers, and enterprise clouds, one can have two VMs running on the same hyper-threaded core. Both Microsoft and Google said their cloud schedulers do not schedule VMs across hyper-threads on the same core. For small VPS providers, this is still common practice. The mitigation in those cases is turning off hyper-threading.

Intel published some startling numbers, in-line with our expectations, about the impacts of turning hyper-threading off.
Click to expand...

The Other Cost
We covered RedHat’s response to L1TF. At Hot Chips 30, Jon Masters on stage said that L1TF has already cost RedHat over 10,000 hours of engineering time.
Click to expand...

Hmm suggestion for some VPS providers turning of hyperthreading is needed !

Final Words
For companies like Google and Microsoft with the ability to get custom chips, and with custom schedulers that can ensure that VMs to not cross hyper-threading boundaries, this is something that can be relatively easily mitigated. For enterprise virtualization clouds, this may increase utilization of underutilized servers, and cause more server purchases in the future.

The more interesting impact is the average VPS provider. Very few users are running Linpack like workloads on VPS providers. For the average VPS provider without access to custom hardware and software, mitigation will mean turning off hyper-threading. These companies often run at a lower margin where seeing a 10-30% performance impact will kill their economic model. That means they will have to choose between turning off hyper-threading and mitigating L1TF / Foreshadow and keeping hyper-threading on to preserve their economic model. That is a dangerous wire to walk.
Click to expand...

eva2000 · Aug 21, 2018

Depends on what you're using bare metal dedicated has less of impact. I would get kernel updated as per 1st post of this thread and server rebooted. There isn't a full fix as yet as current full fix at VPS provider level would mean disabling Hyperthreading which would cut number of cpu threads in half for Intel based servers and I don't think any VPS provider would do that. Or need to find a VPS provider with AMD EPYC cpus instead of Intel cpus.

Log in / Register

Forums

Join the community today

Security August 2018: Kernel Security Update: L1TF - L1 Terminal Fault Attack CVE-2018-3620 & CVE-2018-3646

eva2000 Administrator Staff Member

Yum Updates For CVE-2018-3620 & CVE-2018-3646

Linode

CentOS 7

CentOS 6

L1TF detection script

L1TF - L1 Terminal Fault Attack CVE-2018-3620 & CVE-2018-3646 Information

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Linode

CentOS 7

CentOS 6

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Join the community today

Security August 2018: Kernel Security Update: L1TF - L1 Terminal Fault Attack CVE-2018-3620 & CVE-2018-3646

eva2000 Administrator Staff Member

Yum Updates For CVE-2018-3620 & CVE-2018-3646

Linode

CentOS 7

CentOS 6

L1TF detection script

L1TF - L1 Terminal Fault Attack CVE-2018-3620 & CVE-2018-3646 Information

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Linode

CentOS 7

CentOS 6

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.