Security Intel Processor Flaw 'kernel memory leaking' [Spectre & Meltdown]

eva2000 · Jan 11, 2018

Nvidia updates https://www.bizjournals.com/sanjose/news/2018/01/10/nvidia-gpu-chips-meltdown-spectre-intc-amd.html

In a security note Tuesday, Nvidia said its GeForce, Quadro, NVS, Tesla and GRID chips will need to be patched.

The twin security flaws, dubbed Meltdown and Spectre, impact the chips inside millions of smartphones, computers and servers sold within the last five years.

But CEO Jensen Huang followed up on Wednesday morning, saying that its blog post had been misinterpreted. "Our GPUs are immune, they are not affected by these security issues," Huang said in a presentation at the Consumer Electronics Show, according to MarketWatch. The chipmaker's release of software patches for the vulnerability is similar to the ones that software companies including Apple and Microsoft Corp. released, he said, per MarketWatch.
Click to expand...

Security Bulletin: NVIDIA Driver Security Updates for CPU Speculative Side Channel Vulnerabilities | NVIDIA

Bulletin Summary
NVIDIA is providing an initial security update to mitigate aspects of Google Project Zero’s January 3, 2018 publication of novel information disclosure attacks that combine CPU speculative execution with known side channels.

NVIDIA's core business is GPU computing.

We believe our GPU hardware is immune to the reported security issue. As for our driver software, we are providing updates to help mitigate the CPU security issue.

The vulnerability has three known variants:

Variant 1 (CVE-2017-5753): CPU mitigations are provided with the security update included in this bulletin. NVIDIA expects to work together with its ecosystem partners on future updates to further strengthen mitigations for affected CPUs.

Variant 2 (CVE-2017-5715): NVIDIA’s initial analysis indicates that NVIDIA software running on affected CPUs may require further updates. NVIDIA expects to work together with its ecosystem partners on this variant.

Variant 3 (CVE-2017-5754): At this time, NVIDIA has no reason to believe that NVIDIA software is vulnerable to this variant when running on affected CPUs.

For updates and additional information, actively monitor the NVIDIA Product Security page.
Click to expand...

buik · Jan 11, 2018

eva2000 said: ↑

LOL @bassie
Click to expand...

@eva2000 Grammatically worthless, punctuation bad. Resuming. Just as bad as.....

eva2000 · Jan 12, 2018

well getting back on topic KPTI + Retpoline Linux Benchmarking On Older Clarksfield / Penryn ThinkPads - Phoronix

The PCID - Process Context Identifier - support frequently talked about now following the KPTI patches is present with Intel CPUs since the Westmere launch in 2010. If you want to learn more about the inner-workings of PCID, there's this great and thorough explanation here. It was only with Linux 4.14 that the kernel squared away its PCID support, just in time for this flurry of kernel activity around Meltdown.

When it comes to pre-PCID hardware, the only systems I have that are still accessible and not boxed up or broken are some older ThinkPads, that thanks to the great build quality are still dependable and capable of handling some benchmarks in 2018.
Click to expand...

eva2000 · Jan 12, 2018

Linode updates Linode Blog » CPU Vulnerabilities: Meltdown & Spectre

Update: January 9, 2018

We spent today preparing the plan for deploying the Meltdown mitigations across Linode’s fleet. Over the course of the next day we will be implementing fixes to a subset of the fleet, monitoring for impact, and then continuing the rollout to the rest. The Meltdown mitigation requires reboots of our physical hardware which will reboot the Linodes hosted on them. A subset of Linodes in the Tokyo 1, Frankfurt, and Singapore data centers will be rebooted as part of this initial group. For those affected, you will receive a support ticket and email with scheduling information.

The reboots for this week address Meltdown only. We have testing and planning occurring in parallel to address Spectre. Additional reboots over the coming weeks will be required to properly mitigate all Spectre variants.

Update: January 10, 2018

The rollout to the subset of our fleet has gone well so far for the Meltdown mitigation. We are continuing with this plan and will be conducting reboots over the course of the next several days for the rest of the fleet. Customers affected will receive support ticket and emails with the reboot window for their Linodes with 24 hours of minimum notice.

Due to the ongoing nature of this issue, the following status page has been created.

We are posting a document soon that better talks about Meltdown and Spectre to show what it means to you and what you can do to prepare for it on your Linodes. We’ll share a link to this on an upcoming blog post.

Update: January 11, 2018

The mitigation process is continuing forward for Meltdown and we’re making progress each day across the fleet. There is a new guide available with more information on these vulnerabilities and how you can protect your Linode: What You Need to do to Mitigate Meltdown and Spectre.
Click to expand...

eva2000 · Jan 12, 2018

Intel's Security-First Pledge

An Open Letter from Brian Krzanich, CEO of Intel Corporation, to Technology Industry Leaders

Following announcements of the Google Project Zero security exploits last week, Intel has continued to work closely with our partners with the shared goal of restoring confidence in the security of our customers’ data as quickly as possible. As I noted in my CES comments this week, the degree of collaboration across the industry has been remarkable. I am very proud of how our industry has pulled together and want to thank everyone for their extraordinary collaboration. In particular, we want to thank the Google Project Zero team for practicing responsible disclosure, creating the opportunity for the industry to address these new issues in a coordinated fashion.

As this process unfolds, I want to be clear about Intel’s commitments to our customers. This is our pledge:

1. Customer-First Urgency: By Jan. 15, we will have issued updates for at least 90 percent of Intel CPUs introduced in the past five years, with updates for the remainder of these CPUs available by the end of January. We will then focus on issuing updates for older products as prioritized by our customers.

2. Transparent and Timely Communications: As we roll out software and firmware patches, we are learning a great deal. We know that impact on performance varies widely, based on the specific workload, platform configuration and mitigation technique. We commit to provide frequent progress reports of patch progress, performance data and other information. These can be found at the Intel.com website.

3. Ongoing Security Assurance: Our customers’ security is an ongoing priority, not a one-time event. To accelerate the security of the entire industry, we commit to publicly identify significant security vulnerabilities following rules of responsible disclosure and, further, we commit to working with the industry to share hardware innovations that will accelerate industry-level progress in dealing with side-channel attacks. We also commit to adding incremental funding for academic and independent research into potential security threats.

We encourage our industry partners to continue to support these practices. There are important roles for everyone: Timely adoption of software and firmware patches by consumers and system manufacturers is critical. Transparent and timely sharing of performance data by hardware and software developers is essential to rapid progress.

The bottom line is that continued collaboration will create the fastest and most effective approaches to restoring customer confidence in the security of their data. This is what we all want and are striving to achieve.

— Brian Krzanich
Click to expand...

buik · Jan 12, 2018

What I have my concern with:
If you read quickly you think about speed improvements to fix the performance that occurs after applying security fixes.

The article is talking about: "We know that impact on performance varies widely, based on the specific workload, platform configuration and mitigation technique. We commit to provide frequent progress reports of patch progress, performance data and other information."

This could be a data analysis of the improved secured firmware with speed adjustments or an analysis that with security adjustment and without speed adjustments. And that the performance loss is negligible according to Intel.

And if they make speed improvements in their firmware.
Then the question is whether you get them.

There is no problem for servers, the support of new firmwares is well regulated from the manufacturer.

But consumers hardware. I don't believe that you will get a firmware update with Intel improvements on your 3 year old Acer laptop.

Not that Intel does not offer that, no Acer does not carry it through.
There you are with your slow device. Thats the problem.

eva2000 · Jan 13, 2018

Yeah from Intel CEO latter seems to mean that by Jan 31, 2018 all cpus within past 5yrs will get updates and older cpus after that date.

AMD more vulnerable that initially thought AMD CPUs Are Potentially Vulnerable To Spectre / Variant 2 - Phoronix ?

AMD's updated statement by Mark Papermaster reads, "While we believe that AMD’s processor architectures make it difficult to exploit Variant 2, we continue to work closely with the industry on this threat. We have defined additional steps through a combination of processor microcode updates and OS patches that we will make available to AMD customers and partners to further mitigate the threat."

AMD will be rolling out new microcode updates for Ryzen/EPYC starting this week. For Linux users they also acknowledge the work being done on Retpoline for mitigating the threat.

The Retpoline Linux patches continue to be worked on but have yet to be mainlined. It's looking like Retpoline will likely land for Linux 4.16. There are also needed LLVM / GCC patches too, which have yet to land but hopefully will still make it for the upcoming GCC 8.0 release.

AMD's updated processor security statement can be read at AMD.com. AMD remains certain they are not vulnerable to the Meltdown vulnerability affecting Intel CPUs.
Click to expand...

eva2000 · Jan 13, 2018

updates from Online.net Important note about the security flaw impacting ARM & Intel hardware

Scaleway ARMv8 servers

From our latest information, Cavium ThunderX should not be impacted by Meltdown, Spectre 1 and Spectre 2. It means that the complete range of Scaleway ARMv8 servers is not affected by any of the three vulnerabilities.
Click to expand...

peformance impact from graph looks like 5x times higher cpu usage there !

Update #22 (Web Hosting Important Performance Issue) - 01/11/18 6:00pm UTC
Web Hosting

We finished the first part of the maintenance and upgraded in emergency some servers to more powerful ones. Our teams are carefully monitoring the performance impact.

Meanwhile, our customers faced major performances and stability (segfaults, OOM, etc.) issues. due to this situation, starting tomorrow, we will migrate all Web Hosting servers to more powerful ones as we don't think a fix to address the PTI performance regression will be available in the coming days.

During the maintenances, a downtime of 30 to 60 minutes per site is to expect.

Following are the graphics from the past days showing the pre/post kernel upgrade (with & without PTI enabled). We clearly see the performance impact, bigger than expected, when upgrading to 4.41.11 with PTI enabled. Main workload on those servers is PHP.

Our teams is working hard to find a way to address the performance issue as fast as possible.
Click to expand...

eva2000 · Jan 13, 2018

Not good news at all, Intel cpu microcode updates causing system crashes Intel Xeon E5 V3 and V4 Servers See More Reboots After Meltdown and Spectre Fixes

Recently, we became aware of a stability issue that was confirmed by Intel, on the Broadwell-EP and Haswell-EP (Xeon E5 V3 and V4 series) CPUs that is causing stability issues due to new Intel microcode released as part of the patching process for Meltdown and Spectre. We are seeing evidence that the recently released fixes are causing more crashes and reboots on impacted systems.
Click to expand...

Our takeaway is that there must be a reason vendors have still not updated the Haswell/ Broadwell generation microcode. Then, Intel released a statement on the matter:

We have received reports from a few customers of higher system reboots after applying firmware updates. Specifically, these systems are running Intel Broadwell and Haswell CPUs for both client and data center. We are working quickly with these customers to understand, diagnose and address this reboot issue. If this requires a revised firmware update from Intel, we will distribute that update through the normal channels. We are also working directly with data center customers to directly discuss the issue.
Click to expand...

(Source: Intel)
Click to expand...

eva2000 · Jan 13, 2018

DigitalOcean A Message About Intel Security Findings

Update Friday, January 12th, 2018:

Our engineering team continues to procure and test patches as they become available and we have a significant amount of resources dedicated to this task. While numerous patches have been rolled out for Meltdown, mitigations for Spectre are still sparse and raw. Unfortunately, many distributions have not been able to roll out a full set of patches to address all 3 variants of the exploits. We will continue preliminary testing through the week of January 15th. These tests will have no customer impact, and will be focused on getting machines up and running in these new environments.

Intel released a microcode update this week, unfortunately the update was determined to cause stability issues for other Intel customers, and has since been pulled back. DigitalOcean did not apply this microcode to our fleet, and we are awaiting the release of new a microcode. Once we have the final microcode in hand, we will begin performance/regression testing to validate the update in our environment.
Click to expand...

eva2000 · Jan 13, 2018

Just amazing the pace at which Michael from Phoronix pumps out benchmark articles Debian vs. Ubuntu vs. CentOS vs. openSUSE vs. Clear Linux Post-Meltdown Performance - Phoronix

This article is offering a first-look at the performance of various Linux distributions before and after being patched to address the recent CPU vulnerability disclosures. Most Linux distributions have been shipping KPTI/KAISER support while others are still working on Spectre mitigation with the Retpoline patches, which have yet to be merged to the mainline Linux kernel or the related GCC compiler patches. Hopefully the Retpoline work will settle down in the weeks ahead. A rough recap for these distributions comes down to:

Canonical on 9 January pushed Meltdown (KPTI) patches to the kernels of their supported Ubuntu releases. Spectre/Retpoline patches are still to come.

Debian has patched their Stretch, Jessie, and Wheezy kernels with KPTI for Spectre mitigation but Meltdown is still a work-in-progress.

CentOS has released updated CentoS 6 and 7 kernels based upon RHEL 6/7 fixes. CentOS/RHEL has also published update CPU microcode from Intel.

Clear Linux 20240 added KPTI and Retpoline support for mitigating Meltdown and Spectre. The Retpoline support does include GCC 7.2 that is patched with the necessary changes too. This is some of the most complete protection currently available for Linux distributions with not many shipping Retpoline patches yet let alone a patched compiler with neither the Linux kernel nor GCC seeing support mainlined yet.

SUSE and openSUSE distributions are patched since last week with KPTI/KAISER for their respective supported kernels as well as issuing new AMD/Intel CPU microcode. Retpoline support has yet to land.
Click to expand...

just a typo for Debian as Meltdown fixes are KPTI and waiting on Spectre so mixed them up in statement

Debian has patched their Stretch, Jessie, and Wheezy kernels with KPTI for Spectre mitigation but Meltdown is still a work-in-progress.
Click to expand...

eva2000 · Jan 14, 2018

Woah Solarwinds folks report huge performance losses on Amazon AWS Visualizing Meltdown on AWS - AppOptics Blog

This post highlights the impacts we at SolarWinds Cloud noticed across our AWS infrastructure during the last several weeks due to Meltdown. We, along with many SaaS companies were impacted by these changes and suffered partial downtime due to AWS efforts to mitigate Meltdown. We’ll be posting a full post-mortem for our outage in the coming days.
Click to expand...

As you can see from the following chart taken from a Python worker service tier, when we rebooted our PV instances on Dec 20th ahead of the maintenance date, we saw CPU jumps of roughly 25%.
Click to expand...

Our Cassandra tiers that we use for TSDB storage were also impacted across the board. We saw CPU spikes of roughly 25% CPU on m4.2xlarge instances and similar spikes on other instance types. For now we have enough extra capacity to absorb these spikes, but we are actively looking at how we can reduce the increased load.
Click to expand...

The impacts on performance were also seen in our own service tiers. One internal tier, which interacts with Cassandra, saw a 45% jump in p99 latency committing records to Cassandra:
Click to expand...

We also detected spikes in latency on AWS managed services, like AWS Elasticache Memcached. This snapshot shows an 8% bump in CPU on a given memcached, but that was almost a 100% increase:
Click to expand...

@Matt IIRC your work place also would have been impacted by this ?

Matt · Jan 14, 2018

eva2000 said: ↑

@Matt IIRC your work place also would have been impacted by this ?
Click to expand...

Solarwinds is running on internal hardware. We are however, running Splunk in the AWS Cloud..........

buik · Jan 14, 2018

Intel is quietly advising some customers to hold off installing patches that address new security flaws affecting virtually all of its processors.

Intel just warned that its patches can cause problems for some chips

buik · Jan 14, 2018

Another post because of another topic.
About Linode and Spectre & Meltdown.

All my German servers are being migrated to new hardware with the latest patches.

"As part of our ongoing efforts to patch the recent Meltdown and Spectre CPU vulnerabilities, your Linode has been scheduled for a migration that will relocate your Linode to a physical machine with our latest security patches already in place."

All English servers are patched on site.

"In order to address the disclosed vulnerabilities, the physical hardware on which your Linode resides will need to undergo maintenance. This update will require at least two separate reboots in order to fully mitigate the vulnerabilities."

Different approach. What about you, fellow members?

eva2000 · Jan 14, 2018

Yeah Intel not having much luck with microcode and bios updates right now. I haven't received any emails from Linode for mine but majority are on Intel E5-2680v2 Ivybridge so Intel probably hasn't gotten to those older cpus yet.

hmm stackoverflow updated kernels resulted in between 65-275% increase in cpu utilisation at peak https://twitter.com/mnewswanger/status/951518987925499904 !

Patched the canaries for @ StackOverflow web servers for #Meltdown today. Servicing the same number of requests, CPU is up by ~65%, with peak CPU spiking to 275% of previous peak.
Click to expand...

https://twitter.com/mnewswanger/status/951540572799602690

Breaking down requests for @ StackOverflow web servers: Pre-Meltdown Patch -> Post Meltdown Patch (Increase %) Response time: 18ms -> 22ms (22%) Avg SQL Call: .73ms -> .83ms (13%) Avg Elastic Call: 12.56ms -> 19.71ms (57%)
Click to expand...

really messy The Hidden Toll of Fixing Meltdown and Spectre

eva2000 · Jan 14, 2018

Matt said: ↑

Solarwinds is running on internal hardware. We are however, running Splunk in the AWS Cloud..........
Click to expand...

AWS Cloud ouch.. cpu utilisation up ?

eva2000 · Jan 14, 2018

Intel providing some benchmark numbers before vs after Intel Security Issue Update: Initial Performance Data Results for Client Systems

We shared some initial assessments of performance impactyesterday. We now have additional data on some of our client platforms, and we are sharing that with you today. This is part of our ongoing effort to keep you apprised through frequent updates. We plan to share initial data on some of our server platforms in the next few days. Please know we are working around the clock to generate the data that you want to see as fast as possible. As we endeavor to continue our pace, please understand that – as is common in testing of this type – our results may change as we conduct additional testing.

Jan. 10 Performance Data Results

Today we are sharing data on several 6th, 7th and 8th Generation Intel® Core™ processor platforms using Windows* 10. We previously said that we expected our performance impact should not be significant for average computer users, and the data we are sharing today support that expectation on these platforms.
Click to expand...

eva2000 · Jan 14, 2018

some relief for AWS users Ian Chan on Twitter

Big update here! Looks like AWS/EC2 just deployed an updated #Spectre patch (perhaps Retpoline-ish), addressing high IO HVM instance performance issues! Big credit to Amazon for addressing performance needs & concerns so quickly and diligently.
Click to expand...

Xon · Jan 14, 2018

I've started getting maintenance window notifications from Linode for my various VMs

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

Security Intel Processor Flaw 'kernel memory leaking' [Spectre & Meltdown]

eva2000 Administrator Staff Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Matt Well-Known Member

buik “The best traveler is one without a camera.”

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Xon Active Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

Security Intel Processor Flaw 'kernel memory leaking' [Spectre & Meltdown]

eva2000 Administrator Staff Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Matt Well-Known Member

buik “The best traveler is one without a camera.”

buik “The best traveler is one without a camera.”

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Xon Active Member

.