Security Intel Processor Flaw 'kernel memory leaking' [Spectre & Meltdown]

eva2000 · Jan 25, 2018

Linode Blog » An Update to Meltdown: Mitigation Deployed

Per our last post, we wanted to provide a brief follow up on the status of the Meltdown mitigation at Linode. We are happy to announce that we have successfully mitigated Meltdown across our entire KVM fleet. There are a small number of Xen Linodes that will be migrated to KVM over the coming weeks. We’ll reach out to these customers shortly in order to let them know about their migration window.

We strongly felt and still feel that taking steps to mitigate known vulnerabilities sooner rather than later was the appropriate course of action. We want to ensure that our customers are always safe and secure rather than wait for uncertain mitigations to come to market in the future without a firm timeline.

We are now evaluating mitigation opportunities for the two known Spectre variants. This will be done via a number of channels – kernel updates, microcode updates, a cloud hosting industry best practice sharing community, and others. We anticipate this will take a few additional weeks of investigation and thorough testing. Once a Spectre mitigation is established, we will publish another blog post outlining our plan to address it.

Filed under: announcements, security by Tory Kulick
Click to expand...

eva2000 · Jan 26, 2018

The impromptu Slack war room where ‘Net companies unite to fight Spectre-Meltdown

In June 2017, Intel learned of these threats from researchers who kept the information under wraps so hardware and operating system vendors could furiously work on fixes. But while places like Amazon, Google, and Microsoft were clued in early because of their "Tier 1" nature, most smaller infrastructure companies and data center operators were left in the dark until the news broke on January 3. This sent many organizations immediately scrambling: no warning of the exploits came before proof-of-concept code for exploiting them was already public.

Tory Kulick, director of operations and security at the hosting company Linode, described this as chaos. "How could something this big be disclosed like this without any proper warning? We were feeling out of the loop, like 'What did we miss? Which of the POCs [proofs of concept of the vulnerabilities] are out there now?' All that was going through my mind."

"When this stuff broke, nobody had heard a peep from Intel or from anybody else directly," Zachary Smith, CEO of the hosting service Packet, told Ars. "All we could see is what was going on Google's blog about how to exploit this stuff. So we were all scrambling. The big guys—Google, Amazon, and Microsoft—have had 60 days at least of prep time, and we've had negative prep time."
Click to expand...

So to overcome the chaos, these companies did something kind of novel: they decided to work side by side. A group of second-tier service providers banded together to formally share information about patches from various vendors, metrics on their impact, and best practices for rolling them out. Over the past week, this ad hoc war council—a group of at least 25 companies operating over a simple shared Slack—has attracted a number of higher-profile members, including Netflix and Amazon Web Services. And this impromptu centralization has even allowed the researchers originally behind the Spectre/Meltdown discovery to interact directly with affected companies.

"Probably one of the best things that came out of the whole ordeal was this cross-cloud hosting collaboration," said Linode's Kulick. "Sharing links and things like that was absolutely critical."
Click to expand...

eva2000 · Jan 27, 2018

Linux 3.17 To 4.15 Kernel Benchmarks On Intel Gulftown & Haswell - Phoronix

Here is a look at how the Linux kernel performance has evolved since the release of Linux 3.17 in October 2014. With all the major kernel releases over the past 3+ years, here is how the performance compares using two very different Intel Gulftown and Haswell systems.
Click to expand...

The two systems consisted of:

Core i7 990X - The high-end "Gulftown" processor from 2011. The Core i7 990X is a six-core processor plus Hyper Threading and has a 3.46GHz base clock frequency with 3.73GHz turbo frequency. The i7-990X system was paired with an MSI X58M motherboard, 3 x 4GB DDR3-1666 memory, and PNY CS1211 120GB SATA 3.0 SSD.

Core i7 5960X - The Haswell-E system providing a more recent look at Linux performance potential and as a refresher is an eight core part with Hyper Threading, 3.0GHz base frequency, and 3.5GHz turbo frequency. This system was paired with the ASRock X99 Extreme3 motherboard, 4 x 4GB DDR4-3000 memory, 120GB Intel SSDSC2BW12, and AMD FirePro V7900 graphics card.
Click to expand...

eva2000 · Jan 27, 2018

There was a recent spike in traffic and seems Ubuntu Wiki also lists my Centmin Mod Nginx Kernel KPTI benchmark results at SecurityTeam/KnowledgeBase/SpectreAndMeltdown/PublishedApplicationData - Ubuntu Wiki now

Percona MySQL is listed with 15-25% hit too on Haswell cpu systems so MariaDB and Oracle MySQL probably has same performance impact with Kernel updates 20-30% Performance Hit from the Spectre Bug Fix on Ubuntu

eva2000 · Jan 29, 2018

Intel reportedly notified Chinese companies of chip security flaw before the U.S. government

The Journal is reporting that Intel notified some of its customers of the security flaws in its processors, dubbed Spectre and Meltdown, but left out the U.S. government as part of that. Some of the companies Intel notified included Chinese technology companies, though the report suggests there is no evidence that any information was misused. An Intel spokesperson told The Journal that the company wasn’t able to tell everyone it planned because the news was made public earlier than expected.
Click to expand...

eva2000 · Feb 1, 2018

MariaDB benchmarking KPTI kernel updated performance impact for Meltdown & Spectre Meltdown Vulnerability Impact On MariaDB Server | MariaDB

eva2000 · Feb 1, 2018

Small Datum: Meltdown vs storage

Meltdown vs storage

tl;dr - sysbench fileio throughput for ext4 drops by more than 20% from Linux 4.8 to 4.13
Click to expand...

eva2000 · Feb 1, 2018

Finding out the MySQL performance regression due to kernel mitigation for Meltdown CPU vulnerability – I used to be a MySQL DBA for Hire

After learning about Meltdown and Spectre, I waited patiently to get a fix from my OS vendor. However, there were several reports of performance impact due to the kernel mitigation- for example on the PostgresQL developers mailing list there was reports of up to 23% throughput loss; Red Hat engineers report a regression range of 1-20%, but setting OLTP systems as the worse type of workload. As it will be highly dependent on the hardware and workload, I decided of doing some test myself for the use cases I need.
Click to expand...

Xon · Feb 3, 2018

@eva2000 you can load tcmalloc into MariaDB very easily without recompiling;
Code:
yum install -y gperftools-libs.x86_64
Append to /etc/my.cnf or /etc/my.cnf.d/server.cnf
Code:
[mysqld_safe]
malloc-lib=/usr/lib64/libtcmalloc_minimal.so.4
Restarting mysql will then show something like:
Code:
>service mysql restart
Shutting down MySQL.. SUCCESS!
Starting MySQL.180203 07:47:29 mysqld_safe Adding '/usr/lib64/libtcmalloc_minimal.so.4' to LD_PRELOAD for mysqld
180203 07:47:29 mysqld_safe Logging to '/var/log/mysql/mysqld.log'.
180203 07:47:29 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Double check via:
Code:
> pmap `pidof mysqld` | grep libtcmalloc
00007f9c8180e000    152K r-x--  /usr/lib64/libtcmalloc_minimal.so.4.1.0
00007f9c81834000   2048K -----  /usr/lib64/libtcmalloc_minimal.so.4.1.0
00007f9c81a34000      8K rw---  /usr/lib64/libtcmalloc_minimal.so.4.1.0
This can also be done with jemalloc too

A brief look shows Mariadb 10.1.x use jemalloc, while Mariadb 10.2.x do not

eva2000 · Feb 3, 2018

Cheers @Xon yeah i have dabbled with malloc-lib in the past just didn't recall the change in mariadb 10.2. Will have revisit some mysql testing i guess.

Centmin Mod MariaDB 10.0/10.1 and Nginx already use jemalloc instead of glibc

Code (Text):

lsof | grep jemalloc
nginx     21955            root  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21956           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21957           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21958           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21960           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21961           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21962           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21964           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21965           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
nginx     21967           nginx  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582           mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30583     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30584     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30585     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30586     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30587     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30588     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30589     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30590     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30591     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30592     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30593     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30594     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30595     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30597     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30598     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30599     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30600     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30601     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30602     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30603     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30604     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30605     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30606     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30607     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30608     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30609     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30610     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30611     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1
mysqld    30582 30879     mysql  mem       REG                9,0    212096      18167 /usr/lib64/libjemalloc.so.1

eva2000 · Feb 5, 2018

Jon Masters On Understanding Spectre & Meltdown CPU Vulnerabilities - Phoronix

Arguably the most interesting keynote at this year's FOSDEM event was Red Hat's Jon Masters talking about the Spectre and Meltdown CPU vulnerabilities on an interesting technical level.

While Jon Masters is mostly known for his involvement with Fedora/RedHat on ARM hardware, he was the lead for Red Hat's mitigation efforts around the Meltdown and Spectre vulnerabilities that rocked the world last month.
Click to expand...

eva2000 · Feb 12, 2018

Amazon EC2 Cloud Compute Performance: December vs. February - Phoronix

For those wondering how Amazon's Elastic Compute Cloud (EC2) performance is now after being mitigated for the recent Spectre and Meltdown CPU vulnerabilities, here are benchmarks of five Linux distributions comparing the performance to last December prior to the Linux kernel mitigations coming about to now.
Click to expand...

Ouch Redis definitely got hammered by Spectre/Meltdown Kernel patches in terms of reduced performance

eva2000 · Feb 12, 2018

KPTI/KAISER Meltdown Initial Performance Regressions

In this post I'll look at the Linux kernel page table isolation (KPTI) patches that workaround Meltdown: what overheads to expect, and ways to tune them. Much of my testing was on Linux 4.14.11 and 4.14.12 a month ago, before we deployed in production. Some older kernels have the KAISER patches for Meltdown, and so far the performance overheads look similar. These results aren't final, since more changes are still being developed, such as for Spectre.

Note that there are potentially four layers of overhead for Meltdown/Spectre, this is just one. They are:

Guest kernel KPTI patches (this post)

Intel microcode updates

Cloud provider hypervisor changes (for cloud guests)

Retpoline compiler changes

KPTI Factors
To understand the KPTI overhead, there are at least five factors at play. In summary:

Syscall rate: there are overheads relative to the syscall rate, although high rates are needed for this to be noticable. At 50k syscalls/sec per CPU the overhead may be 2%, and climbs as the syscall rate increases. At my employer (Netflix), high rates are unusual in cloud, with some exceptions (databases).

Context switches: these add overheads similar to the syscall rate, and I think the context switch rate can simply be added to the syscall rate for the following estimations.

Page fault rate: adds a little more overhead as well, for high rates.

Working set size (hot data): more than 10 Mbytes will cost additional overhead due to TLB flushing. This can turn a 1% overhead (syscall cycles alone) into a 7% overhead. This overhead can be reduced by A) pcid, available in Linux 4.14, and B) Huge pages.

Cache access pattern: the overheads are exacerbated by certain access patterns that switch from caching well to caching a little less well. Worst case, this can add an additional 10% overhead, taking (say) the 7% overhead to 17%.

To explore these I wrote a simple microbenchmark where I could vary the syscall rate and the working set size (source). I then analyzed performance during the benchmark (active benchmarking), and used other benchmarks to confirm findings. In more detail:
Click to expand...

eva2000 · Feb 13, 2018

Love the benchmarks that Phoronix does 19-Way CPU Comparison On Ubuntu With Linux 4.15 - Phoronix

Following the release of the Linux 4.15 kernel with KPTI and Retpoline introduction, many Phoronix readers were interested in seeing a fresh Linux CPU performance comparison. For those reasons plus in preparing for the Raven Ridge testing, here are benchmarks of 19 different systems when using Ubuntu x86_64 with the Linux 4.15 stable kernel.
Click to expand...

The CPUs for this interesting test roundabout today included:

- A10-7870K
- FX-8350
- Ryzen 3 1200
- Ryzen 3 1300X
- Ryzen 7 1700
- Ryzen 7 1800X
- Threadripper 1950X
- EPYC 7601
- Core i3 7100
- Core i3 8100
- Core i5 4670
- Core i5 8400
- Core i7 4770K
- Core i7 5960X
- Core i7 7740X
- Core i7 8700K
- Core i9 7980XE
- Xeon Silver 4108
- 2 x Xeon Gold 6138
Click to expand...

Apachebench Nginx tests were single threaded so only really testing each cpu's single threaded performance.

eva2000 · Feb 15, 2018

woah huge overhead for MyISAM

MyISAM and KPTI - Performance Implications From The Meltdown Fix - MariaDB.org

Recently we had a report from a user who had seen a stunning 90% performance regression after upgrading his server to a Linux kernel with KPTI (kernel page-table isolation – a remedy for the Meltdown vulnerability).

A big deal of those 90% was caused by running in an old version of VMware which doesn’t pass the PCID and INVPCID capabilities of the CPU to the guest. But I could reproduce a regression around 40% even on bare metal.

Why?
If we look at the handler status variables, we can see that for 8K rows the query does more than 50 million calls to Handler_read_rnd_next. For MyISAM such a handler call results in a call to fget() which in turn results in a __fget syscall.

This is so, because the MyISAM engine does not have a row cache. While it caches index pages in the Key Buffer, there is no such cache for row data. For that it relies on the generic page cache in the operation system. That works pretty well, however since that cache is in the kernel, there is the syscall barrier between the MariaDB server and the cache.

The page table isolation introduced with KPTI increases the cost for a syscall. Hence a workload like the one above, where many MyISAM rows are read in a tight loop, becomes notably slower. The relative slowdown is actually bigger when the row is already in the page cache!

How To Get Around
There is no way to fix that in the MyISAM engine, as it would require a complete redesign. But the good thing is, that most other engines do have a row cache. For InnoDB it is the InnoDB Buffer Pool and for ARIA there is the ARIA Page Cache.
Click to expand...

eva2000 · Feb 15, 2018

Good news on Kernel front Spectre & KPTI Get More Fixes In Linux 4.16, Offsets Some KVM Performance Losses - Phoronix

When it comes to the kernel page table isolation (PTI/KPTI) code, there are several fixes. But more of the work in this pull request from Ingo Molnar is centered on Spectre. For reducing the speculation attack surface with Spectre, extra registers beyond syscall arguments are cleared as well as registers for compat syscalls and registers for exceptions/interrupts.

This Git merge also updates/corrects the speculation control microcode blacklist based upon the latest microcode information from the CPU vendors about known microcodes where the speculation control should be working fine or not.

Also notable is that KVM's Spectre helpers are made into inline functions to increase the performance and they say should be closer to Linux 4.14 kernel performance levels or when booting with nospectre_v2 for bypassing the Variant Two safeguards.
Click to expand...

Xon · Feb 17, 2018

I've noticed my XenForo dev environment @ home is now vastly slower when doing template rebuilds

eva2000 · Feb 17, 2018

Xon said: ↑

I've noticed my XenForo dev environment @ home is now vastly slower when doing template rebuilds
Click to expand...

Yeah the impact is being felt everywhere Redis server got hit hard too so I suspect it is also affecting Redis Xenforo caching performance too.

eva2000 · Feb 18, 2018

seems like round 2 is about to start Meltdown-Spectre flaws: We've found new attack variants, say researchers | ZDNet

Researchers have developed a tool to uncover new ways of attacking the Meltdown and Spectre CPU side-channel flaws, which may force chipmakers like Intel to re-examine already difficult hardware mitigations.

The tool allowed the researchers to synthesize a software-attack based on a description of a CPU's microarchitecture and an execution pattern that could be attacked.

Though the software attack is specific to a microarchitecture and represent exploits "in their most abstracted form", they can be used to develop fully fledged attacks.
Click to expand...

"By exploiting cache invalidations, MeltdownPrime and SpectrePrime -- two variants of Meltdown and Spectre, respectively -- can leak victim memory at the same granularity as Meltdown and Spectre while using a Prime+Probe timing side-channel."

The researchers developed proof-of-concept malware for SpectrePrime and ran it on a MacBook with an Intel Core i7 Processor running a version of macOS Sierra that hadn't received Apple's Meltdown and Spectre patches.

"Averaged over 100 runs, we observed SpectrePrime to achieve the same average accuracy as Spectre on the same hardware -- 97.9 percent for Spectre and 99.95 percent for SpectrePrime," they write.

The mitigations for Meltdown and Spectre have involved a combination of software fixes, such as Microsoft and Linux versions of 'kernel page table isolation', and hardware fixes such as Intel's microcode updates. Both can cause performance overheads.

But while existing software mitigations will probably suffice for these new variants of Meltdown and Spectre, chipmakers like Intel and AMD are likely to need to develop different hardware mitigations, according to the researchers.
Click to expand...

eva2000 · Feb 23, 2018

Woah Intel didn't tell CERTS, govs, about Meltdown and Spectre because they couldn't help fix it. Bad Intel !

Letters sent to the United States Congress by Intel and the other six companies in the Meltdown/Spectre disclosure cabal have revealed how and why they didn't inform the wider world about the dangerous chip design flaws.

Republican members of the House Energy and Commerce Committee sent letters to the seven in January, to seek answers about the reasons they chose not to disclose the flaws and whether they felt their actions were responsible and safe.

All the letters go over old ground: Google Project Zero spotted the design errors, told Intel, which formed a cabal comprising itself, Google, AMD, Arm, Apple, Amazon and Microsoft. The gang of seven decided that Project Zero's 90-day disclosure deadline had to be extended to January, then spoke to others to help them prepare fixes. But stray posts and sharp-eyed Reg hacks foiled that plan as we broke the news on January 3rd.

The flaws are so serious that Congressman insisted the seven explain themselves, and now we have the letters in which they attempt to do so, with links on this page .

Intel's letter (PDF) is the most informative because it reveals "Before the leak, Intel disclosed information about Spectre and Meltdown only to companies who could assist Intel in enhancing the security of technology users."
Click to expand...

Log in / Register

Forums

Get the most out of your Centmin Mod LEMP stack

Security Intel Processor Flaw 'kernel memory leaking' [Spectre & Meltdown]

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Xon Active Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Xon Active Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Get the most out of your Centmin Mod LEMP stack

Security Intel Processor Flaw 'kernel memory leaking' [Spectre & Meltdown]

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Xon Active Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

Xon Active Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.