In this week’s catchup mega-issue: Linux 4.12-rc1 (including a full summary of the 4.12 merge window), Linux 4.11 final is released, saving TLB flushes, various ongoing development, and a bunch of announcements.
This podcast is a free service that I provide to the community in my spare time. It takes many, many hours to prepare and produce a single episode, much more during the merge window. This means that when I have major events (such as Red Hat Summit followed by OpenStack Summit) it will be delayed, as was the case this last week week. Over the coming months, I hope to automate the production in order to reduce the overhead but there will be some weeks where I need to skip a show. I am however covering the whole 4.12 merge window regardless. So while I would usually have just moved on, the circumstance warrants a mega-length catchup episode. I hope you’re still awake by the end.
Linus Torvalds announced Linux 4.12-rc1, “one day early, because I don’t like last-minute pull requests during the merge window anyway, and tomorrow is mother’s day [in the US], so I may end up being roped into various happenings”. He also noted “Besides, this has actually been a pretty large merge window, so despite there technically being time for one more day of pulls, I actually do have enough changes already. So there.” In his announcement, he says things look smooth so far, but calls those also “Famous last words”. Finally, he calls out the “odd” diffstat which is dominated by the AMD Vega10 headers. As was noted in the pull requests, unlike certain other graphics companies, AMD actually provides nice automatically generated headers and other information about their graphics chipsets, which is why the Vega10 update is plentiful.
Later in the day yesterday, following the 4.12-rc1 announcement, Guenter Roeck posted “watchdog updates for v4.12”, and Jon Mason posted “NTB bug fixes for vv4.12”, along with an apologies for tardiness.
Linus Torvalds announced Linux 4.11 noting that the extra week due a (rare-ish) “rc8” (Release Candidate 8) meant that he had felt “much happier releasing a final 4.11 now”. As usual, Linux Kernel Newbies has a writeup of 4.11, here: https://kernelnewbies.org/Linux_4.11
Greg K-H (Kroah-Hartman) announced Linux 4.4.68, 4.9.28, 4.10.16, and 4.11.1. He later sent “Bad signatures on recent stable updates” in which he noted that “The stable kernels I just released have had signatures due to a mixup using pixz in the new kernel.org backend. It will be fixed soon…”, which were later corrected. He would like to hear from anyone still seeing problems.
Greg also announced (separately) Linux 3.18.52. While Jiri Slaby announced Linux 3.12.74.
Stephen Hemminger announced iproute2 4.11 matching the new kernel release.
Michael Kerrisk announced map-pages-4.11.
Steven Rostedt announced trace-cmd 2.6.1.
Steven also announced Linux 4.4.66-rt79, 3.18.51-rt57, and 3.12.73-rt98 (preempt-rt) kernels.
Con Kolivas posted an updated version of his (renamed) “MuQSS CPU scheduler” [renamed from the BFS – Brain F*** Scheduler] in Linux 4.11-ck1.
Karel Zak announced util-linux v2.30-rc1, which includes a fix to libblkid that “has been fixed to extract LABEL= and UUID= from UDF rather than ISO9660 header on hybrid CDROM/DVD media. This change makes UDF media on Linux user-space more compatible with another operation systems.” but he calls it out since it could also introduce regressions for some other users.
Junio C Hamano announced Git version 2.13.0. Separately, he released maintenance versions of “Git v2.12.3 and others” which include fixes for
“a recently disclosed problem with “git shell”, which may allow a user who comes over SSH to run an interactive pager by causing it to spawn “git upload-pack –help” (CVE-2017-8386).”
Jan Kiszka announced version 0.7 of the Jailhouse hypervisor, which includes various debug and regular console driver updates and gcov debug statistics.
Bartosz Golaszewski announced libgpiod v0.2: “The most prominent new feature is the test suite working together with the gpio-mockup module”.
Christoph Hellwig notes that the Open OSD [an in-kernel OSD – Object-Based Storage Device] SCSI initiator library for Linux seems to be dead. He does this by posting a patch to the MAINTAINERS file “update OSD entries” in which he removes the (now defunct) open-osd.org website, and the bouncing email address for Benny Halevy. Benny appeared and ACKed.
In a similar vain, Ben Hutchings pondered aloud about the “Future of liblockdep”, which apparently “hasn’t been buildable since (I think) Linux
4.6”. Sasha Levin said things would be cleaned up promptly. And they were, with a pull request soon following with fixes for Linux 4.12.
Masahiro Yamada posted an RFC patch entitled “Increase Minimal GNU Make version for Linux Kernel from 3.80 to 3.81” in which he essentially noted that the kernel hadn’t actually worked with 3.80 (which is 15 years old!) in a very long time, but instead actually really needs 3.81 (which was itself released in 2006). It was apparently “broken” 3 years ago, but nobody noticed. Neither Greg K-H (Kroah-Hartman) nor Linus seemed to lose any sleep over this, with Linus saying “you make a strong case of “it hasn’t worked for a while already and nobody even noticed””.
Paolo Bonzini posted “CFP: KVM Forum 2017” announcing that the KVM Forum will be held October 25-27 at the Hilton in Prague, CZ, and that all submissions for proposed topics must be made by midnight June 15.
Thomas Gleixner announced “[CFP] RT-Summit – Call for Presentations” noting that the Real-Time Summit 2017 is being organized by the Linux Foundation Real-Time Linux (RTL) collaborative project in cooperation with OSADL/RTLWS and will be held also in Prague on October 21st. The cutoff for submissions is July 14th via email@example.com.
4.12 Merge Window
In his 4.11 announcement, Linus reminded us that the release of 4.11 meant that “the merge window [for kernel 4.12] is obviously open. I already have two pull request[s] for 4.12 in my inbox, I expect that overnight I’ll get a lot more.” He wasn’t disappointed. The flood gates well and truly opened. And they continued going for the entire two week (less one day) period. Let’s dive into what has been posted so far for 4.12 during the (now closed) merge window.
Stephen Rothwell [linux-next pre-merge development kernel tree maintainer] noted in a head’s up that Linus was going to see a “Large new drm driver” [drm – Direct Rendering Manager, not the “digital rights” technology]. Dave Airlie (the drm maintainer) had a reply but Stephen said everything was just fine and he was simply seeking to avoid surprising Linus (again). Once the pull came in, and Linus had pulled it, he quickly followed up to note that he was getting a lot of warnings about “Atomic update on pipe (A) took”. Daniel Vetter followed up to say that “We [Intel] did improve evasion a lot to the point that it didn’t show up in our CI machines anymore, so we felt we could risk enabling this everywhere. But of course it pops up all over the place as soon as drm-next hits mainline”.
4.12 git Pulls for existing subsystems
Hans-Christian Noren Egtvedt posted “AVR32 change for 4.12 – architecture removal” in which he removes AVR32 and “clean away the most obvious architecture related parts”. He posted followups to pick off more leftovers.
Ingo Molnar posted “RCU changes for 4.12” which includes “Parallelize SRCU callback handling”, performance improvements, documentation updates, and various other fixes. Linus pulled it. But then “after looking at it, ended up un-pulling it again”. He posted a rant about a new header file (linux/rcu_segcblist.h) which was a “header file from hell”, saying “I see absolutely no point in taking a heade file of several hundred lines of code”, along with more venting about the use of too much inline code (code that is always expanded in-place rather than called as a function – leading to a larger footprint sometimes). Finally, Linus said “The RCU code needs to start showing some good taste”. Sir Paul McKenney, the one and only author of RCU followed up swiftly, apologizing for the transgression in attempting to model “the various *list*.h header files”, proposing a fix, which Linus liked. Ingo Molnar implemented the suggestions, in “srcu: Debloat the <linux/rcu_segcblist.h> head”, which Paul provided a minor fix against for the case of !SMP (non-multi-processor kernel) builds.
Ingo Molnar also posted “EFI changes for 4.12” including fixes to the BGRT ACPI table (used for boottime graphics information) to allow it to be shared between x86 and ARM systems, an update to the arm64 boot protocol, improvements to the EFI stub’s command line parsing, and support for randomizing the virtual mapping of UEFI runtime services on arm64. The latter means that the function pointers for UEFI Runtime Services callbacks will be placed into random virtual address locations during the call to ExitBootServices which sets up the new mappings – it’s a good way to look for problems with platforms containing broken firmware that doesn’t correctly handle the change in location of runtime service calls.
Ingo Molnar also posted “x86/process changes for 4.12” which includes a new ARCH_[GET|SET]_CPUID prctl (process control) ABI extension that a running process can use in order to determine whether it has access to call the CPUID instruction directly. This is to support a userspace debugger known as “rr” that would like to trap and emulate calls to “CPUID” which are otherwise normally unprivileged on x86 systems.
Separately, Ingo posted “x86 fixes”, which includes “mostly misc fixes” for such things as “two boot crash fixes”, etc.
Ingo Molnar also posted “perf changes for 4.12” which includes updates to K and uprobes, making their trampolines (the codepaths jumped through when executing the probe sequence) read-only while they are used, changing UPROBES_EVENTS to be default yes in the Kconfig (since distros do this), and various other fixes. He also includes support for AMD IOMMU events, and new events for Intel Goldmont CPUs. The perf tooling itself gets many more updates, including PERF_RECORD_NAMESPACES, which allows the kernel to record information “required to associate samples to namespaces”.
Separately, Ingo posted “perf fixes”, which includes “mostly tooling updates”.
Ingo Molnar also posted “RAS changes for v4.12” which includes a “correct Errors Collector” kernel feature that will gather statistics aout correctable errors affecting physical memory pages. Once a certain watermark is reached, pages generating many correctable errors will be permanently offlined [this is useful both for DDR and NV-DIMMs]. Finally, he deprecates the existing /dev/mcelog driver and includes cleanups for MCE (Machine Check Exception) errors during kexec on x86 (which we covered in previous editions of this podcast).
Ingo Molnar also posted “x86/asm changes for v4.12”, which includes various fixes, among which are cleanups to stack trace unwinding.
Ingo Molanr also posted “x86/cpu changes for v4.12”, which includes support for “an extension of the Intel RDT code to extend it with Intel Memory Bandwidth Allocation CPU support: MBA allows bandwidth allocation between cores, while CBM (already upstream) allows CPU cache partitioning”. Effectively, Intel incorporate changes to their memory controller’s hardware scheduling algorithms as part of RDT. These allow the DDR interface to manage bandwidth for specific cores, which will almost certainly include both explict data operations, as well as separate algorithms for prefetching and speculative fetching of instructions and data. [This author has spent many hours reading about memory controller scheduling over the past year]
Ingo Molnar also posted “x86/debug changes for v4.12”, which includes support for the USB3 “debug port” based early console. As we have mentioned previously, USB3 includes a built-in “debug port” which no longer requires a special dongle to connect a remote machine for debug. It’s common in Windows kernel development to use a debug port, and since USB3 includes baseline support with the need for additional hardware, serial over USB3 is likely to become more common when developing for Linux – especially with the demise of DB9 headers on systems or even IDC10 headers on motherboards internally (to say nothing of laptop systems). As a reminder, with debug ports, usually only one USB port will support debug mode. I
guess my old USB debug port dongle can go in the pile of obsolete gear.
Ingo Molnar also posted “x86/platform changes for v4.12” which includes “continued SGI UV4 hardware-enablement changes, plus there’s also new Bluetooth support for the Intel Edison [a low cost IoT board] platform”.
Ingo Molnar also posted “x86/vdso changes for v4.12” which includes support for a “hyper-V TSC page” which is what it sounds like – a special shared page made available to guests under Microsoft’s Hyper-V hypervisor and providing a fast means to enumerate the current time. This is plumbed into the kernel’s vDSO mechanism (Virtual Dynamic Shared Objects look a bit like software libraries that are automatically linked against every running program when it launches) to allow fast clock reading.
Ingo Molnar also posted “x86/mm changes for v4.12”, which includes yet more work toward Intel 5-level paging among many other updates.
Separately Ingo posted a single “core kernel fix” to “increase stackprotector canary randomness on 64-bit kernels with very little cost”.
Thomas Gleixner posted “irq updates for 4.12”, which include a new driver for a MediaTek SoC, ACPI support for ITS (Interrupt Translation Services) when using a GICv3 on ARM systems, support for shared nested
interrupts, and “the usual pile of fixes and updates all over t[h]e place”.
Thomas Gleixner also posted “timer updates for 4.12” that include more reworking of year 2038 support (the infamous wrap of the Unix epoch), a “massive rework of the arm architected timer”, and various other work.
Separately, Ingo Molnar followed up with “timer fix” including “A single ARM Juno clocksource driver fix”.
Corey Minyard posted “4.12 for IPMI” including a watchdog fix. He “switched over to github at Stephen Rothwell’s [linux-next maintainer] request”.
Jonathan Corbet posted “Docs for 4.12” which includes “a new guide for user-space API documents” along with many other updates. Anil Nair noted “Missing File REPORTING-BUGS in Linux Kernel” which suggests that the Debian kernel package tools need to be taught about the recent changes in the kernel’s documentation layout. Separately, Jonathan replied to a thread entitled “Find more sane first words we have to say about Linux” noting that the kernel’s documentation files might not be the first place that someone completely new to Linux is going to go looking for information: “So I don’t doubt we could put something better there, but can we think for a moment about who the audience is here? If you’re “completely new to Linux”, will you really start by jumping into the kernel source tree?” The guy should do kernel standup in addition to LWN. It’d be hilarious.
Later, Jon posted “A few small documentation updates” which “Connect the newly RST-formatted documentation to the rest; this had to wait until the input pull was done. There’s also a few small fixes that wandered in”.
Tejun Heo posted “libata changes for 4.12-rc1” which includes “removal of SCT WRITE SAME support, which never worked properly”. SCT stands for “SMART [Self Monitoring And Reporting Technology – an error management mechanism common in contemporary disks] Command Transport”. The “write same” part means to set the drive content to a specific pattern (e.g. to zero it out) in cases that TRIM is not available. One wonders if that is also a feature used during destruction, though apparently the only (NSA) trusted way to destroy disks today is shredding and burning after zeroing.
Tejun Heo also posted “workqueue changes for v4.12-rc1”, which includes “One trivial patch to use setup_deferrable_timer() instead of open-coding the initialization”.
Tejun Heo also posted “cgroup changes for v4.12-rc1”, which includes a “second stab at fixing the long-standard race condition in the mount path and suppression of spurious warning from cgroup_get”.
Rafael J. Wysocki posted “Power management updates for v4.12-rc1, part 1” which includes many updates to the cpufreq subsystem and “to the intel_pstate driver in particular”. Its sysfs interface has apparently also been reworked to be more consistent with general expectations. He adds “Apart from that, the AnalyzeSuspend utility for system suspend profiling gets a companion called AnalyzeBoot for the analogous profiling of system boot and they both go into one place”.
Separately, he posted “Power management updates for v4.12-rc1, part 2”, which “add new CPU IDs [Intel Gemini Lake] to a couple of drivers [intel_idle and intel_rapl – Running Average Power Limit], fix a possible NULL pointer deference in the cpuidle core, update DT [DeviceTree]-related things in the generic power domains framwork and finally update the suspend/resume infrastructure to improve the handling of wakeups from suspend-to-idle”.
Rafael J. Wysocki also posted “ACPI updates for v4.12-rc1, part 1”, which includes a new Operation Region driver for the Intel CHT [Cherry Trail] Whiskey Cove PMIC [Power Management Integrated Circuit], and new sysfs entries for CPPC [Collaborative Processor Performance Control], which is a much more fine grained means for OS and firmware to coordinate on power management and CPU frequency/performance state transitions.
Separately, he posted “ACPI updates for v4.12-rc1, part 2”, which “update the ACPICA [ACPI – Advanced Configuration and Power Interface – Component Architecture, the cross-Operating System reference code]” to “add a few minor fixes and improvements”, and also “update ACPI SoC drivers with new device IDs, platform-related information and similar, fix the register information in xpower PMIC [Power Management IC] driver, introduce a concept of “always present” devices to the ACPI device enumeration code and use it to fix a problem with one platform [INT0002, Intel Cherry Trail], and fix a system resume issue related to power resources”.
Separately, Benjamin Tissories posted a patch reverting some ACPI laptop lid logic that had been introduced in Linux 4.10 but was breaking laptops from booting with the lid closed (a feature folks especially in QE use).
Rafael J. Wysocki also posted “Generic device properties framework updates for v4.12-rc1”, which includes various updates to the ACPI _DSD [Device Properties] method call to recognize “ports and endpoints”.
Shaohua Li posted “MD update for 4.12” which includes support for the “Partial Parity Log” feature present on the Intel IMSM RAID array, and a rewrite of the underlying MD bio (the basic storage IO concept used in Linux) handling. He notes “Now MD doesn’t directly access bio bvec, bi_phys_segments and uses modern bio API for bio split”.
Ulf Hansson posted “MMC for v[.]4.12” which includes many driver updates as well as refactoring of the code to “prepare for eMMC CMDQ and blkmq”. This is the planned transition to blkmq (block-multiqueue) for such storage devices. Previously it had stalled due to the performance hit when trying to use a multi-queue approach on legacy and contemporary non-mq devices.
Linus Walleij posted “pin control bulk changes for v4.12” in which he notes that “The extra week before the merge window actually resulted in some of the type of fixes that usually arrive after the merge window already starting to trickle in from eager developers using -next, I’m impressed”. He’s also impressed with the new “Samsung subsystem maintainer” (Krzysztof). Of the many changes, he says “The most pleasing to see is Julia Cartwright[‘]s work to audit the irqchip-providing drivers for realtime locking compliance. It’s one of those “I should really get around to looking into that” things that have been on my TODO list since forever”.
Linus Walliej also posted “Bulk GPIO changes for v4.12”, which has “Nothing really exciting goes on here this time, the most exciting for me is the same as for pin control: realtime is advancing thanks [t]o Julia Cartwright”.
Petr Mladek posted “printk for 4.12” which includes a fix for the “situation when early console is not deregistered because the preferred one matches a wrong entry. It caused messages to appear twice”.
Jiri Kosina posted “HID for 4.12” which includes various fixes, amongst them being an inversion of the HID_QUIRK_NO_INIT_REPORTS to the opposite due to the fact that it is appearently easier to whitelist working devices.
Jiri Kosina also posted “livepatching for 4.12” which includes a new “per-task consistency model” that is “being added for architectures that support reliable stack dumping”, which apparently “extends the nature of the types of patches than can be applied by live patching”.
Lee Jones posted “Backlight for 4.12” which includes various fixes.
Lee Jones also posted “MFD for v4.12” which includes some new drivers, new device support, and various new functionality and fixes.
Juergen Gross posted “xen: fixes and features for 4.12” which includes support for building the kernel with Xen enabled but without enabling paravirtualization, a new 9pfs xen frontend driver(!), and support for EFI “reset_sytem” (needed for ARMv8 Dom0 host to reboot), among various other fixes and cleanups.
Alex Williamson posted “VFIO updates for v4.12-rc1”.
Joerg Roedel posted “IOMMU Updates for Linux v4.12”, which includes “Some code optimizations for the Intel VT-d driver, code to “switch off a previously enabled Intel IOMMU” (presumably in order to place it into bypass mode for performance or other reasons?), “ACPI/IORT updates and fixes” (which enables full support for the ACPI IORT on 64-bit ARM).
Dmitry Torokhov posted “Input updates for v.4.11-rc0” which includes a documentation converstion to ReST (RST, the new kernel doc format), an update to the venerable Synaptics “PS/2” driver to be aware of companion “SMBus” devices and various other miscellaneous fixes.
Darren Hart posted “platform-drivers-x86 for 4.12-1” which includes “a significantly larger and more complex set of changes than those of prior merge windows”. These include “several changes with dependencies on other subsytems which we felt were best managed through merges of immutable branches”.
James Bottomley posted “first round of SCSI updates for the 4.11+ merge window”, which includes many driver updates, but also comes with a warning to Linus that “The major thing you should be aware of is that there’s a clash between a char dev change in the char-misc tree (adding the new cdev_device_add) and the make checking the return value of scsi_device_get() mandatory”. Linus and Greg would later clarify what cdev_device_add does in response to Greg’s request to pull “Char/Misc driver patches for 4.12-rc1”.
David Miller posted “Networking” which includes many fixes.
David also posted “Sparc”, which includes a “bug fix for handling exceptions during bzero on some sparc64 cpus”.
David also posted “IDE”, which includes “two small cleanups”.
Greg K-H (Kroah-Hartman) posted “USB driver patches for 4.12-rc1”, which includes “Lots of good stuff here, after many many many attempts, the kernel finally has a working typeC interface, many thanks to Heikki and Guenter and others who have taken the time to get this merged. It wasn’t an easy path for them at all.” It will be interesting to test that out!
Greg K-H also posted “Driver core patches for 4.12-rc1”, which is “very tiny” this time around and consists mostly of documentation fixes, etc.
Greg K-H also posted “Char/Misc driver patches for 4.12-rc1” which features “lots of new drivers” including Google firmware drivers, FPGA drivers, etc. This lead to a reaction from Linus about how the tree conflicted with James Bottomley’s tree (which he had already pulled, “as per James’ suggestion”, and a back and forth between James and Greg about how to better handle such a conflict next time, and Linus noting that he prefers to fix merge conflicts himself but “*also* really really prefer the two sides of the conflict having been more aware of the clash” and providing him with a head’s up in the pull.
Greg K-H also posted “Staging/IIO driver fixes for 4.12-rc1”, which adds “about 350k new lines of crap^Wcode, mostly all in a big dump of media drivers from Intel”. He notes that the Android low memory killer driver has finally been deleted “much to the celebration of the -mm developers”.
Greg K-H also posted “TTY patches for 4.12-rc1”, which wasn’t big.
Dan Williams posted “libnvdimm for 4.12” which includes “Region media error reporting [a generic interface more friendly to use with multiple namespaces]”, a new “struct dax_device” to allow drivers to have their own custom direct access operations, and various other updates. Dan also posted “libnvdimm: band aid btt vs clear posion locking”, a patch which “continues the 4.11 status quo of disabling of error clearing from the BTT [Block Translation Table] I/O path” and notes that “A solution for tracking and handling media errors natively in the BTT is needed”. The BTT or Block Translation Table is a mechanism used by NV-DIMMs to handle “torn sectors” (partially complete writes) in hardware during error or power failure. As the “btt.txt” in the kernel documentation notes, NV-DIMMs do not have the same atomicity guarantees as regular flash drives do. Flash drives have internal logic and store enough energy in capacitors to complete outstanding writes during a power failure (rotational drives have similar for flushing their memory based caches and syncing remap block state) but NV-DIMMs are designed differently. Thus the BTT provides a level of indirection that is used to provide for atomic sector semantics.
Separately, Dan posted “libnvdimm fixes for 4.12-rc1” which includes “incremental fixes and a small feature addition relative to the libnvdimm 4.12 pull request”. Gert had “noticed hat tinyconfig was bloated by BLOCK selecting DAX [Direct Acess Execution]”, while “Vishal adds a feature that missed the initial pull due to pending review feedback. It allows the kernel to clear media errors when initializing a BTT (atomic sector update driver) instance on a pmem namespace”.
Dave Airlie posted “drm tegra for 4.12-rc1” containing additional updates due because he missed a pull from Thierry Reding for NVidia Tegra patches. He also followed up with a “drm document code of conduct” patch that describes a code of conduct for graphics written by freedesktop.org.
Stafford Horne posted “Initramfs fix for 4.12-rc1” containing a fix “for an issue that has caused 4.11 to not boot on OpenRISC”.
Catalin later posted “arm64 2nd set of updates for 4.12”, which include “Silence module allocation failures when CONFIG_ARM*_MODULE_PLTS is enabled”.
Olof Johansson posted “ARM: SoC contents for 4.12 merge window”. In his pull request, Olof notes that “It’s been a relatively quiet release cycle here. Patch count is about the usual (818 commits, which includes merges).”
He goes on to add, “Besides dts [DeviceTree files], the mach-gemini cleanup by Linus Walleij is the only platform that pops up on its own”. He called out the separate post for the TEE [Trusted Execution Environment] subsystem. Olof also removed Alexandre Courbot and Stephen Warren from NVidia Tega maintainership, and added Jon Hunter in their place.
Rob Herring posted “DeviceTree for 4.12”, which includes updates to the Device Tree Compiler (dtc), and more DeviceTree overlay unit tests, among various other changes.
Darrick J. Wong posted “xfs: updates for 4.12”, which includes the “big new feature for this release” of a “new space mapping ioctl that we’ve been discussing since LSF2016 [Linux Storage and Filesystem conference]”.
Max Filippov posted “Xtensa improvements for 4.12”.
Ted Ts’o posted “ext4 updates for 4.12”, which adds “GETFSMAP support” (discussed previously in this podcast) among other new features.
Ted also posted “fscrypt updates for 4.12” which has “only bug fixes”.
Paul Moore posted “Audit patches for 4.12” which includes 14 patches that “span the full range of fixes, new featuresm and internal cleanups”. These include a move to 64-bit timestamps, converting refcounts to the new refcount_t type from atomic_t, and so on.
Wolfram Sang posted “i2c for 4.12”.
Mark Brown posted “regulator updates for 4.12”, which includes “Quite a lot going on with the regulator API for this release, much more in the core than in the drivers for a change”. This includes “Fixes for voltage change propagation through dumb power switches, a notification when regulators are enabled, a new settling time property for regulators where the time taken to move to a new voltage is not related to the size of the change”, etc.
Mark also posted “SPI updates for 4.12”, which includs “quite a lot of small
driver specific fixes and enhancements”.
Jessica Yu posted “module updates for 4.12”, containing minor fixes.
Mauro Carvalho Chehab posted “media updates” including mostly driver updates and the removal of “two staging LIRC drivers for obscure hardware”. He also posted a 5 part patch series entitled “Conver more books to ReST”, which converted three kernel DocBook format documentation file sets to RST, the new format being used for kernel documentation (on the kernel-doc mailing list, and maintained by Jonathan Corbet of LWN): librs, mtdnand, and sh. He noted that “After this series, there will be just one DocBook pending conversion: ” lsm (Linux Security Modules)”. He also notes that the existing LSM documentation is very out of date and no longer describes the current API.
Michael Ellerman posted “Please pull powerpc/linux.git powerpc-4.12-1 tag”, which includes suppot for “Larger virtual address space on 64-bit server CPUs. By default we use a 128TB virtual address space, but a process can request access to the full 512TB by passing a hint to mmap() [this seems very similar to the 56-bit la57 feature from Intel]”. It also includes “TLB flushing optimisations for the radix MMU on Power9” and “Support for CAPI cards on Power9, using the “Coherent Accelerator Interface Architecture 2.0″ [which definitely sounds like juicy reading]”.
Separately Michael Ellerman posted “Please pull powerpc-linux.git powerpc-4.12-2 tag” which includes “rework the Linux page table geometry to lower memory usage on 64-bit Book3S (IBM chips) using the Hash MMU [IBM uses a special inverse page tables “reverse lookup” hashing format]”.
Eric W. Biederman posted “namespace related changes for v4.12-rc1”, which includes a “set of small fixes that were mostly stumbled over during more significant development. This proc fix and the fix to posix-timers are the most significant of the lot. There is a lot of good development going on but unfortunately it didn’t quite make the merge window”.
Takashi Iwai posted “sound updates for 4.12-rc1”, noting that it was “a relatively calm development cycle, no scaring changes are seen”.
Steven Rostedt posted “tracing: Updates for v4.12” which includes “Pretty much a full rewrite of the process of function probes”. He followed up with “Three more updates for 4.12” that contained “three simple changes”.
Martin Schwidefsky posted “s390 patches for 4.12 merge window” which includes improvements to VFIO support on mainframe(!) [this author was recently amazed to see there are also DPDK ports for s390x], a new true random number generator, perf counters for the new z13 CPU, and many others besides.
Geert Uytterhoeven posted “m68k updates for 4.12” with a couple fixes.
Jacek Anaszewski posted “LED updates for 4.12” with various fixes.
Kees Cook posted “usercopy updates for v4.12-rc1” with a couple fixes.
Kees also posted “pstore updates for v4.12-rc1”, which included “large
internal refactoring along with several smaller fixes”.
James Morris posted “Security subsystem updates for v4.12”.
Sebastian Reichel posted “hsi changes for hsi-4.12”.
Sebastian also posted “power-supply changes for 4.12”, which includes a couple of new drivers and various fixes.
Separately, Sebastian poted “power-supply changes for 4.12 (part 2), which includes some new drivers and some fixes.
Paolo Bonzini posted “First batch of KVM changes for 4.12 merge window” which includes kexec/kdump support on 32-bit ARM, support for a userspace virtual interrupt controller to handle the “weird” Raspberry Pi 3, in-kernel acceleration for VFIO on POWER, nested EPT support for accessed and dirty bits on x86, and many other fixes and improvements besides.
Separately Paolo posted “Second round of KVM changes for 4.12”, which include various ARM (32 and 64-bit) cleanups, support for PPC [POWER] XIVE (eXternal Interrupt Virtualization Engine), and “x86: nVMX improvements, including emulated page modification logging (PML) which brings nice performance improvements [under nested virtualization] on some workloads”.
Ilya Dryomov posted “Ceph updates for 4.12-rc1”, which include “support for disabling automatic rbd [resilent block device] exclusive lock transfers” and “the long awaited -ENOSPC [no space] handling series”. The latter finally handles out of space situations by aborting with -ENOSPC rather than “having them [writers] block indefinitely”.
Miklos Szeredi posted “fuse updates for 4.12”, which “contains support for pid namespaces from Seth and refcount_t work from Elena”.
Miklos also posted “overlayfs update for 4.12”, which includes “making st_dev/st_ino on the overlay behave like a normal filesystem”. “Currently this only wokrs if all layers are on the same filesystem, but future work will move the general case towards more sane behavior”.
Bjorn Helgaas posted “PCI changes for v4.12” which includes a framework for supporting PCIe devices in Endpoint mode from Kishon Vjiay Abraham, fixes for using non-posted PCI config space on ARM from Lorenzo Pieralisi, allowing slots below PCI-to-PCIe “reverse bridges”, a bunch of quirks, and many other fixes and enhancements.
Jaegeuk Kim posted “f2fs for 4.12-rc1”, which “focused on enhancing performance with regards to block allocation, GC [Garbage Collection], and discard/in-place-update IO controls”.
Shuah Khan posted “Kselftest update for 4.12-rc1” with a few fixes.
Richard Weinberg posted “UML changes for v4.12-rc1” which includes “No new stuff, just fixes” to the “User Mode Linux” architecture in 4.12. Separately, Masami Hiramatsu posted an RFC patch entitled “Output messages to stderr and support quiet option” intended to “fix some boot time printf output to stderr by adding os_info() and os_warn(). The information-level messages via os_info() are suppressed when “quiet” kernel option is specified”.
Richard also postd “UBI/UBIFS updates for 4.12-rc1”, which “contains updates for both UBI and UBIFS”. It has a new CONFIG_UBIFS_FS_SECURITY option, among “minor improvements” and “random fixes”.
Thierry Reding posted “pwm: Changes for v4.12-rc1”, which amongst other things includes “a new driver for the PWM controller found on MediaTek SoCs”.
Vinod Koul posted “dmaengine updates” which includes “a smaller update consisting of support for TI DA8xx dma controller” among others.
Chris Mason posted “Btrfs” which “Has fixes and cleanups” as well as “The biggest functional fixes [being] between btrfs raid5/6 and scrub”.
Trond Myklebust posted “Please pull NFS client fixes for 4.12”, which includes various fixes, and new features (such as “Remove the v3-only data server limitation on pNFS/flexfiles”).
J. Bruce Fields posted “nfsd changes for 4.12”, which includes various RDMA updates from Chuck Lever.
Stephen Boyd posted “clk changes for v4.12”. Of the changes, the “biggest things are the TI clk driver rework to lay the groundwork for clkctrl support in the next merge window and the AmLogic audio/graphics clk support”.
Alexandre Belloni posted “RTC [Real Time Clock] for 4.12”, which uses a new GPG subkey that he also let Linus know about at the same time.
Nicholas A. Bellinger posted “target updates for v4.12-rc1”, which was “a lot more calm than previously expected. It’s primarily fixes in various areas, with most of the new functionality centering around TCMU [TCM – Linux iSCSI Target Support in Userspace] backend work with Xiubo Li has been driving”.
Zhang Rui posted “Thermal management updates for v4.12-rc1”, which includes a number of fixes, as well as some new drivers, and a new interface in “thermal devfreq_cooling code so that the driver can provide more precise data regarding actual power to the thermal governor every time the power budget is calculated”.
4.12 git pulls for new subsystems and features
David Howells posted “Hardware module parameter annotation for secure boot” in which he requested that Linus pull in new “kmod” macros (the same name is used for the userspace module tooling, but in this case refers to the in-kernel kernel module infrastructure of the same name). The new macros add annotations to “module_param” of the new form “module_param_hw” with a “hwtype” such as “ioport” or “iomem”, and so forth. These are used by the kernel to prevent those parameters from being used under a UEFI Secure Boot situation in which the kernel is “locked down” (to prevent someone from loading a signed kernel image and then compromising it to circumvent the secure boot mechanism).
Arnd Bergmann sent a special pull request to Linus Torvalds for “TEE driver infrastructure and OP-TEE drivers”, which “introduces a generic TEE [Trusted Execution Environment] framework in the kernel, to handle trusted environ[ments] (security coprocessor or software implementations such as OP-TEE/TrustZone)”. He sent the pull separately from the other arm-soc pull specifically to call it out, and to make sure everyone knew that this was finally headed upstream, but he noted it would probably be maintained through the arm-soc kernel tree. He included a lengthy defense of why now was the right time to merge TEE support into upstream Linux.
Saving TLB flushes on Intel x86 Architecture
Andy Lutomirski posted an RFC patch series entitled “x86 TLB flush cleanups, moving toward PCID support”. Modern (non-legacy) architectures implement a per-process context identifier that can be used in order to tag VMA (Virtual Memory Area) translations that end up in the TLB (Translation Lookaside Buffer) caches within the microprocessor core. The processor’s hardware (or in some mostly embedded cases, software) (page table) “walkers” will navigate the page tables for a process and populate the TLBs (except in the embedded software case, such as on certain PowerPC and MIPS processors, in which the kernel contains special assembly routines to perform this in software). On legacy architectures, the TLB is fairly simple, containing a simple virtual address to physical (or intermediate, in the case of virtualization) address. But on more sophisticated architectures, the TLB includes address space identification information that allows the TLB to distinguish between hits to the same virtual address that are from two different processes (known as tasks from within the kernel). Using additional tagging in the TLB avoids the traditional need to invalidate the entire TLB on process context switch.
Modern architectures, such as AArch64, have implemented context tagging support in their architecture code for some time, and now x86 is finally set to follow, enabling a feature that has actually been present in x86 for some time (but was not wired up), thanks to Andy’s work on PCID (Process Context IDentifier) support. In his patch series, Andy notes that as he has been “polishing [his] PCID code, a major problem [he’s] encountered is that there are too many x86 TLB flushing code paths and that they have too many inconsequential differences”. This patch series aims to “clean up the mess”. Now if x86 finally gains hardware broadcast TLB invalidations it will also be able to remove the wasted IPIs (Inter-Processor-Interrupts) that it implements to cause remote processors to invalidate TLB entries, too. Linus liked Andy’s initial work, but said he is “always a bit nervous about TLB changes like this just because any potential bugs tend to be really really hard to see and catch”. Those of us who have debugged nasty TLB issues on other architectures would be inclined to agree with him.
Laurent Dufour posted version 3 of a patch series entitled “Speculative page faults”. This is a contemporary development inspired by Peter Zijstra’s earlier work, which was based upon ideas of still others. The whole concept dates back to at least 2009 and generally involves removing the traditional locking constraints of updates to VMAs (Virtual Memory Areas) used by Linux tasks (processes) to represent the memory of running programs. Essentially, a “speculative fault” means “not holding mmap_sem” (a semaphore guarding a tasks’ current memory map). Laurent (and Peter) make VMA lookups lockless, and perform updats speculatively, using a seqlock to detect a change to the underlying VMA during the fault. “Once we’ve obtained the page and are ready to update the PTE, we validate if the state we started the fault with is still valid, if not, we’ll fail the fault with VM_FAULT_RETRY, otherwise we update the PTE and we’re done”. Earlier testing showed very significant performance upside to this work due to the reduced lock contention.
Aaron Lu posted “smp: do not send IPI if call_single_queue not empty”. The Linux kernel (and most others) uses a construct known as an IPI – or Inter-Processor-Interrupt – a form of software generated interrupt that a processor will send to one or more others when it needs them to perform some housekeeping work on the kernel’s behalf. Usually, this is to handle such things as a TLB shootdown (invalidating a virtual address translation in a remote processor due to a virtual address space being removed), especially on less sophisticated legacy architectures that do not feature invalidation of TLBs through hardware broadcast, though there are many other uses for IPIs. Aaron’s patch realizes, effectively, that if a remote processor is already going to process a queue of CSD (call_single_data) function calls it has been asked to via IPI then there is no need to send another IPI and generate additional interrupts – the queue will be drained of this entry as well as existing entries by the IPI management code.
Romain Perier posted version 8 of “Replace PCI pool by DMA pool API” which realizes that the current PCI pool API uses “simple macro functions direct expanded to the appropriate dma pool functions”, so it simply replaces them with a direct use of the corresponding DMA pool API instead.
Sandhya Bankar posted “vfs: Convert file allocation code to use the IDR”. This replaces existing filesystem code that allocates file descriptors using a
custom allocator with Matthew (Willy) Wilcox’s idr (ID Radix) tree allocator.
Serge E. Hallyn posted a resend of version 2 of a patch series entitled “Introduce v3 namespaced file capabilities”. We covered this last time.
Heinrich Schuchardt posted “arm64: Always provide “model name” in /proc/cpuinfo”, which was quickly shot down (for the moment).
Christian König posted verision 5 of his “Resizeable PCI BAR support” patch series. We have featured this in a previous episode of the podcast.
Prakash Sangappa posted “hugetlbfs ‘noautofill’ mount option” which aims to allow (optionally) for hugetlbfs pseudo-filesystems to be mounted with an option which will not automatically populate holes in files with zeros during a page fault when the file is accessed though the mapped address. This is intended to benefit applications such as Oracle databases, which make heavy use of such mechanisms but don’t take kindly to the kernel having side effects that change on-disk files even if only zero fill. Dave Hansen pushed back against this change saying that it was “further specializing hugetlbfs” and that Oracle should be using userfaultfd or “an madvise() option that disallows backing allocations”. Prakash replied that they had considered those but with a database there are such a large number of single threaded processes that “The concern with using userfaultfs is the overhead of setup and having an additional thread per process”.
Sameer Goel posted “arm64: Add translation functions for /dev/mem read/write” which “Port architecture specific xlate [translate] and unxlate [untranslate] functions for /dev/mem read/write. This sets up the mapping for a valid physical address if a kernel direct mapping is not alread present”. Depending upon the ARM platform, access to a bad address in /dev/mem could result in a synchronous exception in the core, or a System Error (SError) generated by a system memory controller interface. In either case, it is handled as a fatal error where the same is not true on x86. While access to /dev/mem is restricted, increasingly being deprecated, and has other semantics to prevent its used on 64-bit ARM systems, it still exists and is used. In this case, to read the ACPI FPDT table which provides performance pointer records. Nevertheless, both Will Deacon and Leif Lindholm objected to the reasoning given here, saying that the kernel should instead be taught how to parse this table and expose its information via /sys rather than having userspace tools go poking in /dev/mem to try to read from the table directly.
Minchan Kim posted “vmscan: scan pages until it f[inds] eligible pages” in which he notes that “There are premature OOM [Out Of Memory killer invocations] happening. Although there are ton of free swap and anonymous LRU list of eligible zones, OOM happened. With investigation, skipping page of isolate_lru_pages makes reclaim void because it returns zero nr_taken easily so LRU shrinking is effectively nothing and just increases priority aggressively. Finally, OOM happens”.
Julius Werner posted version 3 of his “Memconsole changes for new coreboot format” which teaches the Google firmware driver for their memconsole to deal with the newer type of persistent ring buffer console they introduced.
Olliver Schinagl and Jamie Iles had a back and forth about the latter’s work on “glue-code” (generic handling code) for the DW (DesignWare) 8250 (a type of serial port interface made popular by PC) IP block as used in many different designs. Depending upon how the block is configured, it can behave differently, and there was some discussion about how to handle that. In particular the location of the UART_USR register.
Xiao Guangrong posted “KVM: MMU: fast write protect” which “introduces a[n] extremely fast way to write protec all the guest memory. Comparing with the ordinary algorthim which write protects last level sptes [the page table entries used by the guest] based on the rmap [the “reverse” map, the means that Linux uses to encode page table information within the kernel] one by one, it just simply updates the generation number to ask all vCPUs to reload its root page table, particularly it can be done out of mmu-lock”. The idea was apparently originally proposed by Avi (Kivity). Paolo Bonzini thought “This is clever” and wondered “how the alternative write protection mechanism would affect performance of the dirty page ring buffer patches”. Xiao thought it could be used to speed up those patches after merging, too [Paolo noted that he aims to merge these early in 4.13 development].
Bogdan Mirea posted version 2 of”Add “Preserve Boot Time Support””, which follows up on a previous discussion about retaining “Boot Time Preservation between Bootloader and Linux Kernel. It is based on the idea that the Bootloader (or any other early firmware) will start the HW Timer and Linux Kernel will count the time starting with the cycles elapsed since timer start”. By “Bootloader” he means “firmware” to those who live in x86-land.
Igor Stoppa posted “post-init-read-only protection for data allocated dynamically” which aims to provide a mechanism for dynamically allocated data which is similar to the “__read_only” special linker section that certain annotated (using special GCC directives) code will be placed into. That works great for read-only data (which is protected by the MMU locking down the corresponding region early in boot). His “wish” is to start with the “policy DB of SE Linux and the LSM Hooks, but eventually I would like to extend the protection also to other subsystems, in a way that can merged into mainline.” His patch includes an analysis of how he feels he can be as “little invasive as possible”, noting that “In most, if not all, the cases that could be enhanced, the code will be calling kmalloc/vmalloc, including GFP_KERNEL [Get Free Pages of Kernel Type Memory] as the desired type of memory”. Consequently, he says, “I suspect/hope that the various maintainer[s] won’t object too much if my changes are limited to replacing GFP_KERNEL with some other macro, for example what I previously called GFP_LOCKABLE”. Michal Hocko had some feedback, largely along the lines of a “master toggle” (tha would allow protection to be disabled for small periods in order to make changes to “read only” data) was largely pointless – due to it re-exposing the data. Instead, he wanted to see the protection being done at the kmem_cache_create time by adding a “SLAB_SEAL” parameter that would later be enabled on a per kmem_cache basis using “kmem_cache_seal(cache)” or a similar mechanism.
Bharat Bhushan posted “ARM64/PCI: Allow userspace to mmap PCI resources”, which Lorenzo Pieralisi noted was already implemented by another patch.
A lengthy, and “spirited” discussion took place between Timur Tabi and the various maintainers of the 64-bit ARM Architecture and SoC platform trees over the desire for the maintainers to have changes to “defconfigs” for the architecture go through a special “firstname.lastname@example.org” alias. Except that after they had told Timur to use that, they objected to him posting a patch informing others of this alias in the kernel documentation. Instead, as Timur put it “without a MAINTAINERS entry, how would anyone know to CC: that address? I posted 3 versions of my defconfig patchset before someone told me that I had to send it to email@example.com.” The discussion thread is entitled “MAINTAINERS: add firstname.lastname@example.org as the list for arm64 defconfig changes”.
Xunlei Pang posted version 3 of his “x86/mm/ident_map: Add PUD level 1GB page support” which helps “kernel_ident_mapping_init” to create a single and very large identitiy page mapping in order to reduce TLB (Translation Lookaside Buffer – the caches that store virtual to physical memory lookups performed by hardware) pressure on an architecture that is currently using many 2MB (PMD – Page Middle Directory) level pages for this process.
Anju T Sudhakar posted version 8 of “IMC Instrumentation Support”, which provides support for POWER9’s “In-Memory-Collection” or IMC infrastructure, which “contains various Performance Monitoring Units (PMUs) at Nest level (these are on-chip but off-core), Core level and Thread level.”
Greg K-H (Kroah-Hartman) posted an RFC patch entitled “add more new kernel pointer filter options” which “implemnt[s] some new restrictions when printing out kernel pointers, as well as the ability to whitelist kernel pointers where needed.”
Kees Cook posted “x86/refcount: Implement fast refcount overflow protection”, which seeks to upstream a “modified version of the x86 PAX_REFCOUNT defense from PaX/grsecurity. This speeds up the refcount_t API by duplicating the existing atomic_t implementation with a single instruction added to detect if the refcount has wrapped past INT_MAC (or below 0) resuling in a negative value, where the handler then restores the refcount_t to INT_MAX”.
David Howlls posted an RFC patch entitled “VFS: Introduce superblock configuration context” which is a “set of patches to create a superblock configuration contenxt prior to setting up a new mount, populating it with the parsed options/binary data, creating the superblock and then effecting the mount. This allows namespaces and other information to be conveyed through the mount procedure. It also allows extra error information”.
The Google Chromebook team let folks know that they were (rarely, like one in a million) seeing “Threads stuck in zap_pid_ns_processes()”. Guenter Roeck noted that the “Problem is that if the main task [which has children that are being ptraced] doesn’t exit, it [the child] hangs forever. Chrome OS (where we see the problem in the field, and the application is chrome) is configured to reboot on hung tasks – if a task is hung for 120 seconds on those systems, it tends to be in a bad shape. This makes it a quite severe problem for us”. He asked “Are there other conditions besides ptrace where a task isn’t reaped?”. Reaping refers to the behavior in which tasks, when they exit are reparented to the init task, which “reaps” them (cleans up and makes sure the state that exit with is seen), except under ptrace in this case where the parent task spawning the children “was outside of the pid namespace and was choosing not to reap the child”. Various proposals as to how to deal with this in the namespace code were discussed.
Mahesh Bandewar posted “kmod: don’t load module unless req process has CAP_SYS_MODULE” which notes that “A process inside random user-ns [a user namespace] should not load a module, which is currently possible”. He shows how a user namespace can be created that causes the kernel to load a module upon access to a file node indirectly. This could be a security risk if this approach were used to cause a host kernel to load a vulnerable but otherwise not loaded kernel driver through the privileged permissions in the namespace.
Marc Zyngier posted “irqdomain: Improve irq_domain_mapping facility” in which he “Update[s] IRQ-domain.txt to document irq_domain_mapping” among otherwise seeking to make it easier to access and understand this kernel feature.
Jens Axboe accepted a patch from Ulf Hansson adding Paolo Valente as a MAINTAINER of the BFQ I/O scheduler.
Cyrille Pitchen updated the git repos for the SPI NOR subsystem, which is “now hosted on MTD repos, spi-nor/next is on l2-mtd and spi-nor/fixes will be on linux-mtd”.
Alexandre Courbot posted “MAINTAINERS: remove self from GPIO maintainers”.
The folks at Codeaurora posted a lengthy analysis of the Linux kernel scheduler and specific problems with load_balance that will be covered next time around, along with work by Peter Zijlstra on the “cgroup/PELT overhaul (again).
Finally, Paul McKenney previously posted “Make SRCU be once again optional”, after having noted that the need to build it in by default (caused by other recent changes in header files) increased the kernel by 2K. Nico(las) Pitre was happy to hear this, saying “If every maintainer finds a way to (optionally) reduce the size of the code they maintain by 2K then we’ll get a much smaller kernel pretty soon”.