From Electronics Engineer to FreeBSD 15.1: An MSR Offset Bug on AMD Zen 3
hwpmc. Thirty-two years on the same operating system.The Discovery: A Bug Hidden in Plain Sight
As an electronics engineer, I learned early to trust the data sheet before trusting the code. Hardware doesn’t lie. Documentation can be wrong, but in this story the documentation was the source of truth, and the kernel was the one drifting away from it.
Earlier this year I was profiling router performance for PRTR, my FreeBSD-based routing distribution, on an AMD Ryzen 5 5600X (a Family 19h, Zen 3 processor). I reached for the hardware performance counters (hwpmc), and the pmccontrol -l output revealed only 6 PMCs per CPU, exposing just the legacy K8 Core counters. The L3 cache and Data Fabric counters that newer AMD processors expose were silently disabled. Sixteen counters per CPU were sitting there in hardware, unused.
I pulled up the AMD BKDG #24594, page 440, the canonical reference table for RDPMC ECX values across counter classes. Reading that table line by line against sys/dev/hwpmc/hwpmc_amd.c::amd_get_msr(), I noticed the math didn’t agree with the hardware.
RDPMC encoding on Family 17h and later uses three separate base values:
One base for Core PMCs
One for L3 cache counters
One for Data Fabric counters
The FreeBSD kernel was assuming a linear MSR offset across all three classes. The calculation worked for Core (because the base happened to match), but it produced garbage indices for L3 and Data Fabric. The processor was rejecting those reads, and the kernel was concluding the counters didn’t work.
AMD’s documentation was correct. The FreeBSD code wasn’t following it.
The 11-Day Arc: Three Patches, Three Reviewers
What started as a debugging session on a Thursday morning became an eleven-day apprenticeship in kernel development. The pace surprised me. I had assumed FreeBSD contributions moved like geological time. Phabricator and the review team had different ideas.
March 12: The MSR offset fix lands (D55607)
The fix went through seven diff revisions over eleven days, responding to feedback from Ali Mashtizadeh (Netflix) and Mitchell Horne, before landing in main as commit ce9aff829e02. The patch refactored amd_get_msr() to use class-specific base offsets, matching the AMD documentation. After the fix, the L3 cache and Data Fabric counter classes (ten previously inaccessible counters per CPU) became available to performance tooling. On my Ryzen, pmccontrol -l now reports the full set of 35 PMCs per CPU.
March 23: Better diagnostics for tunables (D56029)
While testing D55607, I hit the kern.hwpmc.nbuffers_pcpu tunable limit and noticed two problems. First, the validation warning used printf(), which only appears on the early console, invisible to anyone checking dmesg or /var/log/messages after boot. Second, the message was confusing: it said the value “must be greater than zero” when I had actually set 256, exceeding the 32 MB per-CPU buffer ceiling. The patch replaced printf() with log(LOG_WARNING, ...) in pmclog_initialize() so failures are properly logged, and rewrote the message to report both the rejected value and the default the kernel fell back to. It landed as commit b3a18736ec2f.
March 23: Manual page corrections (D56050)
Engineering isn’t done until documentation matches reality. Auditing the hwpmc(4) manual page, I found two default values that had been wrong since 2018. Commit 2981a3420cb1 had updated the compiled defaults but never touched the man page, so for eight years the documentation claimed logbuffersize defaulted to 4KB (actually 256KB) and nbuffers_pcpu defaulted to 64 (actually 32). My patch corrected both values, documented the maximums, added the 32 MB per-CPU product limit, and updated the DIAGNOSTICS section to reflect the new messages from D56029. It landed as commit 2318ea10a3af. Fixing an eight-year-old documentation bug felt oddly satisfying.
Three commits in eleven days: the offset fix on March 12, then the diagnostics improvement and the man page correction both landing on March 23. From zero commits to three in less than two weeks. That kind of momentum is what makes you want to keep contributing.
The Reviewers: Kernel Engineering as a Craft
The actual learning didn’t happen while writing the patches. It happened in the Phabricator comment threads.
Ali Mashtizadeh (Netflix) caught architectural issues I hadn’t considered. His feedback on D55607 reshaped how I think about counter class abstraction. He didn’t accept code that merely worked; he asked whether the code was structured the way it should be.
Mitchell Horne focused on style consistency with the rest of the FreeBSD tree. His comments taught me that kernel code isn’t just about correctness. It’s about being readable by the next person who has to maintain it.
Warner Losh (imp) committed all three patches and provided the final sign-off. His role as committer meant he had to vouch for the work to the rest of the project.
The standout lesson for me came on D55607. I had defined the MSR base values as local constants inside amd_get_msr(). The review asked me to promote them to module-level statics, where they would be visible to other functions and serve as documentation in their own right. That single comment reshaped how I think about scope. A kernel developer doesn’t just fix the immediate problem. They write code that future contributors can reason about without re-deriving the AMD BKDG.
The Release: June 16, 2026
Earlier this month, on June 16, 2026, the FreeBSD Project shipped FreeBSD 15.1-RELEASE.
Two of my three patches made it into the release:
290a7adfb1df: improved diagnostic messages (MFC of D56029)928091964581: hwpmc.4 man page corrections (MFC of D56050)
You can verify this on any FreeBSD 15.1 system:
git log release/15.1.0 --author="Paulo Fragoso"The third patch, D55607, changes core counter behavior, so it appropriately baked longer in CURRENT before being merged to a stable branch. Update, June 28: while I was finishing this post, Mitchell Horne merged it to stable/15 as commit 067755c74a3d. All three patches are now in the stable branch, and the MSR offset fix will ship in FreeBSD 15.2.
But two patches in 15.1 means that today, somewhere, a sysadmin running pmccontrol or reading man hwpmc is seeing words I wrote. Code I shipped from a workstation in Recife is running on production servers around the world.
That’s a strange and humbling feeling.
The Roots: Thirty Years With FreeBSD
People sometimes ask why an electronics engineer would obsess over operating system internals. For me, the answer goes back to 1994.
I started as an undergraduate researcher at the LFTC laboratory in the Physics Department (DF) of the Federal University of Pernambuco (UFPE), where we ran SunOS 4.1.3 on the SPARC workstations. That was my first deep exposure to a real Unix system. Around that time I was exchanging emails over the early internet with Pedro Vasquez, in the Chemistry Department at Unicamp, who was active in the nascent Brazilian FreeBSD community. He advised me to try FreeBSD, since both SunOS 4.x and FreeBSD descend from 4.3BSD; it would feel like home, but running on commodity hardware I could actually afford.
That advice shaped the next thirty years. In 1995 I started NLINK out of my home garage in Recife, and FreeBSD was the foundation from day one. It gave me the same Unix discipline I had learned on SunOS, on hardware I could buy myself.
That decision held for three decades. FreeBSD became the foundation for everything:
The wireless broadband network NLINK built across Recife, starting on i386 PCs and PC Engines ALIX boards
My development workstation and my daily driver at work
The managed firewalls we sold to corporate clients
The BGP routing that carries our autonomous system, AS28585, across multiple Brazilian Internet Exchanges
For years I ran BSDRP (the BSD Router Project by Olivier Cochard-Labbé) in production at both NLINK and GMNET Telecomunicações. BSDRP taught me that a FreeBSD-based router could match carrier-grade equipment on commodity hardware. Eventually I forked it into my own distribution, PRTR, a router platform built around BIRD 3.x, carrying full DFZ tables of 3M+ routes, with a post-quantum cryptography stack for the day quantum computers threaten classical BGP session security.
And here’s the thing about this kernel work: these three hwpmc patches came directly out of PRTR development. I wasn’t looking for kernel bugs in the abstract. I was profiling router performance on AMD hardware, reached for hwpmc, and found the counters didn’t work the way the AMD documentation said they should. The patches are listed in the PRTR README under “Upstream Contributions” alongside the BSDRP pull requests, because that’s where they came from: real work on a real production system. Every one of them carries the line Sponsored by: NLINK in its commit message. This wasn’t a corporate contribution from a big vendor; it was an independent Brazilian ISP giving back to the operating system it has run for thirty years.
Finding the bug in amd_get_msr() wasn’t an accident. It was the inevitable result of thirty years of running FreeBSD in production (a path that started with an email tip from Pedro Vasquez in 1994) and finally needing to look one layer deeper than I ever had before.
If you’re an engineer who’s been running open-source infrastructure for years without contributing back, you already have what it takes. The bugs you hit in production are real bugs. When the code disagrees with the hardware documentation, that’s not a sign you’re using it wrong. It’s a sign you’ve found something worth fixing. Read the manual, write the patch, and send it. FreeBSD has space for you.
Verification Links
Phabricator reviews:
D56029: hwpmc: improve diagnostic messages for invalid tunables
D56050: hwpmc.4: correct stale default values and update diagnostics
Commits in main:
Commits in stable/15 (shipped in FreeBSD 15.1):
290a7adfb1df: improved diagnostic messages928091964581: hwpmc.4 corrections
Hardware test harness:
The router project these patches came from:
github.com/jailedengineer/PRTR, PQC-Ready Telecommunications Router