The Embedded Frontier: From i386 PCs to Jailed Multi-Core Routers
How we scaled a wireless broadband empire in Recife and evolved into High-Availability Managed Security using FreeBSD Jails.
The Beginning: Breaking Free from Proprietary Systems (2000-2006)
In the early days, our first Access Points (APs) were proprietary closed systems. By 2006, we had transitioned to StarOS for our AP-bridges, but we were already planning our move toward something more powerful and open.
We started deploying FreeBSD-based APs for our primary building infrastructure. For the main nodes—where we had multiple directional antennas for peer-to-peer links and narrow-beam sectors—we divided the workload, utilizing Mikrotik motherboards for the radio distribution while FreeBSD handled the core routing and logic.
Hardening the i386 Kernel
In the third quarter of 2000, we began automating our installation scripts to transform standard PCs into robust routers. We customized our FreeBSD i386 kernels specifically for the task:
Pruning the Architecture: We removed support for
I386_CPUandI486_CPUto ensure the kernel was optimized and stable for our specific hardware.Security at the Edge: We integrated
IPFIREWALL(IPFW) directly into the kernel to protect our clients and prevent unwanted traffic from congesting our incipient outdoor network.
2008: The ALIX Revolution and Read-Only Flash
At the end of 2008, we adopted the PC Engines ALIX motherboard. This was our first true SOC (System on a Chip) for the NLINK project. To protect the Compact Flash cards from failing, we mastered the Read-Only environment:
The /usr/local/flash Logic: We stored permanent configs in
/usr/local/flash. At boot, these were recovered to a memory filesystem (MFS) in/var, ensuring the system could write logs to RAM without wearing out the physical Flash memory.
# Protecting our Compact Flash via RAM disks
diskless_mount=/etc/rc.diskless2
varsize="32m"
# Permanent files recovered from /usr/local/flash to /var at bootScaling Up: The APU and Managed Firewalls
As our network grew, we transitioned from the ALIX to the APU series. With its multi-core architecture and gigabit ports, the APU became the backbone of the NLINK ISP main network. This allowed us to expand our business by selling managed firewalls to our corporate clients—hardened, enterprise-grade security boxes managed directly by our team.
The Modern Edge: Jails and VRRP
Today, the evolution continues. On our modern multi-core routers, we utilize FreeBSD Jails to implement virtual routing. By combining Jails with VRRP (Virtual Router Redundancy Protocol), we create highly available, isolated routing instances on a single physical device.
This “Jailed” approach allows us to manage complex virtual route table (VRT) environments with total isolation. If one routing instance requires a change or a restart, the others remain untouched, ensuring that the 225+ buildings we serve in Recife never lose their connection.
Reflection: Parallel Paths with Meraki
Looking back now, it’s incredible to realize how close our paths were without us knowing it. In 2005–2006, while we were deep into deploying StarOS platforms for mobile packet core gateways in Brazil, a small team in Mountain View was quietly building the very first cloud-managed WiFi access points on FreeBSD with Atheros cards—the founding spark of what would become Meraki.
Two completely separate projects, both pushing the boundaries of networking in their own corners of the industry, running in parallel at exactly the same time. I had no idea back then—it is one of those remarkable coincidences you only notice years later. It reinforces my belief: when you need the most resilient networking in the world, all roads eventually lead to FreeBSD