The magic of eBPF I: What is this?

 Introduction

    This post has been mainly inspired because I've been tinkering with eBPF for the past few months, getting to know it works. Now that I have what could be consider "Solid" knowledge on the matter, I thought to share and tell my experience and the possibilities I foresee with this technology.

    If I say that the technology world, especially IT, is in constant evolution and change I surprise no one however, it's been a long time since we get something so potentially game-changing as eBPF is. Originally, it was designed to filter network packets, but now has grown into an incredibly versatile and powerful tool that enables developers and security engineers to run sandboxed programs in the Kernel space.

    I'm fully aware that the moment you read Kernel a shiver was sent down your spine, and if it's not the case, I'm glad to meet another fellow low-level enthusiast. Regardless, you shouldn't be scared of any of this. To you as a programmer eBPF code will seem as any other regular code you write. Certain syntax can be a bit odd and debugging can be a bit of a pain for many reasons, but chiefly amongst them is that errors are not self explanatory so it's up to you to take a guess and figure it out.

Understanding eBPF

What is eBPF?

    Good, everything I said is nice and all but what is eBPF? It stands for extended Berkley Packet Filter. Imagine having a magical magnifying glass that not only lets you see every tiny detail of your computer's inner workings but also allows you to interact with and modify these details in real-time. That's pretty much what eBPF is like for modern operating systems.

     In simple terms, eBPF is a virtual machine inside the Linux kernel. It allows you to run custom programs in the kernel without changing the kernel’s source code or risking its stability. Think of it as a sandbox where small programs can safely play, analyze, and even change what's happening inside your system.

    Here’s the cool part: eBPF can hook into various points of the operating system, like network events, system calls, or even specific functions within the kernel. This gives you unprecedented visibility and control. Want to see every single network packet your server handles? eBPF can do that. Need to monitor how often a particular function is called and how long it takes to execute? Fear no more.

     But it doesn't stop at just observing. eBPF programs can also take action based on what they observe. For example, if a certain type of suspicious activity is detected, an eBPF program can automatically block it or trigger an alert. This makes eBPF incredibly powerful for tasks like performance monitoring, debugging, and security enforcement.

    What is eBPF then? It's like an Swiss army knife to help you monitor and control many different events happening inside the Linux Kernel at runtime with almost no performance overhead.

A bit of history

    Picture the internet in the '90s: dial-up connections, basic HTML websites, and the need for a way to efficiently filter network packets. Enter the Berkeley Packet Filter (BPF), a nifty tool developed to do just that. BPF was a simple and elegant solution for capturing and analyzing network traffic.

    Fast forward to today, and BPF has hit the gym, bulked up, and evolved into the what we now know as eBPF. This evolution isn't just a minor upgrade, it's a complete metamorphosis. The leap from BPF to eBPF is like upgrading from a skateboard to a super sports car. By building on the foundation of BPF, eBPF expanded its scope far beyond just network packets. Now, eBPF programs can hook into a myriad of system events, from file operations and system calls to kernel functions and user-space applications.

    The Linux kernel community, seeing the potential, embraced eBPF wholeheartedly. Over the years, it has been integrated into various parts of the kernel, making it a versatile tool for developers and sysadmins. With each new Linux release, eBPF gets more hooks, more capabilities and more popularity within the community.

    So whether you're a developer hunting down performance bottlenecks, a sysadmin keeping systems secure, or a researcher exploring the depths of kernel space, eBPF is the magic wand you've been longing for.

 How does it work?

    Alright, let's dive into the nuts and bolts of eBPF, but don't worry, I'll try my best to keep it light and interesting.

    As I mentioned before, eBPF is, at it's core, a virtual machine that runs inside the Linux kernel capable of executing small programs that you've written. These programs are then verified by the Verifier (we'll talk about this fella later) so they are safe to run. Furthermore, they run in a sandboxed environment, which means they can do a lot of powerful stuff without crashing your system. If the program fails, it affects nothing but the sandbox created for it's execution. What a relief, right? Tinkering with the Kernel without having to worry about killing it forever.

    When you write an eBPF program, you're essentially writing code that hooks into specific points in the Kernel. These hooks can be various events, such as network packet processing, entering or exiting system calls and even low-level kernel functions. The beauty of eBPF is that it lets you attach these tiny programs to virtually any part of the system, giving you unprecedented visibility and control.

    Typically, these programs are written in C or a specialized eBPF assembly language. Once you've written your code, you compile it into eBPF bytecode. This is then loaded into the kernel using a system call (usually bpf()). But before it can run, it has to pass through a strict verifier.

    Think of the Verifier as the bouncer at a nightclub. It checks your program to ensure it's safe to run in the kernel. This means no infinite loops, no illegal memory access, and no crashing the party. If your code doesn't meet these criteria, it's not getting in.

   What are these criteria de Verifier follows? Many of them, but mainly the following:

    What's Allowed in eBPF Programs:

1. Reading Kernel Data: eBPF programs can read data from various parts of the kernel, like network packets, system call arguments, or process information.
   
2. Manipulating Data: You can modify certain data structures or packets in transit. For example, you can change the contents of a network packet before it reaches its destination.
3. Collecting Metrics: eBPF excels at gathering performance metrics, such as tracking how long specific functions take to execute or counting the number of times certain events occur.
4. Filtering Events: You can filter events based on specific criteria. For instance, you can drop network packets that meet certain conditions, like suspicious IP addresses.
5. Communicating with User Space: eBPF programs can pass data to user-space applications through special maps, allowing you to build real-time monitoring tools.

    What's Not Allowed in eBPF Programs:

1. Infinite Loops: Your eBPF program must terminate in a finite number of steps. The verifier enforces this to prevent any runaway code from freezing the system.
2. Direct Memory Access: eBPF programs can't directly access arbitrary memory locations. This prevents them from reading or writing sensitive kernel memory.
3. Complex Logic: While you can perform basic arithmetic and logic operations, eBPF isn't designed for complex algorithms or heavy computations. This defeats the biggest advantage of eBPF programs.
   
4. Kernel Modification: You can't make changes to the Kernel's core functionality. eBPF is meant for observation and minor modifications, not altering the kernel's behavior.
5. Unbounded Loops or Recursion: eBPF programs must be simple and predictable, which means no recursive functions or loops that could run indefinitely.

    A quick way of understanding this is the following diagram:

     Once the eBPF program is executing, it will be triggered every time the hook you attached it to launches. So if you hooked the program to a system call, for example execve, every time this syscall executes, the program will be triggered.

    The idea of a eBPF program is mainly to collect data and then send it to the user space and process it with a program written in any language you like. Obviously there are certain languages, like Go, that offer more eBPF support and can make things a lot easier.

    Why do we need to send data to the user space? Because we should not (or out right can't) process information in a eBPF program. How? By using data structures and maps, but we'll get to it later.

The Advantages of eBPF

Enhanced Observability

    As we have already mentioned, one of the main advantages of eBPF is its ability to observe kernel events with almost no performance overhead, making it a big competitor to traditional observing tools that may require more of the computer's resources in order to do it's job.

    This is crucial in Cloud Computing where a company can save loads of money by reducing the amount of computational power needed to run their infrastructure. Given that nowadays many companies run the software and infrastructure on the cloud for convenience and flexibility reasons, reducing costs is always on the side of the eye.

    Even for more traditional companies that "own their metal" this is a big improvement. The less resources needed for performance observability, the more can be redirected towards computational tasks that bring revenue to the company.

   Furthermore, eBPF allows not only to see things without affecting performance or needing a particular infrastructure to run on, it also gives more detail, allowing you to specifically and exactly get what you need. This comes at a cost of course, that being you have to know what you need and where to locate it.

    Many GUI observability tools allow you to fish for metrics and more or less hope it's what you want, and modify it later if that's not the case. With eBPF not only you need to know what you need, you also need to know where it's located, what Kernel event or function has the information you're so desperately looking for.

Security and Network Filtering

    eBPF is also incredibly useful for security and network filtering due to its ability to run sandboxed programs inside the Linux kernel itself. We already mentioned that these programs can be attached to various kernel hooks and tracepoints, allowing deep inspection and modification of virtually every kernel subsystem.

    For security, eBPF programs can be used to implement sophisticated system call filtering and audit logging. By attaching a filter to a system call like openat() for instance, an eBPF program could check if a process is trying to access sensitive files and prevent unauthorized access. 

    This might seem scary for those of you who are a bit paranoid about security, because if a program can prevent access to a file, how can you be certain it won't be used to grant it? Good question however, the eBPF program must be first coded and then attached to the Kernel tracepoint with root privileges, meaning only someone who already has administrator access can modify or attach new programs or modify existing ones.

   All this means, basically, that first you have to develop an eBPF file to obtain the information you want and manage file accesses, then you need to compile it and attach it to the Linux Kernel with administrator access and finally some magic happens. If someone not authorized can do all this in your system, trust me, the fact a forbidden file is accessed is the least of your concerns.

    I'm sure eventually some eBPF-related vulnerabilities will be found and will need to be patched. No program is perfect and no system is unhackable, but so far is decently secured.

    eBPF is also powerful for network filtering and monitoring. Programs can be attached to traffic control hooks to implement firewall rules, DDoS mitigation, load balancing and more - all from within the kernel itself. This avoids costly kernel-user transitions and allows extremely high packet processing rates.

    As packets traverse the networking stack, eBPF programs can inspect and modify packets, hook into traffic control disciplines like HTML, perform encapsulation/decapsulation and more. This makes eBPF very effective for implementing software-defined networking and network function virtualization.

    Additionally, eBPF provides secure unprivileged capabilities - tools can be given specific kernel privileges without fully elevated root access. This sandboxing improves security compared to legacy kernel modules and minimizes attack surfaces.

Flexibility and Programmability

    eBPF is like the Swiss Army knife of kernel programming—versatile, multi-functional, and incredibly handy. One of the standout advantages of eBPF is its flexibility and programmability. Unlike traditional kernel modules, which require recompiling and rebooting the system for each change, eBPF allows you to load and run programs on the fly, without disrupting the running kernel. This dynamic nature is a game-changer for developers and system administrators alike, enabling rapid iteration and deployment of kernel-level functionality.

    eBPF programs can be written in a restricted C dialect, which is then translated into bytecode that can be loaded into the kernel via tools like bpftrace or bcc. These programs have access to a rich set of helper functions and data structures provided by the eBPF virtual machine, enabling complex operations like socket filtering, tracing, and security policy enforcement.

    Once loaded, eBPF programs can interact with user-space applications through eBPF maps, which act as shared memory regions. This bi-directional communication channel allows eBPF programs to collect data and make decisions based on real-time information from both the kernel and user space. For example, an eBPF program could monitor network traffic and relay relevant statistics to a user-space monitoring tool, or receive configuration updates from a user-space control plane.

    A lesser-known aspect of eBPF's programmability is its ability to interact with various kernel subsystems in a unified manner. Whether you're dealing with network stack events, filesystem operations, process scheduling, or tracing system calls, eBPF provides a consistent programming interface and a rich set of hooks and tracepoints. This makes it easier to develop cross-cutting tools that can gather insights from different parts of the system and correlate them to get a more holistic view.

    Furthermore, the ecosystem around eBPF is rapidly evolving, with projects like cilium (for networking and security), bpftrace (for advanced tracing), and bcc (BPF Compiler Collection) making it easier to write and deploy eBPF programs. These tools provide higher-level abstractions, domain-specific languages, and utilities, enabling you to harness the power of eBPF without delving too deep into the complexities of low-level kernel programming.

 Finally

    There's a lot more I had to say about eBPF but I decide to split it into more short and digestible posts so they don't take as long and are more interesting to the reader. I know it's been a year and something since my last post and I'll be honest, I forgot I had this blog at all so yeah. Now I decide to take it back and see how long it lasts.

    Regardless, in the following posts I'll be discussing things like why isn't eBPF more popular, the learning curve, biggest drawbacks and maybe I'll illustrate with an example I have coded with eBPF and Golang.

    Until then, this adventure stalls here, for now.