Skip to content
TheChatArchiveThe Chat Archive
GeminiGemini 3 Pro Preview· googleHugging Face Frontier Reasoning Traces

Develop a kernel module in C that adds a new system call to Linux.

  1. User

    Develop a kernel module in C that adds a new system call to Linux.

  2. Gemini Assistant

    Developing the System Call

    I'm focusing now on understanding the project's goal - a C kernel module adding a system call. I'm deep diving into how to do this, particularly exploring the system call table modification method. I'm weighing up the pros and cons of this approach compared to more modern alternatives and the implications for portability and maintainability.

    Assessing System Call Methods

    I've been evaluating different system call implementation approaches. While older tutorials heavily emphasized direct sys_call_table manipulation and disabling write protection, this is now strongly discouraged due to security concerns. Modern kernels favor modifying the kernel source and recompiling or utilizing ftrace/kprobes for function hooking. For this task, I'm leaning toward the "syscall table hacking" approach, but I need to include significant disclaimers due to its inherent risks and unreliability. I'm focusing on acknowledging kallsyms_lookup_name and its potential implications.

    Outlining Code Implementation

    Currently, I'm working through the code step-by-step. I've decided to demonstrate the "hacking" method, emphasizing its academic context. I'm focusing on hijacking sys_mkdir as a target, as this avoids the complexities of defining a new system call number. The prerequisites are clear, and I'm outlining how to locate sys_call_table, save the original function, disable write protection, modify the table, and restore everything upon module exit. I'm taking notes on potential pitfalls.

    Adding a System Call

    I'm now determining how to handle adding a new system call in the code. Hijacking __NR_mkdir is tempting, but the task specifies an addition. This necessitates choosing an unused slot in the system call table. Locating this unused slot is the next challenge. For newer kernels (post-5.7), kallsyms_lookup_name is no longer directly available, so I'll need a kprobe workaround. I'm focusing on crafting a helper function using kprobe for symbol resolution.

    Finalizing System Call Logic

    I'm now implementing the new system call function and working through the kprobe workaround for symbol resolution, since kallsyms_lookup_name isn't directly available in newer kernels. I've designed the my_new_syscall function and included printk for verification. I'm focusing on the initialization and exit routines, and I've decided to hijack __NR_mkdir since the user wants addition, I'll use it to showcase. I'm taking care to include appropriate safety mechanisms and disclaimers.

    Choosing a System Call

    I've decided to proceed with implementing a replacement mechanism, showcasing how to modify a specific existing system call. I will use a #define for the chosen system call number, making it easily configurable. I'll make the replacement clear for the user to understand. I'm focusing on locating an appropriate, unused syscall.

    Preparing Code for User

    I'm now integrating the code for presentation, including the Makefile, user application, and thorough instructions for compilation, loading, testing, and unloading the module. I'm focusing on making it as user-friendly as possible, with clear explanations of each step. I've re-examined the code, confirming syntactic correctness and the logic's soundness. The warnings about the approach's inherent risks are in place.

    This is a complex task because the Linux kernel developers have intentionally made this difficult to improve security.

    Historically, you could simply export the sys_call_table symbol and modify it. In modern kernels (v5.7+), the kernel no longer exports sys_call_table or even kallsyms_lookup_name to modules to prevent rootkits. Furthermore, the system call table is in read-only memory.

    To achieve this in a modern kernel (Standard Ubuntu/Debian/Fedora distributions today), we must:

    1. Use kprobes to dynamically find the memory address of kallsyms_lookup_name.
    2. Use that to find the address of sys_call_table.
    3. Manipulate the CPU Control Register (CR0) to disable write protection temporarily.
    4. Hijack an existing (or unimplemented) system call vector.

    ⚠️ Critical Disclaimer

    Do not run this on a production server. Modifying the system call table at runtime is unsafe. It can lead to system instability, crashes (Kernel Panics), and security vulnerabilities. This is strictly for educational purposes.


    1. Validating the Kernel Version

    This code is designed for x86_64 kernels.

    2. The Kernel Module Source (syscall_module.c)

    We will overwrite syscall 335. In many standard Linux kernels, vectors in the high 300s/400s are often unimplemented (pointing to sys_ni_syscall), or we can hijack a defined one. For this example, we will define a new function and hook it into the table.

    #include <linux/module.h>
    #include <linux/kernel.h>
    #include <linux/init.h>
    #include <linux/unistd.h>
    #include <linux/syscalls.h>
    #include <linux/kprobes.h>
    
    MODULE_LICENSE("GPL");
    MODULE_AUTHOR("Educational User");
    MODULE_DESCRIPTION("A module to add a custom system call");
    MODULE_VERSION("0.1");
    
    // The system call number we want to hijack/implement.
    // 335 is usually free on many x86_64 kernels, but check /usr/include/asm/unistd_64.h or equivalent.
    // If 335 is taken, this will corrupt that syscall.
    #define NEW_SYSCALL_NUM 335
    
    // Pointer to the original system call table
    unsigned long *sys_call_table_ptr;
    
    // Function pointer to store the original system call implementation (so we can restore it later)
    asmlinkage long (*original_syscall)(const struct pt_regs *);
    
    // Helper to resolve access to kallsyms_lookup_name (unexported in kernel > 5.7)
    typedef unsigned long (*kallsyms_lookup_name_t)(const char *name);
    kallsyms_lookup_name_t kallsyms_lookup_name_func;
    
    // Kprobe struct to find kallsyms_lookup_name's address
    struct kprobe kp = {
        .symbol_name = "kallsyms_lookup_name",
    };
    
    // --------------------------------------------------------------------------------
    // The Custom System Call Function
    // --------------------------------------------------------------------------------
    // Since Kernel 4.17+, syscalls take a pt_regs struct on x86_64
    asmlinkage long sys_my_custom_call(const struct pt_regs *regs)
    {
        printk(KERN_INFO "kernel_module: Custom system call %d invoked!\n", NEW_SYSCALL_NUM);
        return 0; // Return 0 for success
    }
    
    // --------------------------------------------------------------------------------
    // Helper functions to bypass Write Protection (CR0 Register)
    // --------------------------------------------------------------------------------
    static inline void write_cr0_forced(unsigned long val)
    {
        unsigned long __force_order;
        // Assembly to write to CR0
        asm volatile("mov %0, %%cr0" : : "r"(val), "m"(__force_order));
    }
    
    static inline void unprotect_memory(void)
    {
        // Bit 16 is the Write Protect (WP) flag
        write_cr0_forced(read_cr0() & (~0x10000));
        printk(KERN_INFO "kernel_module: Write protection disabled.\n");
    }
    
    static inline void protect_memory(void)
    {
        // Re-enable Bit 16
        write_cr0_forced(read_cr0() | 0x10000);
        printk(KERN_INFO "kernel_module: Write protection enabled.\n");
    }
    
    // --------------------------------------------------------------------------------
    // Module Initialization
    // --------------------------------------------------------------------------------
    static int __init mod_init(void)
    {
        printk(KERN_INFO "kernel_module: Loading...\n");
    
        // 1. Find kallsyms_lookup_name using kprobes (the workaround for modern kernels)
        int ret = register_kprobe(&kp);
        if (ret < 0) {
            printk(KERN_ERR "kernel_module: Failed to register kprobe. Can't resolve symbols.\n");
            return ret;
        }
        kallsyms_lookup_name_func = (kallsyms_lookup_name_t) kp.addr;
        unregister_kprobe(&kp); // We found the address, we don't need the probe anymore
    
        if (!kallsyms_lookup_name_func) {
            printk(KERN_ERR "kernel_module: kallsyms_lookup_name not found.\n");
            return -EFAULT;
        }
    
        // 2. Find the system call table address
        sys_call_table_ptr = (unsigned long *)kallsyms_lookup_name_func("sys_call_table");
    
        if (!sys_call_table_ptr) {
            printk(KERN_ERR "kernel_module: sys_call_table not found.\n");
            return -EFAULT;
        }
    
        printk(KERN_INFO "kernel_module: sys_call_table found at %p\n", sys_call_table_ptr);
    
        // 3. Save the original syscall so we can restore it on exit
        // Note: We cast the address in the table to a function pointer
        original_syscall = (void *)sys_call_table_ptr[NEW_SYSCALL_NUM];
    
        // 4. Overwrite the system call table
        unprotect_memory(); // Disable Read-Only
        sys_call_table_ptr[NEW_SYSCALL_NUM] = (unsigned long)sys_my_custom_call;
        protect_memory();   // Re-enable limit
    
        printk(KERN_INFO "kernel_module: Syscall %d hijacked.\n", NEW_SYSCALL_NUM);
    
        return 0;
    }
    
    // --------------------------------------------------------------------------------
    // Module Cleanup
    // --------------------------------------------------------------------------------
    static void __exit mod_exit(void)
    {
        if (sys_call_table_ptr) {
            unprotect_memory();
            // Restore the original system call
            sys_call_table_ptr[NEW_SYSCALL_NUM] = (unsigned long)original_syscall;
            protect_memory();
            printk(KERN_INFO "kernel_module: Syscall %d restored.\n", NEW_SYSCALL_NUM);
        }
        printk(KERN_INFO "kernel_module: Unloaded.\n");
    }
    
    module_init(mod_init);
    module_exit(mod_exit);
    

    3. The Makefile

    You need the kernel headers installed for your current kernel (sudo apt install linux-headers-$(uname -r) on Debian/Ubuntu).

    Create a file named Makefile:

    obj-m += syscall_module.o
    
    all:
    	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
    
    clean:
    	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
    

    4. Userspace Tester (test_syscall.c)

    Since this is a new/hijacked syscall, glibc doesn't know about it. We must use the syscall() function from unistd.h.

    #include <stdio.h>
    #include <unistd.h>
    #include <sys/syscall.h>
    
    // Must match the ID in the kernel module
    #define NEW_SYSCALL_NUM 335
    
    int main() {
        printf("Invoking kernel syscall %d...\n", NEW_SYSCALL_NUM);
        
        // Invoke the syscall
        // Arguments: Syscall Number, optional args...
        long res = syscall(NEW_SYSCALL_NUM);
        
        if(res == 0) {
            printf("System call returned success!\n");
            printf("Check 'dmesg' to see the kernel log output.\n");
        } else {
            perror("System call failed");
        }
    
        return 0;
    }
    

    5. Instructions to Run

    1. Compile the Module:

      make
      

      This generates syscall_module.ko.

    2. Compile the Tester:

      gcc test_syscall.c -o test_syscall
      
    3. Load the Module:

      sudo insmod syscall_module.ko
      
    4. Check Kernel Logs:

      dmesg | tail
      

      You should see "Loading...", "sys_call_table found...", and "Syscall hijacked".

    5. Run the Tester:

      ./test_syscall
      

      It should output "System call returned success!".

    6. Verify Execution: Run dmesg | tail again. You should see: kernel_module: Custom system call 335 invoked!

    7. Unload and Clean Up:

      sudo rmmod syscall_module
      

      Note: Always unload the module before shutting down or modifying the code, as leaving the syscall table pointing to unallocated memory (after the module is removed) will instantly crash the OS next time something calls that ID. The code provided handles restoration in the mod_exit function.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.