Develop a multi-threaded file server in C with epoll-based I/O.
- User
Develop a multi-threaded file server in C with epoll-based I/O.
Assistant
Initiating Server Design
I'm starting by clarifying the core purpose: a multi-threaded file server in C, leveraging
epollfor efficient, non-blocking I/O. This involves handling multiple client connections concurrently. My initial focus is on understanding the fundamental design.Refining Architecture Choices
I'm now zeroing in on the Reactor Pattern with a thread pool. The main thread will manage
epoll, accepting connections and distributing tasks. I plan to use a job queue synchronized by mutexes and condition variables. Each worker thread will retrieve jobs from the queue, perform file I/O, and send responses. I'm exploring the nuances ofsendfile()versusread()/write()for file transmission efficiency.Defining Concurrency Strategy
I'm now contemplating how to integrate the thread pool effectively. Thinking about the main thread's role: accepting connections, and adding them to
epoll. When data arrives, I'm leaning toward removing the socket fromepolland enqueueing a task. The worker threads will then process the request and respond. The use ofsendfile()for zero-copy file transfer will also be utilized.Finalizing Implementation Steps
I'm solidifying the steps now. First, the boilerplate and thread pool setup. Then, a non-blocking listener socket. The core
epollloop comes next, handling new connections and data events. Worker threads will then process requests and serve the files. Finally, there's error handling, with a 404 response. I am finalizing the queue type as a fixed-size circular buffer.Defining the Data Structures
I'm now focusing on data structures. I'll employ a fixed-size circular buffer for the task queue, enhancing efficiency. I'm finalizing the structure to hold client file descriptors. The worker thread will handle reading the request, serving the file, and then closing the connection. I plan to use a basic HTTP/1.0-style GET request with error handling.
Synthesizing Code Structure
I am now focusing on the practical coding details. I am using the provided notes to draft the essential components. The task queue will be a fixed-size circular buffer. I will use
sendfile()to boost performance. The main loop will handle new connections and events. The worker threads process requests, serve files, and then close the connection. Error handling will be implemented, specifically a 404 response.Implementing The Core Logic
I am now structuring the
handle_clientfunction, integratingreadandsendfile. I'm using non-blocking I/O throughout, and implementing a basic HTTP GET request handler. Currently, I'm focusing on error handling, and specifically, the 404 response. I am testing the worker thread interaction and synchronization with mutexes and condition variables.This is a complete implementation of a high-performance static file server in C. It combines Epoll (for event notification), a Thread Pool (for processing requests concurrently), and Sendfile (for zero-copy data transfer).
Architecture
- Main Thread (The Listener):
- Listens on a port.
- Uses
epoll_waitto wait for events. - Accepts new connections and adds them to the epoll instance with
EPOLLONESHOT. - When a client sends a request, the main thread moves the file descriptor to a Task Queue and wakes up a worker.
- Worker Threads:
- Wait for tasks in the queue.
- Parse the simple HTTP GET request.
- Open the requested file.
- Use
sendfile()to stream the file directly from disk implementation to the network socket (kernel-space copy only). - Close the connection.
The Code
Save this code as
server.c.#define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include <errno.h> #include <fcntl.h> #include <sys/socket.h> #include <netinet/in.h> #include <sys/epoll.h> #include <sys/stat.h> #include <sys/sendfile.h> #include <sys/types.h> #include <pthread.h> #include <signal.h> #define PORT 8080 #define MAX_EVENTS 64 #define THREAD_POOL_SIZE 8 #define QUEUE_SIZE 256 #define BUFFER_SIZE 4096 #define WEB_ROOT "." // --- Task Queue for Thread Pool --- typedef struct { int client_fd; } task_t; task_t task_queue[QUEUE_SIZE]; int queue_head = 0; int queue_tail = 0; int queue_count = 0; pthread_mutex_t queue_mutex = PTHREAD_MUTEX_INITIALIZER; pthread_cond_t queue_cond = PTHREAD_COND_INITIALIZER; // --- Helper: Set Non-Blocking --- int set_nonblocking(int fd) { int flags = fcntl(fd, F_GETFL, 0); if (flags == -1) return -1; if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1) return -1; return 0; } // --- Helper: Send HTTP Error --- void send_error(int fd, int status, const char *msg) { char buffer[BUFFER_SIZE]; snprintf(buffer, sizeof(buffer), "HTTP/1.1 %d %s\r\n" "Content-Length: 0\r\n" "Connection: close\r\n\r\n", status, msg); write(fd, buffer, strlen(buffer)); } // --- Request Handler --- void handle_client(int client_fd) { char buffer[BUFFER_SIZE]; ssize_t bytes_read = read(client_fd, buffer, sizeof(buffer) - 1); if (bytes_read <= 0) { close(client_fd); return; // Connection closed or error } buffer[bytes_read] = '\0'; // Simple parsing to get the file path (GET /file.html HTTP/1.1) char method[16], path[256], protocol[16]; sscanf(buffer, "%s %s %s", method, path, protocol); if (strcasecmp(method, "GET") != 0) { send_error(client_fd, 405, "Method Not Allowed"); close(client_fd); return; } // Handle root path char file_path[512]; snprintf(file_path, sizeof(file_path), "%s%s", WEB_ROOT, strcmp(path, "/") == 0 ? "/index.html" : path); // Open file int file_fd = open(file_path, O_RDONLY); if (file_fd == -1) { send_error(client_fd, 404, "Not Found"); close(client_fd); return; } // Get file stats for Content-Length struct stat file_stat; fstat(file_fd, &file_stat); // Send HTTP Header char header[BUFFER_SIZE]; int header_len = snprintf(header, sizeof(header), "HTTP/1.1 200 OK\r\n" "Content-Length: %ld\r\n" "Connection: close\r\n\r\n", file_stat.st_size); /* Note: In a production server, we would handle partial writes here. For simplicity, we assume the header fits in one socket buffer. */ write(client_fd, header, header_len); // Send File using Zero-Copy (sendfile) off_t offset = 0; ssize_t sent_bytes = 0; while (offset < file_stat.st_size) { sent_bytes = sendfile(client_fd, file_fd, &offset, file_stat.st_size - offset); if (sent_bytes <= 0) { break; // Error or Client closed } } close(file_fd); close(client_fd); } // --- Workers --- void *worker_thread(void *arg) { while (1) { int client_fd; pthread_mutex_lock(&queue_mutex); while (queue_count == 0) { pthread_cond_wait(&queue_cond, &queue_mutex); } client_fd = task_queue[queue_head].client_fd; queue_head = (queue_head + 1) % QUEUE_SIZE; queue_count--; pthread_mutex_unlock(&queue_mutex); handle_client(client_fd); } return NULL; } // --- Main Server Setup --- int main() { int server_fd, epoll_fd; struct sockaddr_in address; struct epoll_event event, events[MAX_EVENTS]; // Ignore SIGPIPE (happens if client disconnects while we write) signal(SIGPIPE, SIG_IGN); // 1. Create Thread Pool pthread_t threads[THREAD_POOL_SIZE]; for (int i = 0; i < THREAD_POOL_SIZE; i++) { pthread_create(&threads[i], NULL, worker_thread, NULL); } // 2. Socket Setup if ((server_fd = socket(AF_INET, SOCK_STREAM, 0)) == 0) { perror("Socket failed"); exit(EXIT_FAILURE); } // Allow address reuse int opt = 1; setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)); address.sin_family = AF_INET; address.sin_addr.s_addr = INADDR_ANY; address.sin_port = htons(PORT); if (bind(server_fd, (struct sockaddr *)&address, sizeof(address)) < 0) { perror("Bind failed"); exit(EXIT_FAILURE); } if (listen(server_fd, 100) < 0) { perror("Listen failed"); exit(EXIT_FAILURE); } set_nonblocking(server_fd); // 3. Epoll Setup epoll_fd = epoll_create1(0); if (epoll_fd == -1) { perror("Epoll create failed"); exit(EXIT_FAILURE); } event.events = EPOLLIN | EPOLLET; // Edge-triggered for listener event.data.fd = server_fd; if (epoll_ctl(epoll_fd, EPOLL_CTL_ADD, server_fd, &event) == -1) { perror("Epoll ctl failed"); exit(EXIT_FAILURE); } printf("Server listening on port %d with %d threads...\n", PORT, THREAD_POOL_SIZE); // 4. Event Loop while (1) { int n = epoll_wait(epoll_fd, events, MAX_EVENTS, -1); for (int i = 0; i < n; i++) { if (events[i].data.fd == server_fd) { // --- Handle New Connections --- while (1) { struct sockaddr_in client_addr; socklen_t client_len = sizeof(client_addr); int client_fd = accept(server_fd, (struct sockaddr *)&client_addr, &client_len); if (client_fd == -1) { if (errno == EAGAIN || errno == EWOULDBLOCK) { break; // Processed all incoming connections } else { perror("Accept failed"); break; } } set_nonblocking(client_fd); struct epoll_event client_event; // EPOLLONESHOT: Ensures only one thread handles this socket at a time. // We re-add/mod the socket if we want to keep connection alive, // but here we close after one request (simple HTTP). client_event.events = EPOLLIN | EPOLLET | EPOLLONESHOT; client_event.data.fd = client_fd; epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client_fd, &client_event); } } else { // --- Handle Client Request (Data Ready) --- int client_fd = events[i].data.fd; // Add to Task Queue for Workers pthread_mutex_lock(&queue_mutex); if (queue_count < QUEUE_SIZE) { task_queue[queue_tail].client_fd = client_fd; queue_tail = (queue_tail + 1) % QUEUE_SIZE; queue_count++; pthread_cond_signal(&queue_cond); } else { // Queue full: reject connection to prevent server lockup close(client_fd); printf("Queue full, dropped connection\n"); } pthread_mutex_unlock(&queue_mutex); } } } close(server_fd); return 0; }How to Compile and Run
- Create a dummy file for testing:
echo "Hello, multithreaded world!" > index.html - Compile:
gcc -o server server.c -lpthread - Run:
./server - Test:
Open a browser and go to
http://localhost:8080/, or use curl:curl -v http://localhost:8080/index.html
Key Technical Decisions
EPOLLONESHOT: This is crucial in a multi-threaded environment. Without it, if a client sends data while a worker thread is already reading from the socket,epoll_waitmight fire again for the same socket. This would cause a second worker thread to pick up the same socket, leading to data corruption or race conditions.EPOLLONESHOTdisables the event after it triggers once.- Locking Strategy: The main thread only locks the mutex briefly to push an integer (file descriptor) onto the queue. The worker threads lock briefly to pop the integer. Actual I/O (reading request, reading disk, sending file) happens outside the lock, ensuring true concurrency.
sendfile(): We use thesendfilesystem call. Standardread()from disk andwrite()to socket copies data: Disk Kernel Buffer -> User Buffer -> Socket Kernel Buffer.sendfilecopies directly from Disk Buffer to Socket Buffer within the kernel, significantly reducing CPU usage and context switches.- Edge Triggered (
EPOLLET): Used for the listening socket to accept multiple incoming connections in a tight loop if a burst of clients arrives simultaneously.
- Main Thread (The Listener):
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.