Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Engineer a secure file upload backend with Tus protocol, malware scanning, EXIF…

  1. User

    Engineer a secure file upload backend with Tus protocol, malware scanning, EXIF stripping, and auditing.

  2. GLM Assistant

    The user wants a secure file upload backend using the Tus protocol. The key requirements are:

    1. Tus Protocol: Resumable file uploads.
    2. Malware Scanning: Checking files for viruses/malware.
    3. EXIF Stripping: Removing metadata from images for privacy.
    4. Auditing: Logging actions for security and accountability.

    I need to architect a solution. A diagram or a textual architecture description would be helpful. I'll choose a stack. Python (FastAPI) or Go (Golang) are good candidates. Go is often preferred for high-performance network services, but Python has great libraries for data processing. Let's go with Go for the core server (using tusd as a base or a library) or perhaps a custom implementation using a library like tus-go (though tusd is the standard reference implementation). Actually, running tusd as a sidecar or binary and wrapping it is often easier, but writing a custom handler in Go or Node.js gives more control.

    Let's stick to Python (FastAPI) for readability and ease of demonstrating the logic, using a library like tuspy on the client side and implementing a custom Tus handler or using gunicorn with a specific worker. Actually, implementing the full Tus protocol from scratch is error-prone. The best practice is usually to use tusd (the Go implementation) via hooks. tusd handles the protocol complexity (chunking, resuming, HEAD requests), and calls HTTP hooks (webhooks) for the logic (scanning, stripping).

    Architecture Plan:

    1. Tus Server: Use tusd (standard Go binary). It handles the file receiving.
    2. Hook Service: A Python (FastAPI) or Go service that tusd calls.
      • pre-finish: Trigger EXIF stripping and Malware scanning.
      • post-finish: Trigger auditing.
    3. Malware Scanner: ClamAV is the industry standard for open-source.
    4. EXIF Tool: exiftool via command line or an Image library (Pillow) in Python.
    5. Storage: Local filesystem or S3 (keep it simple: local).
    6. Database: PostgreSQL or Redis for audit logs and file metadata.

    Let's refine the approach for a single cohesive response: I will provide a solution based on Go using the tusd library programmatically. This allows embedding the protocol handling directly into the application, which is more "engineering" focused than just configuring a binary.

    Key Components:

    • Language: Go (Golang) - performant, type-safe, standard for backend infra.
    • Tus Library: github.com/tus/tusd (standard implementation).
    • Malware: Interface for ClamAV (using clamd go client).
    • EXIF: Go library for image processing (e.g., github.com/dsoprea/go-exif or just calling exiftool binary for robustness). Let's use imaging or a specific exif library.
    • Auditing: Structured logging to a file or database.

    Step-by-Step Implementation Plan:

    1. Setup & Dependencies: Define the go.mod.
    2. Tus Handler Setup: Configure the tusd handler (storage, locker).
    3. Interceptor/Hook: Implement the PostReceive or PreFinish hook.
    4. Malware Scanning: Integrate a function to send the file stream to ClamAV socket.
    5. EXIF Stripping: Process image files to remove metadata.
    6. Auditing: Middleware or hooks to log events (UploadStart, UploadEnd, SecurityAlert).
    7. Main Server: Wire everything up to an HTTP listener.

    Specific Details:

    • Malware: In a real prod env, you scan during upload (streaming) or immediately after chunks arrive. With Tus, since it's chunked, scanning usually happens on POST-CREATE (on empty file? no), POST-RECEIVE (on chunk), or POST-FINISH. Scanning on POST-FINISH is safest to ensure the whole file is there.
    • EXIF: Needs to happen before the file is marked "finished" or immediately move the file to a "clean" area. Let's modify the file in place on POST-FINISH.
    • Audit: Log to STDOUT (JSON format) for collection by a logger (like ELK/Splunk).

    Refining the Malware/EXIF logic:

    • Wait for upload to finish.
    • Open file.
    • Scan for Malware. If found, delete and log "Malware detected".
    • If clean, check MIME type. If image, strip EXIF.
    • Save changes.
    • Log success.

    Code Structure:

    • main.go: Entry point.
    • security.go: Malware and EXIF logic.
    • audit.go: Logging.

    Let's stick to a single file for the example to make it copy-pasteable, or well-structured blocks.

    Requirements Check:

    • Tus: Using tusd.
    • Malware: Simulating ClamAV integration (mock implementation with interface for real ClamAV).
    • EXIF: Using a library like github.com/disintegration/imaging (doesn't preserve EXIF by default usually when re-encoding, but dsoprea/go-exif is better for specific stripping). Actually, exiftool is the most reliable. I will write a helper function that attempts to strip EXIF using standard Go image libraries (decoding and re-encoding) which effectively strips it, or mention exiftool. I'll use the decode/re-encode method for pure Go dependency management.

    Auditing: Use a structured logger like zerolog or standard log/slog.

    Let's write the code.

    • Import github.com/tus/tusd/v2/pkg/handler.
    • Import github.com/tus/tusd/v2/pkg/filestore.
    • Setup filestore.FileStore.
    • Setup handler.Config.
    • Create NotifyCompleteUpload channel to trigger the post-processing pipeline.

    Self-Correction on Security: Simply stripping EXIF isn't enough if the file extension is spoofed. I should check magic bytes. Malware scanning is resource-intensive. In a real architecture, this should be asynchronous. For this prompt, synchronous processing after upload but before finalizing (or immediately after) is acceptable for a demo, but I'll add a note about async queues.

    Drafting the Response:

    1. Architecture Overview: Explain the tusd hooks.
    2. Prerequisites: Go, ClamAV (optional but recommended).
    3. The Code: A robust Go application.
    4. Docker/Compose: To run ClamAV easily.

    Let's build the Go solution.

    • Malware Scanner: Create a ClamAVScanner struct.
    • EXIF Stripper: Create a function StripExif. It reads the file, checks if it's JPEG/PNG, decodes, and re-encodes (without EXIF).
    • Auditor: JSON formatted logs.

    Wait, the user asked for "Engineer a secure...". This implies architecture + code.

    I will provide a complete system design using Go.

    1. Tusd wrapper: We don't write the protocol from scratch. We use tusd.
    2. Hooks: We use the tusd hook system to intercept finished uploads.
    3. Pipeline:
      • Upload completes -> Trigger Hook.
      • Step 1: Antivirus (ClamAV). If fail, delete, audit, abort.
      • Step 2: Integrity/Exif. Read file, strip exif, overwrite.
      • Step 3: Audit Success.

    Code specifics:

    • Need golang.org/x/image for decoding.
    • Need a ClamAV client. github.com/dutchcoders/go-clamd.

    Let's refine the EXIF stripping. Re-encoding an image (Decoding to image.Image and Encoding back to JPEG) effectively strips metadata because the standard Go image/jpeg library doesn't write EXIF blocks unless explicitly told to handle them. This is a "good enough" pure-Go approach for stripping.

    The Plan:

    1. Docker Compose: To spin up ClamAV.
    2. Go Backend Code:
      • Structs for Configuration.
      • FileStore setup.
      • HookHandler for post-processing.
      • Malware Scan function.
      • Image Process function.
      • Logger.

    This covers all requirements: Tus (resumable), Malware (ClamAV), EXIF (Processing), Audit (Logging).

    One complexity: tusd locks files. Modifying a file in-place while tusd manages it can be tricky. tusd uses a .info file alongside the binary. Ideally, we move the file to a "quarantine/processing" folder, scan it, process it, then move to "final storage". For simplicity in this code, I will implement a "Post-Receive" hook that waits for the offset to match the size (finished), then acts.

    Let's write the response. Use Go because it's the native language of tusd.

    Components:

    • Server: Go + tusd library.
    • Scanner: ClamAV (running in docker).
    • Storage: Local disk (for demo).

    Security Note: I must mention that stripping EXIF is best done by re-encoding, and that re-encoding can slightly degrade quality. For a production app, exiftool -all= is preferred, but requires CGO or exec. I will implement the Go-native re-encoding for portability.

    Final Polish: Ensure headers are correct. Ensure the code compiles mentally. composer is tusd handler. Hook PostReceive is called every time data is written. Checking for completion there allows blocking the "success" response until validation is done? No, tusd usually returns 204 immediately on chunk completion. Better to use the PreFinish hook (if available) or just handle it asynchronously on PostFinish. For "Secure", blocking is better. If the file has a virus, we don't want it sitting there even for a second as "valid". Actually, tusd doesn't easily support blocking the upload after the data is written but before the resource is considered valid without using the PreUploadCreate hook (for metadata validation) or custom logic. The standard way is:

    1. Upload finishes.
    2. Backend receives "Upload Finished" event.
    3. Backend marks file as "Pending Scan".
    4. Scanner runs.
    5. If clean, file moved to "Ready". If not, deleted.

    I will implement the asynchronous pattern via hooks for better stability.

    Auditing: Log fields: Timestamp, FileID, Size, IP (from X-Forwarded-For), EventType (Upload/ScanPass/ScanFail/ExifStrip).

    Let's go. Use Go.

    One detail: EXIF stripping. Go's image package handles decoding. jpeg.Encode writes minimal metadata. This is the safest pure-Go way.

    Let's structure the code. main.go:

    1. Setup Logger.
    2. Setup Store (FileStore).
    3. Setup ClamAV connection.
    4. Setup Tus Composer.
    5. Set up NotifyCompleteUpload channel.
    6. Start worker to read from channel -> Scan -> Strip -> Log.
    7. Start HTTP server.

    This is a solid, engineer-grade plan. This solution architects a production-grade file upload service in Go (Golang). Go is chosen for its concurrency model (goroutines), which is ideal for handling high-throughput file streams and blocking I/O operations (like malware scanning) efficiently.

    Architecture Overview

    1. Protocol: Tus (tusd) implementation for resumable uploads.
    2. Storage: Local filesystem (abstracted for easy S3 migration).
    3. Malware Protection: Integration with ClamAV.
    4. Privacy: EXIF stripping using imaging libraries.
    5. Auditing: Structured JSON logging with file integrity hashes (SHA-256).

    Prerequisites

    To run this locally, you need:

    • Go (1.21+)
    • ClamAV (running on localhost:3310 or via Docker)
    • Git

    1. Project Setup

    Initialize the module and install dependencies.

    mkdir secure-upload
    cd secure-upload
    go mod init secure-upload
    
    # Install Tus server core
    go get github.com/tus/tusd/v2/pkg/handler
    go get github.com/tus/tusd/v2/pkg/filestore
    
    # Install ClamAV driver and Image processing
    go get github.com/dutchcoders/go-clamd
    go get github.com/disintegration/imaging
    

    2. The Implementation (main.go)

    This single file contains the server setup, the middleware logic, and the processing pipeline.

    package main
    
    import (
    	"bytes"
    	"context"
    	"crypto/sha256"
    	"encoding/json"
    	"fmt"
    	"image"
    	"image/jpeg"
    	"image/png"
    	"io"
    	"log"
    	"log/slog"
    	"net/http"
    	"os"
    	"path/filepath"
    	"strings"
    	"time"
    
    	"github.com/disintegration/imaging"
    	clamd "github.com/dutchcoders/go-clamd"
    	"github.com/tus/tusd/v2/pkg/handler"
    	"github.com/tus/tusd/v2/pkg/filestore"
    )
    
    // Config holds application configuration
    type Config struct {
    	UploadDir   string
    	ClamAVAddr  string
    	ListenAddr  string
    	ProcessedDir string
    }
    
    // AuditLogger handles structured security logging
    type AuditLogger struct {
    	logger *slog.Logger
    }
    
    func NewAuditLogger() *AuditLogger {
    	// In production, output to a dedicated security log file or SIEM forwarder
    	return &AuditLogger{
    		logger: slog.New(slog.NewJSONHandler(os.Stdout, nil)),
    	}
    }
    
    func (a *AuditLogger) Log(event string, details map[string]interface{}) {
    	details["event"] = event
    	details["timestamp"] = time.Now().UTC().Format(time.RFC3339)
    	jsonStr, _ := json.Marshal(details)
    	a.logger.Info(string(jsonStr))
    }
    
    // SecurityPipeline handles post-upload processing
    type SecurityPipeline struct {
    	config  *Config
    	audit   *AuditLogger
    	clamd   *clamd.Clamd
    }
    
    func NewSecurityPipeline(cfg *Config, audit *AuditLogger) *SecurityPipeline {
    	// Initialize connection to ClamAV
    	c := clamd.NewClamd(cfg.ClamAVAddr)
    	return &SecurityPipeline{
    		config: cfg,
    		audit:  audit,
    		clamd:  c,
    	}
    }
    
    // Process runs the file through Malware Scan and EXIF stripping
    func (sp *SecurityPipeline) Process(fileInfo handler.FileInfo) error {
    	ctx := context.Background()
    	filePath := filepath.Join(sp.config.UploadDir, fileInfo.ID)
    
    	// 1. Open the uploaded file
    	f, err := os.Open(filePath)
    	if err != nil {
    		return fmt.Errorf("failed to open file for processing: %w", err)
    	}
    	defer f.Close()
    
    	// Calculate Hash for Integrity Auditing (Chain of Custody)
    	hasher := sha256.New()
    	// We use a TeeReader to calculate hash while reading for other operations
    	// But since we need to read multiple times (Scan + Image + Hash), 
    	// we read into memory or buffer. For large files, stream sequentially.
    	
    	// For simplicity in this example, we buffer. In prod, stream strictly once if RAM is tight.
    	var buf bytes.Buffer
    	tee := io.TeeReader(f, hasher)
    	if _, err := io.Copy(&buf, tee); err != nil {
    		return err
    	}
    	sha256Hash := fmt.Sprintf("%x", hasher.Sum(nil))
    
    	sp.audit.Log("UPLOAD_RECEIVED", map[string]interface{}{
    		"file_id":   fileInfo.ID,
    		"filename":  fileInfo.MetaData["filename"],
    		"size":      fileInfo.Size,
    		"sha256":    sha256Hash,
    		"uploader_ip": fileInfo.Context.Value("RemoteAddr"), // Extracted from request context
    	})
    
    	// 2. Malware Scanning
    	sp.audit.Log("SCAN_START", map[string]interface{}{"file_id": fileInfo.ID})
    	
    	// Use the buffered data for scanning
    	response, err := sp.clamd.ScanStream(ctx, &buf, nil)
    	if err != nil {
    		// Log connectivity error but fail closed (safe by default)
    		sp.audit.Log("SCAN_ERROR", map[string]interface{}{"file_id": fileInfo.ID, "error": err.Error()})
    		return fmt.Errorf("malware scanner unavailable: %w", err)
    	}
    
    	// Check scan results
    	for r := range response {
    		if r.Status == clamd.RES_FOUND {
    			sp.audit.Log("MALWARE_DETECTED", map[string]interface{}{
    				"file_id":    fileInfo.ID,
    				"virus":      r.Description,
    				"action":     "QUARANTINE_DELETE",
    			})
    			// Delete malicious file immediately
    			os.Remove(filePath)
    			return fmt.Errorf("malware detected: %s", r.Description)
    		}
    	}
    	sp.audit.Log("SCAN_PASS", map[string]interface{}{"file_id": fileInfo.ID})
    
    	// 3. EXIF Stripping (Sanitization)
    	// Only process if it's an image
    	contentType := fileInfo.MetaData["filetype"]
    	if strings.HasPrefix(contentType, "image/") {
    		sp.audit.Log("EXIF_STRIP_START", map[string]interface{}{"file_id": fileInfo.ID})
    		
    		cleanData, err := sp.stripExif(&buf, contentType)
    		if err != nil {
    			// If we can't strip it, we might reject it depending on policy.
    			// Here we log a warning but might proceed if we couldn't decode (e.g. not actually an image)
    			sp.audit.Log("EXIF_STRIP_WARN", map[string]interface{}{"file_id": fileInfo.ID, "error": err.Error()})
    		} else {
    			// Overwrite original file with cleaned version
    			if err := os.WriteFile(filePath, cleanData, 0644); err != nil {
    				return fmt.Errorf("failed to write sanitized file: %w", err)
    			}
    			sp.audit.Log("EXIF_STRIP_SUCCESS", map[string]interface{}{"file_id": fileInfo.ID})
    		}
    	}
    
    	sp.audit.Log("UPLOAD_FINALIZED", map[string]interface{}{"file_id": fileInfo.ID, "status": "CLEAN"})
    	return nil
    }
    
    // stripExif decodes the image and re-encodes it, effectively dropping metadata
    func (sp *SecurityPipeline) stripExif(r io.Reader, contentType string) ([]byte, error) {
    	img, err := imaging.Decode(r, imaging.AutoOrientation(true))
    	if err != nil {
    		return nil, fmt.Errorf("failed to decode image: %w", err)
    	}
    
    	var buf bytes.Buffer
    	switch contentType {
    	case "image/jpeg":
    		err = jpeg.Encode(&buf, img, &jpeg.Options{Quality: 90})
    	case "image/png":
    		err = png.Encode(&buf, img)
    	default:
    		// Fallback for other types (gif, bmp) or just return raw if unsupported
    		return nil, fmt.Errorf("unsupported image type for exif stripping")
    	}
    
    	if err != nil {
    		return nil, fmt.Errorf("failed to encode sanitized image: %w", err)
    	}
    	return buf.Bytes(), nil
    }
    
    func main() {
    	// Configuration
    	cfg := &Config{
    		UploadDir:    "./uploads",
    		ProcessedDir: "./processed",
    		ClamAVAddr:   "tcp://localhost:3310",
    		ListenAddr:   ":8080",
    	}
    
    	// Setup Dirs
    	os.MkdirAll(cfg.UploadDir, 0755)
    
    	audit := NewAuditLogger()
    	pipeline := NewSecurityPipeline(cfg, audit)
    
    	// 1. Setup File Store
    	store := filestore.New(cfg.UploadDir)
    
    	// 2. Setup Tus Composer
    	composer := handler.NewStoreComposer()
    	store.UseIn(composer)
    
    	// 3. Setup Handler Config
    	handlerConfig := handler.Config{
    		StoreComposer: composer,
    		// Notify us when an upload is finished
    		CompletedUploads: make(chan handler.FileInfo, 10),
    	}
    
    	tusHandler, err := handler.New(handlerConfig)
    	if err != nil {
    		log.Fatalf("Failed to create tus handler: %v", err)
    	}
    
    	// 4. Start Processing Worker
    	// This listens for completed uploads in the background
    	go func() {
    		for {
    			select {
    			case fileInfo := <-handlerConfig.CompletedUploads:
    				// Process asynchronously to not block the HTTP handler
    				go func(info handler.FileInfo) {
    					if err := pipeline.Process(info); err != nil {
    						log.Printf("Processing failed for %s: %v", info.ID, err)
    					}
    				}(fileInfo)
    			}
    		}
    	}()
    
    	// 5. Setup HTTP Server
    	http.Handle("/files/", http.StripPrefix("/files/", tusHandler))
    	
    	// Middleware to extract IP for auditing
    	loggedHandler := loggingMiddleware(http.DefaultServeMux, audit)
    
    	audit.Log("SERVER_START", map[string]interface{}{"port": cfg.ListenAddr})
    	log.Printf("Server listening on %s", cfg.ListenAddr)
    	if err := http.ListenAndServe(cfg.ListenAddr, loggedHandler); err != nil {
    		log.Fatalf("Server failed: %v", err)
    	}
    }
    
    func loggingMiddleware(next http.Handler, audit *AuditLogger) http.Handler {
    	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    		start := time.Now()
    		
    		// Basic header security checks (example)
    		if r.Header.Get("X-Forwarded-For") != "" {
    			// In a real prod setup behind a LB, you'd trust the LB's header
    		}
    
    		next.ServeHTTP(w, r)
    		
    		// We don't log every chunk request (Tus uses PATCH), only significant ones usually
    		// But for auditing we might want specific request logs.
    		// The audit logs inside the pipeline are the source of truth for files.
    		log.Printf("%s %s %s", r.Method, r.URL.Path, time.Since(start))
    	})
    }
    

    3. Docker Compose for Infrastructure

    To test this, you need a running ClamAV instance. Create a docker-compose.yml:

    version: '3.8'
    services:
      clamav:
        image: clamav/clamav:latest
        container_name: clamav
        ports:
          - "3310:3310"
        # Ensure it stays healthy and fresh
        restart: always
    

    Run docker-compose up -d.

    4. Security Design Breakdown

    A. The Tus Protocol (Resumable Uploads)

    We utilize tusd which implements the open Tus protocol.

    • Why? Large file uploads over unstable mobile networks often fail. Tus splits files into chunks. If a connection drops, the client resumes from the last byte without re-uploading.
    • Implementation: We use filestore for local disk storage. The CompletedUploads channel acts as a webhook trigger for our internal processing pipeline.

    B. Malware Scanning (The "Gatekeeper")

    • Mechanism: We use go-clamd to interface with ClamAV via TCP.
    • Fail Closed: If the scanner is down (connection error), the pipeline returns an error. The file is effectively in a "limbo" state (uploaded but not processed) or deleted based on policy.
    • Stream Efficiency: We use ScanStream so we don't have to write the file to a temporary location just for scanning; we scan the byte buffer.

    C. EXIF Stripping (Data Sanitization)

    • Mechanism: We use imaging to decode the image into raw pixel data and then re-encode it.
    • Result: This destroys all metadata tags (GPS location, Camera Model, User Name) embedded in JPEG/PNGs.
    • Safety: Re-encoding ensures no "polyglot" files (files that are valid images and valid scripts) can survive in image form, as the image structure is rebuilt from scratch.

    D. Auditing (Chain of Custody)

    • Structured Logging: We use JSON logs (slog) so they can be ingested by ELK stacks or Splunk.
    • Integrity: We calculate the SHA-256 hash of the original uploaded file. This proves what was uploaded before we modified it (stripped EXIF).
    • Events:
      • UPLOAD_RECEIVED: Tracks who uploaded it and the hash.
      • MALWARE_DETECTED: Security alert.
      • EXIF_STRIP_SUCCESS: Privacy compliance.

    5. How to Run and Test

    1. Start Dependencies: docker-compose up -d

    2. Run Server: go run main.go

    3. Upload a file: You can use curl or the TUS client libraries (JS/Python/iOS). A simple way to test the endpoint via curl (using the tus-php or simple manual creation simulation):

      # 1. Create the upload (HEAD request to check existence, then POST)
      curl -X POST http://localhost:8080/files/ \
        -H "Upload-Length: 1000" \
        -H "Upload-Metadata: filename dGVzdC5qcGc=,filetype aW1hZ2UvanBlZw==" \
        -v
      
      # Note the Location header from response, e.g., http://localhost:8080/files/abc123
      

      Note: TUS is binary protocol. For a real demo, use the Uppy.io dashboard configured to point to http://localhost:8080/files/.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.