write me a golang function to do file transfer with configurable workers and buffer…
- User
write me a golang function to do file transfer with configurable workers and buffer which works fir any file size.
- Assistant
Certainly! Below is a Go function that efficiently transfers a file from a source to a destination using concurrent workers and configurable buffer sizes. This implementation ensures that it can handle files of any size by dividing the file into chunks and processing them in parallel.
Overview
-
Function Signature:
func TransferFile(srcPath, dstPath string, numWorkers int, bufferSize int) errorsrcPath: Path to the source file.dstPath: Path to the destination file.numWorkers: Number of concurrent workers to use.bufferSize: Size of the buffer (in bytes) each worker uses.
-
Approach:
- Divide the File: The source file is divided into chunks based on the number of workers.
- Concurrent Processing: Each worker reads a chunk from the source and writes it to the destination.
- Synchronization: A
WaitGroupensures that all workers complete their tasks before the function returns. - Error Handling: Any error encountered by a worker is captured and returned.
-
Advantages:
- Scalability: Handles large files by processing them in chunks.
- Configurability: Users can adjust the number of workers and buffer size based on system resources.
- Efficiency: Concurrent processing can significantly speed up the transfer, especially on multi-core systems.
Implementation
package main import ( "errors" "fmt" "io" "os" "sync" ) // TransferFile copies a file from srcPath to dstPath using concurrent workers. // numWorkers defines how many goroutines will perform the copy. // bufferSize defines the size of the buffer each worker uses. func TransferFile(srcPath, dstPath string, numWorkers int, bufferSize int) error { // Open the source file for reading srcFile, err := os.Open(srcPath) if err != nil { return fmt.Errorf("failed to open source file: %w", err) } defer srcFile.Close() // Get the file information to determine the size srcInfo, err := srcFile.Stat() if err != nil { return fmt.Errorf("failed to stat source file: %w", err) } fileSize := srcInfo.Size() // Create the destination file with the same permissions dstFile, err := os.Create(dstPath) if err != nil { return fmt.Errorf("failed to create destination file: %w", err) } defer func() { cerr := dstFile.Close() if err == nil && cerr != nil { err = cerr } }() // Ensure the destination file has the same size if err := dstFile.Truncate(fileSize); err != nil { return fmt.Errorf("failed to set destination file size: %w", err) } // Calculate chunk size per worker var chunkSize int64 = fileSize / int64(numWorkers) var chunks []struct { offset int64 size int64 } var i int for i = 0; i < numWorkers-1; i++ { offset := int64(i) * chunkSize chunks = append(chunks, struct { offset int64 size int64 }{ offset: offset, size: chunkSize, }) } // Last chunk takes the remaining bytes lastChunkSize := fileSize - int64(i)*chunkSize if lastChunkSize > 0 { chunks = append(chunks, struct { offset int64 size int64 }{ offset: int64(i) * chunkSize, size: lastChunkSize, }) } // Channel to capture errors from workers errCh := make(chan error, numWorkers) var wg sync.WaitGroup // Start workers for _, chunk := range chunks { wg.Add(1) go func(chunk struct{ offset, size int64 }) { defer wg.Done() buffer := make([]byte, bufferSize) remaining := chunk.size currentOffset := chunk.offset for remaining > 0 { readSize := int64(bufferSize) if remaining < readSize { readSize = remaining } // Read from source bytesRead, readErr := srcFile.ReadAt(buffer[:readSize], currentOffset) if bytesRead > 0 { // Write to destination bytesWritten, writeErr := dstFile.WriteAt(buffer[:bytesRead], currentOffset) if writeErr != nil { errCh <- fmt.Errorf("write error at offset %d: %w", currentOffset, writeErr) return } if int64(bytesWritten) != int64(bytesRead) { errCh <- fmt.Errorf("incomplete write at offset %d: wrote %d of %d bytes", currentOffset, bytesWritten, bytesRead) return } currentOffset += int64(bytesWritten) remaining -= int64(bytesWritten) } if readErr != nil { if errors.Is(readErr, io.EOF) { break } errCh <- fmt.Errorf("read error at offset %d: %w", currentOffset, readErr) return } } }(chunk) } // Wait for all workers to finish wg.Wait() close(errCh) // Check for errors for e := range errCh { if e != nil { return e } } return nil } // Example usage func main() { src := "path/to/source/file" dst := "path/to/destination/file" workers := 4 buffer := 1024 * 1024 // 1MB buffer err := TransferFile(src, dst, workers, buffer) if err != nil { fmt.Printf("File transfer failed: %v\n", err) } else { fmt.Println("File transfer completed successfully.") } }Explanation
-
Opening Files:
- The source file is opened in read-only mode.
- The destination file is created (or truncated if it exists) with the same permissions as the source.
-
Dividing the File:
- The file size is determined using
Stat(). - The file is divided into
numWorkerschunks. The last chunk may be larger if the file size isn't perfectly divisible by the number of workers.
- The file size is determined using
-
Concurrent Workers:
- For each chunk, a goroutine is spawned.
- Each goroutine reads its assigned chunk in segments defined by
bufferSizeand writes them to the correct position in the destination file usingReadAtandWriteAt. - Errors during read/write operations are sent to an error channel.
-
Synchronization and Error Handling:
- A
WaitGroupwaits for all goroutines to finish. - After all workers finish, the error channel is checked. If any worker encountered an error, the function returns that error.
- A
-
Example Usage:
- The
mainfunction demonstrates how to use theTransferFilefunction. Adjust thesrc,dst,workers, andbuffervariables as needed.
- The
Considerations
- File Permissions: The destination file inherits default permissions. You may want to adjust the permissions to match the source file.
- Atomicity: This implementation does not guarantee atomicity. If you require atomic file transfers, consider writing to a temporary file and renaming it upon successful completion.
- Error Handling: The function returns on the first error encountered. You might want to collect all errors or implement retries based on your specific requirements.
- Resource Limits: Ensure that
bufferSize * numWorkersdoes not exceed your system's memory constraints, especially for very large buffer sizes or high numbers of workers.
This implementation provides a robust and efficient way to transfer files of any size using Go's concurrency features. Adjust the
numWorkersandbufferSizeparameters to optimize performance based on your system's capabilities and the nature of the files being transferred. -