Skip to content
TheChatArchiveThe Chat Archive
GeminiGemini 3 Pro Preview· googleHugging Face Frontier Reasoning Traces

Implement a Python function to detect plagiarism in text documents.

  1. User

    Implement a Python function to detect plagiarism in text documents.

  2. Gemini Assistant

    Evaluating the Goal and Scope

    I've been drilling down on the core objective: building a Python function for plagiarism detection. Now, I'm defining the scope, exploring levels of complexity. I'm seeing the limitations of simple string matching, which is a good starting point, but clearly insufficient. We need to move beyond the naïve approach.

    Choosing the Implementation Method

    I'm leaning towards the TF-IDF and Cosine Similarity approach, because it's a solid, efficient, and well-understood solution. I'll include Jaccard similarity as a useful comparison, to illustrate the differences in approach and sensitivity. Now I'm working through the TF-IDF implementation details, aiming for clarity and efficiency.

    Planning the Code Structure

    I'm solidifying the code's structure now. I'll include a preprocessing function that handles cleaning the text, crucial for both similarity methods. After that, two primary functions, one for Jaccard, one for TF-IDF Cosine Similarity. I'm focusing on modularity, to keep it clean.

    Structuring the Function and Output

    I'm now focusing on the final presentation of the function. It should be modular, taking text as input and returning a similarity score. It will also clarify input flexibility, i.e., whether the input is text from a file, or a string. I will include a brief introductory explanation of TF-IDF and Cosine Similarity.

    Refining the Solution's Structure

    I am now structuring the function for the final presentation. It needs to be versatile, so I'm considering text input as both a raw string and also as the path to a file. I will also make sure the user understands how the TF-IDF and Cosine Similarity approach works. I'm focusing on clarity of output.

    Outlining the Code's Structure

    \n\n

    I'm now detailing the plagiarism detection code. It will start with the import statements, then the pre-processing function, followed by separate functions for Jaccard and TF-IDF similarity. The main part will call these functions, process the input (file or string), and generate the similarity score. I'm focusing on ensuring modularity for flexibility.

    Here are two ways to implement plagiarism detection.

    1. The Standard Approach (TF-IDF & Cosine Similarity): This uses the scikit-learn library. It is robust, handles word frequency, and detects similarity even if the document lengths differ.
    2. The Simple Approach (Jaccard Similarity): This uses standard Python (no heavy libraries) and compares the intersection of unique words.

    Approach 1: TF-IDF with Cosine Similarity (Recommended)

    This method converts text into numerical vectors (considering how unique words are to a specific document) and calculates the angle between them. Identifying sequence matches (bigrams) makes it much more accurate.

    Prerequisites:

    pip install scikit-learn
    

    Python Implementation:

    import os
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.metrics.pairwise import cosine_similarity
    
    def check_plagiarism(texts):
        """
        Calculates the similarity between a list of text strings.
        
        Args:
            texts (list): A list of strings (documents).
            
        Returns:
            list: A list of tuples (index_a, index_b, score).
        """
        
        # 1. Initialize the Vectorizer
        # ngram_range=(1, 2) considers both single words and 2-word phrases (bigrams)
        # to capture context and sentence structure, not just keywords.
        vectorizer = TfidfVectorizer(stop_words='english', ngram_range=(1, 2))
        
        # 2. Transform texts to TF-IDF Vectors
        tfidf_matrix = vectorizer.fit_transform(texts)
        
        # 3. Calculate Cosine Similarity
        # This returns a matrix where [i][j] is the similarity between doc i and j
        similarity_matrix = cosine_similarity(tfidf_matrix)
        
        plagiarism_results = []
        
        # 4. Extract and sort results
        # We iterate through the matrix, avoiding the diagonal (self-comparison)
        # and duplicates (comparing A vs B and B vs A).
        for i in range(len(texts)):
            for j in range(i + 1, len(texts)):
                score = similarity_matrix[i][j]
                plagiarism_results.append((i, j, score))
                
        # Sort by highest similarity score
        plagiarism_results.sort(key=lambda x: x[2], reverse=True)
        
        return plagiarism_results
    
    # --- Example Usage ---
    
    student_a = "The quick brown fox jumps over the lazy dog."
    student_b = "The quick brown fox jumped over the lazy dog." # Highly similar
    student_c = "Machine learning is a field of inquiry devoted to understanding and building methods that learn."
    student_d = "A field of study that gives computers the ability to learn without being explicitly programmed." # Same topic, different words
    
    documents = [student_a, student_b, student_c, student_d]
    doc_names = ["Student A (Fox)", "Student B (Fox Edit)", "Student C (ML Def 1)", "Student D (ML Def 2)"]
    
    results = check_plagiarism(documents)
    
    print("--- Plagiarism Detection Report ---")
    for idx_a, idx_b, score in results:
        print(f"Similarity: {score:.2%} | {doc_names[idx_a]} vs {doc_names[idx_b]}")
    

    How it works:

    1. TF-IDF: Checks not just if a word exists, but how "important" it is (rare words weigh more).
    2. N-grams: ngram_range=(1,2) ensures that "University of New York" is treated differently than just "New", "York", and "University" scattered randomly.
    3. Cosine Similarity: Returns a score between 0 (no similarity) and 1 (identical).

    Approach 2: Jaccard Similarity (No External Libraries)

    If you cannot install scikit-learn, this simple method calculates the intersection over the union of unique words sets.

    import string
    
    def preprocess(text):
        # Remove punctuation and convert to lowercase
        text = text.translate(str.maketrans('', '', string.punctuation)).lower()
        return set(text.split())
    
    def get_jaccard_similarity(str1, str2): 
        a = preprocess(str1)
        b = preprocess(str2)
        
        intersection = len(a.intersection(b))
        union = len(a.union(b))
        
        return intersection / union
    
    # --- Example Usage ---
    doc1 = "Deep learning is a subset of machine learning."
    doc2 = "Deep learning is a subset of machine learning strategies."
    
    similarity = get_jaccard_similarity(doc1, doc2)
    
    print(f"Jaccard Similarity: {similarity:.2%}")
    

    Limitations of Jaccard:

    • It ignores word frequency (repeating a word 10 times is the same as writing it once).
    • It is sensitive to document length (a short sentence copied into a massive book will result in a very low score).

    Summary

    For a production-level or serious project, use Approach 1. It is mathematically robust against the tricks people usually use to hide plagiarism (like changing one or two words in a sentence).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.