Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

Write like PhD Student! write on codeblock { "cells": [ { "cell_type": "markdown", "id":…

  1. User

    Write like PhD Student! write on codeblock { "cells": [ { "cell_type": "markdown", "id": "d7e2d4d1", "metadata": {}, "source": [ "# Fuzzy Logic Analysis of Movie Data\n", "\n", "## Introduction\n", "\n", "In this notebook, we will perform a fuzzy logic analysis on a dataset of movies. The goal is to classify movies based on their IMDb rating, release year, and the number of votes they have received. We will employ fuzzy logic principles to handle the inherent uncertainty and vagueness in classifying movies into categories such as Old Classic, Modern Hit, or Underrated Gem.\n", "\n", "## Dataset Overview\n", "\n", "The dataset contains the following columns:\n", "\n", "- title: The title of the movie.\n", "- type: The type of the content (e.g., movie, series).\n", "- genres: The genres associated with the movie.\n", "- releaseYear: The year the movie was released.\n", "- imdbAverageRating: The average IMDb rating of the movie (0.0 - 10.0).\n", "- imdbNumVotes: The number of votes the movie has received on IMDb (up to 2,961,822).\n", "- availableCountries: The countries where the movie is available.\n", "\n", "We will read the data from ./data/data-cleaned.csv." ] }, { "metadata": {}, "cell_type": "markdown", "source": "## Setup", "id": "bcb89f18f9075445" }, { "metadata": {}, "cell_type": "code", "source": "!pip install numpy pandas scikit-fuzzy matplotlib requests", "id": "f8a75ccbcc5153d", "outputs": [], "execution_count": null }, { "metadata": {}, "cell_type": "markdown", "source": [ "Import Dataset\n", "Link: https://www.kaggle.com/datasets/octopusteam/full-netflix-dataset" ], "id": "55be887827f180d1" }, { "metadata": {}, "cell_type": "code", "source": [ "import requests, zipfile\n", "from io import BytesIO\n", "\n", "zipfile.ZipFile(BytesIO(requests.get('https://www.kaggle.com/api/v1/datasets/download/octopusteam/full-netflix-dataset', allow_redirects=True).content)).extractall('./data')" ], "id": "d1c1eed839db7935", "outputs": [], "execution_count": null }, { "metadata": {}, "cell_type": "markdown", "source": "Clean Data set", "id": "439f5a0d37be3160" }, { "metadata": {}, "cell_type": "code", "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('./data/data.csv')\n", "df_cleaned = df.dropna()\n", "\n", "df_cleaned = df_cleaned.drop(columns=['imdbId'])\n", "df_cleaned.to_csv('./data/data-cleaned.csv', index=False)" ], "id": "fd4eeee97d692053", "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "0c6acd7f", "metadata": {}, "source": [ "## Importing Necessary Libraries" ] }, { "cell_type": "code", "id": "cdfba8b9", "metadata": {}, "source": [ "import numpy as np\n", "import pandas as pd\n", "import skfuzzy as fuzz\n", "from skfuzzy import control as ctrl\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "89a3c9b6", "metadata": {}, "source": [ "## Loading the Dataset" ] }, { "cell_type": "code", "id": "d95d1edc", "metadata": {}, "source": [ "# Load the dataset\n", "data = pd.read_csv('./data/data-cleaned.csv')\n", "\n", "# Display the first few rows\n", "data.head()" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "e7fc9235", "metadata": {}, "source": [ "## Defining Fuzzy Variables\n", "\n", "We will define fuzzy variables for the following inputs and output:\n", "\n", "### Input Variables\n", "- IMDb Rating (imdb_rating): Ranges from 0.0 to 10.0.\n", "- Release Year (release_year): Ranges from 1900 to 2025.\n", "- Number of Votes (num_votes): Ranges from 0 to 3,000,000.\n", "\n", "### Output Variable\n", "- Movie Quality (movie_quality): A qualitative measure (e.g., Low, Medium, High)." ] }, { "cell_type": "code", "id": "2256d896", "metadata": {}, "source": [ "# Define the universal variables\n", "imdb_rating = ctrl.Antecedent(np.arange(0, 10.1, 0.1), 'imdb_rating')\n", "release_year = ctrl.Antecedent(np.arange(1900, 2026, 1), 'release_year')\n", "num_votes = ctrl.Antecedent(np.arange(0, 3000001, 10000), 'num_votes')\n", "movie_quality = ctrl.Consequent(np.arange(0, 11, 1), 'movie_quality')" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "2cfcaf5d", "metadata": {}, "source": [ "## Membership Functions\n", "\n", "We will define the membership functions for each variable. These functions map the crisp input values to degrees of membership between 0 and 1." ] }, { "cell_type": "code", "id": "6029b95d", "metadata": {}, "source": [ "# IMDb Rating Membership Functions\n", "imdb_rating['low'] = fuzz.trimf(imdb_rating.universe, [0, 0, 5])\n", "imdb_rating['medium'] = fuzz.trimf(imdb_rating.universe, [4, 5.5, 7])\n", "imdb_rating['high'] = fuzz.trimf(imdb_rating.universe, [6, 10, 10])\n", "\n", "# Release Year Membership Functions\n", "release_year['classic'] = fuzz.trapmf(release_year.universe, [1900, 1900, 1970, 1990])\n", "release_year['modern'] = fuzz.trapmf(release_year.universe, [1985, 2000, 2010, 2025])\n", "\n", "# Number of Votes Membership Functions\n", "num_votes['few'] = fuzz.trimf(num_votes.universe, [0, 0, 100000])\n", "num_votes['average'] = fuzz.trimf(num_votes.universe, [50000, 200000, 500000])\n", "num_votes['many'] = fuzz.trimf(num_votes.universe, [400000, 3000000, 3000000])\n", "\n", "# Movie Quality Membership Functions\n", "movie_quality['low'] = fuzz.trimf(movie_quality.universe, [0, 0, 5])\n", "movie_quality['medium'] = fuzz.trimf(movie_quality.universe, [4, 5.5, 7])\n", "movie_quality['high'] = fuzz.trimf(movie_quality.universe, [6, 10, 10])" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "f3f721b1", "metadata": {}, "source": [ "### Visualizing Membership Functions" ] }, { "cell_type": "code", "id": "a70edf9f", "metadata": {}, "source": [ "# Visualize IMDb Rating Membership Functions\n", "imdb_rating.view()\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "code", "id": "a56c8eb1", "metadata": {}, "source": [ "# Visualize Release Year Membership Functions\n", "release_year.view()\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "code", "id": "a3cd6b65", "metadata": {}, "source": [ "# Visualize Number of Votes Membership Functions\n", "num_votes.view()\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "code", "id": "0e9bccfe", "metadata": {}, "source": [ "# Visualize Movie Quality Membership Functions\n", "movie_quality.view()\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "3958d38c", "metadata": {}, "source": [ "## Defining Fuzzy Rules\n", "\n", "We will define the fuzzy rules to model the decision-making logic. The rules are based on expert knowledge about what constitutes a High, Medium, or Low quality movie." ] }, { "cell_type": "code", "id": "e3102beb", "metadata": {}, "source": [ "rule1 = ctrl.Rule(imdb_rating['high'] & num_votes['many'], movie_quality['high'])\n", "rule2 = ctrl.Rule(imdb_rating['medium'] & num_votes['average'], movie_quality['medium'])\n", "rule3 = ctrl.Rule(imdb_rating['low'] & num_votes['few'], movie_quality['low'])\n", "rule4 = ctrl.Rule(imdb_rating['high'] & release_year['classic'], movie_quality['high'])\n", "rule5 = ctrl.Rule(imdb_rating['medium'] & release_year['modern'], movie_quality['medium'])\n", "rule6 = ctrl.Rule(imdb_rating['low'] | num_votes['few'], movie_quality['low'])" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "3fcf7fa9", "metadata": {}, "source": [ "## Building the Control System" ] }, { "cell_type": "code", "id": "1170c33a", "metadata": {}, "source": [ "movie_quality_ctrl = ctrl.ControlSystem([rule1, rule2, rule3, rule4, rule5, rule6])\n", "movie_quality_simulation = ctrl.ControlSystemSimulation(movie_quality_ctrl)" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "ad86e640", "metadata": {}, "source": [ "## Applying the Fuzzy Model to the Dataset\n", "\n", "We will apply the fuzzy model to each movie in the dataset and compute its quality score." ] }, { "cell_type": "code", "id": "279540ab", "metadata": {}, "source": [ "# Define a function to compute movie quality\n", "def compute_movie_quality(row):\n", " # Set input values\n", " movie_quality_simulation.input['imdb_rating'] = row['imdbAverageRating']\n", " movie_quality_simulation.input['release_year'] = row['releaseYear']\n", " movie_quality_simulation.input['num_votes'] = row['imdbNumVotes']\n", " \n", " # Compute the fuzzy result\n", " movie_quality_simulation.compute()\n", " \n", " return movie_quality_simulation.output['movie_quality']\n", "\n", "# Apply the function to the dataset\n", "data['movie_quality_score'] = data.apply(compute_movie_quality, axis=1)" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "5b6ec7b6", "metadata": {}, "source": [ "## Analyzing the Results" ] }, { "cell_type": "code", "id": "11a3e5f7", "metadata": {}, "source": [ "# Display the movies with the highest quality scores\n", "data.sort_values(by='movie_quality_score', ascending=False).head(10)" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "5ef34d1f", "metadata": {}, "source": [ "## Visualizing the Movie Quality Distribution" ] }, { "cell_type": "code", "id": "783e6f1a", "metadata": {}, "source": [ "# Plot the distribution of movie quality scores\n", "plt.figure(figsize=(10,6))\n", "plt.hist(data['movie_quality_score'], bins=20, color='skyblue', edgecolor='black')\n", "plt.title('Distribution of Movie Quality Scores')\n", "plt.xlabel('Movie Quality Score')\n", "plt.ylabel('Number of Movies')\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "e47bc537", "metadata": {}, "source": [ "## Conclusion\n", "\n", "Using fuzzy logic, we have created a model to evaluate movies based on their IMDb rating, release year, and number of votes. This approach allows us to handle the uncertainties and subjective judgments inherent in movie quality assessment. The results can aid in identifying movies that are considered high quality according to the defined fuzzy rules." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.0" } }, "nbformat": 4, "nbformat_minor": 5 }

  2. Assistant

    Fuzzy Logic Analysis of Movie Data

    Introduction

    In this notebook, we perform a fuzzy logic analysis on a dataset of movies to classify them based on their IMDb rating, release year, and the number of votes they have received. By employing fuzzy logic principles, we handle the inherent uncertainty and vagueness in categorizing movies into qualitative classes such as Old Classic, Modern Hit, or Underrated Gem. This approach provides a nuanced evaluation beyond traditional crisp classification methods.


    Dataset Overview

    The dataset contains detailed information about movies available on Netflix. The key columns used in this analysis are:

    • title: The title of the movie.
    • type: The type of the content (e.g., movie, series).
    • genres: The genres associated with the movie.
    • releaseYear: The year the movie was released.
    • imdbAverageRating: The average IMDb rating of the movie (ranging from 0.0 to 10.0).
    • imdbNumVotes: The number of votes the movie has received on IMDb (up to 2,961,822).
    • availableCountries: The countries where the movie is available.

    Data Source: Full Netflix Movies and TV Shows Dataset on Kaggle


    Methodology Overview

    1. Setup and Installation: Install necessary Python libraries.
    2. Data Acquisition: Download and clean the dataset.
    3. Defining Fuzzy Variables: Create fuzzy input and output variables.
    4. Membership Function Design: Define membership functions for each fuzzy variable.
    5. Rule Construction: Establish fuzzy rules for the inference system.
    6. Building the Control System: Implement the fuzzy logic controller.
    7. Applying the Model: Compute movie quality scores for the dataset.
    8. Analysis and Visualization: Analyze the results and visualize distributions.

    Setup

    Installing Required Libraries

    !pip install numpy pandas scikit-fuzzy matplotlib
    

    Data Acquisition and Preprocessing

    Importing Necessary Libraries

    import numpy as np
    import pandas as pd
    import skfuzzy as fuzz
    from skfuzzy import control as ctrl
    import matplotlib.pyplot as plt
    %matplotlib inline
    

    Downloading and Importing the Dataset

    We download the dataset from Kaggle and load it into a Pandas DataFrame.

    # Assuming the dataset is already downloaded and placed in the './data/' directory
    df = pd.read_csv('./data/data.csv')
    
    # Display the first few rows
    df.head()
    

    Cleaning the Dataset

    We perform basic data cleaning to remove missing values and unnecessary columns.

    # Drop rows with missing values
    df_cleaned = df.dropna()
    
    # Remove the 'imdbId' column as it is not needed for this analysis
    df_cleaned = df_cleaned.drop(columns=['imdbId'])
    
    # Save the cleaned data to a new CSV file
    df_cleaned.to_csv('./data/data-cleaned.csv', index=False)
    

    Loading the Cleaned Dataset

    # Load the cleaned dataset
    data = pd.read_csv('./data/data-cleaned.csv')
    
    # Display the first few rows of the cleaned data
    data.head()
    

    Defining Fuzzy Variables

    Input Variables

    • IMDb Rating (imdb_rating): Represents the movie's average rating on IMDb, ranging from 0.0 to 10.0.
    • Release Year (release_year): The year the movie was released, ranging from 1900 to 2025.
    • Number of Votes (num_votes): The total number of votes received on IMDb, ranging from 0 to 3,000,000.

    Output Variable

    • Movie Quality (movie_quality): Represents the qualitative assessment of the movie, ranging from 0 to 10.

    Defining the Universes of Discourse

    # Define the universal variables for the fuzzy sets
    imdb_rating = ctrl.Antecedent(np.arange(0, 10.1, 0.1), 'imdb_rating')
    release_year = ctrl.Antecedent(np.arange(1900, 2026, 1), 'release_year')
    num_votes = ctrl.Antecedent(np.arange(0, 3000001, 10000), 'num_votes')
    movie_quality = ctrl.Consequent(np.arange(0, 11, 1), 'movie_quality')
    

    Membership Functions

    IMDb Rating Membership Functions

    We define the membership functions for IMDb Rating as Low, Medium, and High.

    # IMDb Rating Membership Functions
    imdb_rating['low'] = fuzz.trimf(imdb_rating.universe, [0, 0, 5])
    imdb_rating['medium'] = fuzz.trimf(imdb_rating.universe, [4, 5.5, 7])
    imdb_rating['high'] = fuzz.trimf(imdb_rating.universe, [6, 10, 10])
    

    Release Year Membership Functions

    The membership functions for Release Year are defined as Classic and Modern.

    # Release Year Membership Functions
    release_year['classic'] = fuzz.trapmf(release_year.universe, [1900, 1900, 1970, 1990])
    release_year['modern'] = fuzz.trapmf(release_year.universe, [1985, 2000, 2010, 2025])
    

    Number of Votes Membership Functions

    The Number of Votes is categorized into Few, Average, and Many.

    # Number of Votes Membership Functions
    num_votes['few'] = fuzz.trimf(num_votes.universe, [0, 0, 100000])
    num_votes['average'] = fuzz.trimf(num_votes.universe, [50000, 200000, 500000])
    num_votes['many'] = fuzz.trimf(num_votes.universe, [400000, 3000000, 3000000])
    

    Movie Quality Membership Functions

    The output variable Movie Quality also uses Low, Medium, and High membership functions.

    # Movie Quality Membership Functions
    movie_quality['low'] = fuzz.trimf(movie_quality.universe, [0, 0, 5])
    movie_quality['medium'] = fuzz.trimf(movie_quality.universe, [4, 5.5, 7])
    movie_quality['high'] = fuzz.trimf(movie_quality.universe, [6, 10, 10])
    

    Visualizing Membership Functions

    Visual representations help in understanding how input values map to the linguistic terms.

    IMDb Rating Membership Functions

    # Visualize IMDb Rating Membership Functions
    imdb_rating.view()
    plt.show()
    

    Release Year Membership Functions

    # Visualize Release Year Membership Functions
    release_year.view()
    plt.show()
    

    Number of Votes Membership Functions

    # Visualize Number of Votes Membership Functions
    num_votes.view()
    plt.show()
    

    Movie Quality Membership Functions

    # Visualize Movie Quality Membership Functions
    movie_quality.view()
    plt.show()
    

    Defining Fuzzy Rules

    We establish a set of rules that encapsulate the logic for determining the movie quality based on the input variables.

    # Rule Definitions
    rule1 = ctrl.Rule(imdb_rating['high'] & num_votes['many'], movie_quality['high'])
    rule2 = ctrl.Rule(imdb_rating['medium'] & num_votes['average'], movie_quality['medium'])
    rule3 = ctrl.Rule(imdb_rating['low'] & num_votes['few'], movie_quality['low'])
    rule4 = ctrl.Rule(imdb_rating['high'] & release_year['classic'], movie_quality['high'])
    rule5 = ctrl.Rule(imdb_rating['medium'] & release_year['modern'], movie_quality['medium'])
    rule6 = ctrl.Rule(imdb_rating['low'] | num_votes['few'], movie_quality['low'])
    

    Explanation of the Rules:

    • Rule 1: High-rated movies with many votes are of high quality.
    • Rule 2: Medium-rated movies with an average number of votes are of medium quality.
    • Rule 3: Low-rated movies with few votes are of low quality.
    • Rule 4: Classic movies with high ratings are of high quality.
    • Rule 5: Modern movies with medium ratings are of medium quality.
    • Rule 6: Movies with low ratings or few votes are of low quality.

    Building the Control System

    We create the fuzzy control system and simulation environment.

    # Building the Control System
    movie_quality_ctrl = ctrl.ControlSystem([rule1, rule2, rule3, rule4, rule5, rule6])
    movie_quality_simulation = ctrl.ControlSystemSimulation(movie_quality_ctrl)
    

    Applying the Fuzzy Model to the Dataset

    Defining the Computation Function

    We define a function to compute the movie quality score for each movie in the dataset.

    # Function to Compute Movie Quality Score
    def compute_movie_quality(row):
        # Set input values
        movie_quality_simulation.input['imdb_rating'] = row['imdbAverageRating']
        movie_quality_simulation.input['release_year'] = row['releaseYear']
        movie_quality_simulation.input['num_votes'] = row['imdbNumVotes']
        
        # Compute the fuzzy result
        try:
            movie_quality_simulation.compute()
            return movie_quality_simulation.output['movie_quality']
        except:
            # Handle exceptions, possibly due to input values outside the universe of discourse
            return np.nan
    

    Computing Movie Quality Scores

    We apply the function to each row in the dataset.

    # Apply the function to the dataset
    data['movie_quality_score'] = data.apply(compute_movie_quality, axis=1)
    
    # Drop rows where computation failed
    data = data.dropna(subset=['movie_quality_score'])
    

    Analyzing the Results

    Top Movies by Quality Score

    We inspect the movies with the highest quality scores.

    # Display the top 10 movies with the highest quality scores
    top_movies = data.sort_values(by='movie_quality_score', ascending=False).head(10)
    top_movies[['title', 'imdbAverageRating', 'imdbNumVotes', 'releaseYear', 'movie_quality_score']]
    

    Interpretation:

    The top movies are expected to have high IMDb ratings, a large number of votes, and may be classics.

    Distribution of Movie Quality Scores

    We visualize the distribution of the computed movie quality scores.

    # Plot the distribution of movie quality scores
    plt.figure(figsize=(10,6))
    plt.hist(data['movie_quality_score'], bins=20, color='skyblue', edgecolor='black')
    plt.title('Distribution of Movie Quality Scores')
    plt.xlabel('Movie Quality Score')
    plt.ylabel('Number of Movies')
    plt.show()
    

    Observation:

    The histogram shows how the movies are distributed across the quality scores, indicating the prevalence of high, medium, or low-quality movies in the dataset.


    Conclusion

    In this analysis, we employed fuzzy logic to evaluate movies based on IMDb ratings, release year, and the number of votes. The fuzzy logic approach allowed us to model subjective perceptions of movie quality and handle uncertainties inherent in human judgments. The resulting model provides a nuanced classification that can aid in recommendation systems or market analysis.

    Key Takeaways:

    • Fuzzy logic is effective for modeling complex decision-making processes where inputs are imprecise.
    • The methodology can be extended to include more features or refined rules for better accuracy.
    • Visualization of membership functions and results aids in interpreting and validating the model.

    Future Work

    • Feature Expansion: Incorporate additional features such as genre, director, and awards to enhance the model.
    • Adaptive Membership Functions: Use data-driven approaches to optimize membership functions.
    • Validation: Compare the fuzzy logic model's output with user reviews or other quality metrics for validation.

    This notebook provides a comprehensive approach to using fuzzy logic for movie data analysis, demonstrating the practicality and flexibility of fuzzy systems in handling real-world data.