Write like PhD Student! write on codeblock { "cells": [ { "cell_type": "markdown", "id":…
- User
Write like PhD Student! write on codeblock { "cells": [ { "cell_type": "markdown", "id": "d7e2d4d1", "metadata": {}, "source": [ "# Fuzzy Logic Analysis of Movie Data\n", "\n", "## Introduction\n", "\n", "In this notebook, we will perform a fuzzy logic analysis on a dataset of movies. The goal is to classify movies based on their IMDb rating, release year, and the number of votes they have received. We will employ fuzzy logic principles to handle the inherent uncertainty and vagueness in classifying movies into categories such as Old Classic, Modern Hit, or Underrated Gem.\n", "\n", "## Dataset Overview\n", "\n", "The dataset contains the following columns:\n", "\n", "-
title: The title of the movie.\n", "-type: The type of the content (e.g., movie, series).\n", "-genres: The genres associated with the movie.\n", "-releaseYear: The year the movie was released.\n", "-imdbAverageRating: The average IMDb rating of the movie (0.0 - 10.0).\n", "-imdbNumVotes: The number of votes the movie has received on IMDb (up to 2,961,822).\n", "-availableCountries: The countries where the movie is available.\n", "\n", "We will read the data from./data/data-cleaned.csv." ] }, { "metadata": {}, "cell_type": "markdown", "source": "## Setup", "id": "bcb89f18f9075445" }, { "metadata": {}, "cell_type": "code", "source": "!pip install numpy pandas scikit-fuzzy matplotlib requests", "id": "f8a75ccbcc5153d", "outputs": [], "execution_count": null }, { "metadata": {}, "cell_type": "markdown", "source": [ "Import Dataset\n", "Link: https://www.kaggle.com/datasets/octopusteam/full-netflix-dataset" ], "id": "55be887827f180d1" }, { "metadata": {}, "cell_type": "code", "source": [ "import requests, zipfile\n", "from io import BytesIO\n", "\n", "zipfile.ZipFile(BytesIO(requests.get('https://www.kaggle.com/api/v1/datasets/download/octopusteam/full-netflix-dataset', allow_redirects=True).content)).extractall('./data')" ], "id": "d1c1eed839db7935", "outputs": [], "execution_count": null }, { "metadata": {}, "cell_type": "markdown", "source": "Clean Data set", "id": "439f5a0d37be3160" }, { "metadata": {}, "cell_type": "code", "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('./data/data.csv')\n", "df_cleaned = df.dropna()\n", "\n", "df_cleaned = df_cleaned.drop(columns=['imdbId'])\n", "df_cleaned.to_csv('./data/data-cleaned.csv', index=False)" ], "id": "fd4eeee97d692053", "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "0c6acd7f", "metadata": {}, "source": [ "## Importing Necessary Libraries" ] }, { "cell_type": "code", "id": "cdfba8b9", "metadata": {}, "source": [ "import numpy as np\n", "import pandas as pd\n", "import skfuzzy as fuzz\n", "from skfuzzy import control as ctrl\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "89a3c9b6", "metadata": {}, "source": [ "## Loading the Dataset" ] }, { "cell_type": "code", "id": "d95d1edc", "metadata": {}, "source": [ "# Load the dataset\n", "data = pd.read_csv('./data/data-cleaned.csv')\n", "\n", "# Display the first few rows\n", "data.head()" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "e7fc9235", "metadata": {}, "source": [ "## Defining Fuzzy Variables\n", "\n", "We will define fuzzy variables for the following inputs and output:\n", "\n", "### Input Variables\n", "- IMDb Rating (imdb_rating): Ranges from 0.0 to 10.0.\n", "- Release Year (release_year): Ranges from 1900 to 2025.\n", "- Number of Votes (num_votes): Ranges from 0 to 3,000,000.\n", "\n", "### Output Variable\n", "- Movie Quality (movie_quality): A qualitative measure (e.g., Low, Medium, High)." ] }, { "cell_type": "code", "id": "2256d896", "metadata": {}, "source": [ "# Define the universal variables\n", "imdb_rating = ctrl.Antecedent(np.arange(0, 10.1, 0.1), 'imdb_rating')\n", "release_year = ctrl.Antecedent(np.arange(1900, 2026, 1), 'release_year')\n", "num_votes = ctrl.Antecedent(np.arange(0, 3000001, 10000), 'num_votes')\n", "movie_quality = ctrl.Consequent(np.arange(0, 11, 1), 'movie_quality')" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "2cfcaf5d", "metadata": {}, "source": [ "## Membership Functions\n", "\n", "We will define the membership functions for each variable. These functions map the crisp input values to degrees of membership between 0 and 1." ] }, { "cell_type": "code", "id": "6029b95d", "metadata": {}, "source": [ "# IMDb Rating Membership Functions\n", "imdb_rating['low'] = fuzz.trimf(imdb_rating.universe, [0, 0, 5])\n", "imdb_rating['medium'] = fuzz.trimf(imdb_rating.universe, [4, 5.5, 7])\n", "imdb_rating['high'] = fuzz.trimf(imdb_rating.universe, [6, 10, 10])\n", "\n", "# Release Year Membership Functions\n", "release_year['classic'] = fuzz.trapmf(release_year.universe, [1900, 1900, 1970, 1990])\n", "release_year['modern'] = fuzz.trapmf(release_year.universe, [1985, 2000, 2010, 2025])\n", "\n", "# Number of Votes Membership Functions\n", "num_votes['few'] = fuzz.trimf(num_votes.universe, [0, 0, 100000])\n", "num_votes['average'] = fuzz.trimf(num_votes.universe, [50000, 200000, 500000])\n", "num_votes['many'] = fuzz.trimf(num_votes.universe, [400000, 3000000, 3000000])\n", "\n", "# Movie Quality Membership Functions\n", "movie_quality['low'] = fuzz.trimf(movie_quality.universe, [0, 0, 5])\n", "movie_quality['medium'] = fuzz.trimf(movie_quality.universe, [4, 5.5, 7])\n", "movie_quality['high'] = fuzz.trimf(movie_quality.universe, [6, 10, 10])" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "f3f721b1", "metadata": {}, "source": [ "### Visualizing Membership Functions" ] }, { "cell_type": "code", "id": "a70edf9f", "metadata": {}, "source": [ "# Visualize IMDb Rating Membership Functions\n", "imdb_rating.view()\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "code", "id": "a56c8eb1", "metadata": {}, "source": [ "# Visualize Release Year Membership Functions\n", "release_year.view()\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "code", "id": "a3cd6b65", "metadata": {}, "source": [ "# Visualize Number of Votes Membership Functions\n", "num_votes.view()\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "code", "id": "0e9bccfe", "metadata": {}, "source": [ "# Visualize Movie Quality Membership Functions\n", "movie_quality.view()\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "3958d38c", "metadata": {}, "source": [ "## Defining Fuzzy Rules\n", "\n", "We will define the fuzzy rules to model the decision-making logic. The rules are based on expert knowledge about what constitutes a High, Medium, or Low quality movie." ] }, { "cell_type": "code", "id": "e3102beb", "metadata": {}, "source": [ "rule1 = ctrl.Rule(imdb_rating['high'] & num_votes['many'], movie_quality['high'])\n", "rule2 = ctrl.Rule(imdb_rating['medium'] & num_votes['average'], movie_quality['medium'])\n", "rule3 = ctrl.Rule(imdb_rating['low'] & num_votes['few'], movie_quality['low'])\n", "rule4 = ctrl.Rule(imdb_rating['high'] & release_year['classic'], movie_quality['high'])\n", "rule5 = ctrl.Rule(imdb_rating['medium'] & release_year['modern'], movie_quality['medium'])\n", "rule6 = ctrl.Rule(imdb_rating['low'] | num_votes['few'], movie_quality['low'])" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "3fcf7fa9", "metadata": {}, "source": [ "## Building the Control System" ] }, { "cell_type": "code", "id": "1170c33a", "metadata": {}, "source": [ "movie_quality_ctrl = ctrl.ControlSystem([rule1, rule2, rule3, rule4, rule5, rule6])\n", "movie_quality_simulation = ctrl.ControlSystemSimulation(movie_quality_ctrl)" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "ad86e640", "metadata": {}, "source": [ "## Applying the Fuzzy Model to the Dataset\n", "\n", "We will apply the fuzzy model to each movie in the dataset and compute its quality score." ] }, { "cell_type": "code", "id": "279540ab", "metadata": {}, "source": [ "# Define a function to compute movie quality\n", "def compute_movie_quality(row):\n", " # Set input values\n", " movie_quality_simulation.input['imdb_rating'] = row['imdbAverageRating']\n", " movie_quality_simulation.input['release_year'] = row['releaseYear']\n", " movie_quality_simulation.input['num_votes'] = row['imdbNumVotes']\n", " \n", " # Compute the fuzzy result\n", " movie_quality_simulation.compute()\n", " \n", " return movie_quality_simulation.output['movie_quality']\n", "\n", "# Apply the function to the dataset\n", "data['movie_quality_score'] = data.apply(compute_movie_quality, axis=1)" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "5b6ec7b6", "metadata": {}, "source": [ "## Analyzing the Results" ] }, { "cell_type": "code", "id": "11a3e5f7", "metadata": {}, "source": [ "# Display the movies with the highest quality scores\n", "data.sort_values(by='movie_quality_score', ascending=False).head(10)" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "5ef34d1f", "metadata": {}, "source": [ "## Visualizing the Movie Quality Distribution" ] }, { "cell_type": "code", "id": "783e6f1a", "metadata": {}, "source": [ "# Plot the distribution of movie quality scores\n", "plt.figure(figsize=(10,6))\n", "plt.hist(data['movie_quality_score'], bins=20, color='skyblue', edgecolor='black')\n", "plt.title('Distribution of Movie Quality Scores')\n", "plt.xlabel('Movie Quality Score')\n", "plt.ylabel('Number of Movies')\n", "plt.show()" ], "outputs": [], "execution_count": null }, { "cell_type": "markdown", "id": "e47bc537", "metadata": {}, "source": [ "## Conclusion\n", "\n", "Using fuzzy logic, we have created a model to evaluate movies based on their IMDb rating, release year, and number of votes. This approach allows us to handle the uncertainties and subjective judgments inherent in movie quality assessment. The results can aid in identifying movies that are considered high quality according to the defined fuzzy rules." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.0" } }, "nbformat": 4, "nbformat_minor": 5 } - Assistant
Fuzzy Logic Analysis of Movie Data
Introduction
In this notebook, we perform a fuzzy logic analysis on a dataset of movies to classify them based on their IMDb rating, release year, and the number of votes they have received. By employing fuzzy logic principles, we handle the inherent uncertainty and vagueness in categorizing movies into qualitative classes such as Old Classic, Modern Hit, or Underrated Gem. This approach provides a nuanced evaluation beyond traditional crisp classification methods.
Dataset Overview
The dataset contains detailed information about movies available on Netflix. The key columns used in this analysis are:
title: The title of the movie.type: The type of the content (e.g., movie, series).genres: The genres associated with the movie.releaseYear: The year the movie was released.imdbAverageRating: The average IMDb rating of the movie (ranging from 0.0 to 10.0).imdbNumVotes: The number of votes the movie has received on IMDb (up to 2,961,822).availableCountries: The countries where the movie is available.
Data Source: Full Netflix Movies and TV Shows Dataset on Kaggle
Methodology Overview
- Setup and Installation: Install necessary Python libraries.
- Data Acquisition: Download and clean the dataset.
- Defining Fuzzy Variables: Create fuzzy input and output variables.
- Membership Function Design: Define membership functions for each fuzzy variable.
- Rule Construction: Establish fuzzy rules for the inference system.
- Building the Control System: Implement the fuzzy logic controller.
- Applying the Model: Compute movie quality scores for the dataset.
- Analysis and Visualization: Analyze the results and visualize distributions.
Setup
Installing Required Libraries
!pip install numpy pandas scikit-fuzzy matplotlib
Data Acquisition and Preprocessing
Importing Necessary Libraries
import numpy as np import pandas as pd import skfuzzy as fuzz from skfuzzy import control as ctrl import matplotlib.pyplot as plt %matplotlib inlineDownloading and Importing the Dataset
We download the dataset from Kaggle and load it into a Pandas DataFrame.
# Assuming the dataset is already downloaded and placed in the './data/' directory df = pd.read_csv('./data/data.csv') # Display the first few rows df.head()Cleaning the Dataset
We perform basic data cleaning to remove missing values and unnecessary columns.
# Drop rows with missing values df_cleaned = df.dropna() # Remove the 'imdbId' column as it is not needed for this analysis df_cleaned = df_cleaned.drop(columns=['imdbId']) # Save the cleaned data to a new CSV file df_cleaned.to_csv('./data/data-cleaned.csv', index=False)
Loading the Cleaned Dataset
# Load the cleaned dataset data = pd.read_csv('./data/data-cleaned.csv') # Display the first few rows of the cleaned data data.head()
Defining Fuzzy Variables
Input Variables
- IMDb Rating (
imdb_rating): Represents the movie's average rating on IMDb, ranging from 0.0 to 10.0. - Release Year (
release_year): The year the movie was released, ranging from 1900 to 2025. - Number of Votes (
num_votes): The total number of votes received on IMDb, ranging from 0 to 3,000,000.
Output Variable
- Movie Quality (
movie_quality): Represents the qualitative assessment of the movie, ranging from 0 to 10.
Defining the Universes of Discourse
# Define the universal variables for the fuzzy sets imdb_rating = ctrl.Antecedent(np.arange(0, 10.1, 0.1), 'imdb_rating') release_year = ctrl.Antecedent(np.arange(1900, 2026, 1), 'release_year') num_votes = ctrl.Antecedent(np.arange(0, 3000001, 10000), 'num_votes') movie_quality = ctrl.Consequent(np.arange(0, 11, 1), 'movie_quality')
Membership Functions
IMDb Rating Membership Functions
We define the membership functions for IMDb Rating as Low, Medium, and High.
# IMDb Rating Membership Functions imdb_rating['low'] = fuzz.trimf(imdb_rating.universe, [0, 0, 5]) imdb_rating['medium'] = fuzz.trimf(imdb_rating.universe, [4, 5.5, 7]) imdb_rating['high'] = fuzz.trimf(imdb_rating.universe, [6, 10, 10])Release Year Membership Functions
The membership functions for Release Year are defined as Classic and Modern.
# Release Year Membership Functions release_year['classic'] = fuzz.trapmf(release_year.universe, [1900, 1900, 1970, 1990]) release_year['modern'] = fuzz.trapmf(release_year.universe, [1985, 2000, 2010, 2025])Number of Votes Membership Functions
The Number of Votes is categorized into Few, Average, and Many.
# Number of Votes Membership Functions num_votes['few'] = fuzz.trimf(num_votes.universe, [0, 0, 100000]) num_votes['average'] = fuzz.trimf(num_votes.universe, [50000, 200000, 500000]) num_votes['many'] = fuzz.trimf(num_votes.universe, [400000, 3000000, 3000000])Movie Quality Membership Functions
The output variable Movie Quality also uses Low, Medium, and High membership functions.
# Movie Quality Membership Functions movie_quality['low'] = fuzz.trimf(movie_quality.universe, [0, 0, 5]) movie_quality['medium'] = fuzz.trimf(movie_quality.universe, [4, 5.5, 7]) movie_quality['high'] = fuzz.trimf(movie_quality.universe, [6, 10, 10])
Visualizing Membership Functions
Visual representations help in understanding how input values map to the linguistic terms.
IMDb Rating Membership Functions
# Visualize IMDb Rating Membership Functions imdb_rating.view() plt.show()Release Year Membership Functions
# Visualize Release Year Membership Functions release_year.view() plt.show()Number of Votes Membership Functions
# Visualize Number of Votes Membership Functions num_votes.view() plt.show()Movie Quality Membership Functions
# Visualize Movie Quality Membership Functions movie_quality.view() plt.show()
Defining Fuzzy Rules
We establish a set of rules that encapsulate the logic for determining the movie quality based on the input variables.
# Rule Definitions rule1 = ctrl.Rule(imdb_rating['high'] & num_votes['many'], movie_quality['high']) rule2 = ctrl.Rule(imdb_rating['medium'] & num_votes['average'], movie_quality['medium']) rule3 = ctrl.Rule(imdb_rating['low'] & num_votes['few'], movie_quality['low']) rule4 = ctrl.Rule(imdb_rating['high'] & release_year['classic'], movie_quality['high']) rule5 = ctrl.Rule(imdb_rating['medium'] & release_year['modern'], movie_quality['medium']) rule6 = ctrl.Rule(imdb_rating['low'] | num_votes['few'], movie_quality['low'])Explanation of the Rules:
- Rule 1: High-rated movies with many votes are of high quality.
- Rule 2: Medium-rated movies with an average number of votes are of medium quality.
- Rule 3: Low-rated movies with few votes are of low quality.
- Rule 4: Classic movies with high ratings are of high quality.
- Rule 5: Modern movies with medium ratings are of medium quality.
- Rule 6: Movies with low ratings or few votes are of low quality.
Building the Control System
We create the fuzzy control system and simulation environment.
# Building the Control System movie_quality_ctrl = ctrl.ControlSystem([rule1, rule2, rule3, rule4, rule5, rule6]) movie_quality_simulation = ctrl.ControlSystemSimulation(movie_quality_ctrl)
Applying the Fuzzy Model to the Dataset
Defining the Computation Function
We define a function to compute the movie quality score for each movie in the dataset.
# Function to Compute Movie Quality Score def compute_movie_quality(row): # Set input values movie_quality_simulation.input['imdb_rating'] = row['imdbAverageRating'] movie_quality_simulation.input['release_year'] = row['releaseYear'] movie_quality_simulation.input['num_votes'] = row['imdbNumVotes'] # Compute the fuzzy result try: movie_quality_simulation.compute() return movie_quality_simulation.output['movie_quality'] except: # Handle exceptions, possibly due to input values outside the universe of discourse return np.nanComputing Movie Quality Scores
We apply the function to each row in the dataset.
# Apply the function to the dataset data['movie_quality_score'] = data.apply(compute_movie_quality, axis=1) # Drop rows where computation failed data = data.dropna(subset=['movie_quality_score'])
Analyzing the Results
Top Movies by Quality Score
We inspect the movies with the highest quality scores.
# Display the top 10 movies with the highest quality scores top_movies = data.sort_values(by='movie_quality_score', ascending=False).head(10) top_movies[['title', 'imdbAverageRating', 'imdbNumVotes', 'releaseYear', 'movie_quality_score']]Interpretation:
The top movies are expected to have high IMDb ratings, a large number of votes, and may be classics.
Distribution of Movie Quality Scores
We visualize the distribution of the computed movie quality scores.
# Plot the distribution of movie quality scores plt.figure(figsize=(10,6)) plt.hist(data['movie_quality_score'], bins=20, color='skyblue', edgecolor='black') plt.title('Distribution of Movie Quality Scores') plt.xlabel('Movie Quality Score') plt.ylabel('Number of Movies') plt.show()Observation:
The histogram shows how the movies are distributed across the quality scores, indicating the prevalence of high, medium, or low-quality movies in the dataset.
Conclusion
In this analysis, we employed fuzzy logic to evaluate movies based on IMDb ratings, release year, and the number of votes. The fuzzy logic approach allowed us to model subjective perceptions of movie quality and handle uncertainties inherent in human judgments. The resulting model provides a nuanced classification that can aid in recommendation systems or market analysis.
Key Takeaways:
- Fuzzy logic is effective for modeling complex decision-making processes where inputs are imprecise.
- The methodology can be extended to include more features or refined rules for better accuracy.
- Visualization of membership functions and results aids in interpreting and validating the model.
Future Work
- Feature Expansion: Incorporate additional features such as genre, director, and awards to enhance the model.
- Adaptive Membership Functions: Use data-driven approaches to optimize membership functions.
- Validation: Compare the fuzzy logic model's output with user reviews or other quality metrics for validation.
This notebook provides a comprehensive approach to using fuzzy logic for movie data analysis, demonstrating the practicality and flexibility of fuzzy systems in handling real-world data.