Skip to content
TheChatArchiveThe Chat Archive
Unspecified model (huggingface.xxx)Hugging Face ShareLM (browser-extension contributions)

Sort the heatmap by hierarchical clustering and use column names as ticks: import pandas…

  1. User

    Sort the heatmap by hierarchical clustering and use column names as ticks:

    import pandas as pd import numpy as np import seaborn as sns import matplotlib.pyplot as plt

    Assuming 'df' is your DataFrame with rows as papers and columns as labels

    Convert the DataFrame to binary (0/1) values

    df_binary = df.applymap(lambda x: 1 if x else 0)

    Calculate the co-occurrence matrix

    co_occurrence_matrix = np.dot(df_binary.T, df_binary)

    Create a heatmap

    plt.figure(figsize=(10, 8)) sns.heatmap(co_occurrence_matrix, annot=True, cmap='Blues', square=True) plt.title('Topic Co-occurrence Matrix') plt.xlabel('Topics') plt.ylabel('Topics') plt.xticks(rotation=90) plt.show()

  2. Assistant

    Here is the modified code that sorts the heatmap by hierarchical clustering and uses column names as ticks: import pandas as pd import numpy as np import seaborn as sns import matplotlib.pyplot as plt

    Assuming 'df' is your DataFrame with rows as papers and columns as labels

    Convert the DataFrame to binary (0/1) values

    df_binary = df.applymap(lambda x: 1 if x else 0)

    Calculate the co-occurrence matrix

    co_occurrence_matrix = np.dot(df_binary.T, df_binary)

    Create a heatmap with hierarchical clustering

    plt.figure(figsize=(10, 8)) sns.clustermap(co_occurrence_matrix, annot=True, cmap='Blues', square=True, xticklabels=df.columns, yticklabels=df.columns)

    plt.title('Topic Co-occurrence Matrix') plt.xlabel('Topics') plt.ylabel('Topics') plt.xticks(rotation=90) plt.show() CopiedIn this code, we replaced sns.heatmap with sns.clustermap, which performs hierarchical clustering on the rows and columns of the matrix before creating the heatmap. We also added the xticklabels and yticklabels parameters to sns.clustermap to use the column names as ticks. Note that this will cluster the rows and columns simultaneously, which may not always be desirable. If you want to cluster only the columns, you can use the cluster parameter and set it to True for columns and False for rows: sns.clustermap(co_occurrence_matrix, annot=True, cmap='Blues', square=True, xticklabels=df.columns, yticklabels=df.columns, cluster='col', figsize=(10, 8)) CopiedThis will cluster the columns only. You can also cluster the rows only by setting cluster='row'.

  3. User

    good, now create a new function to plot UMAP dimensionality reduction of the df

  4. Assistant

    Here is an example of a function that uses UMAP to reduce the dimensionality of the df DataFrame and plots the resulting 2D projection: import umap import matplotlib.pyplot as plt

    def plot_umap(df, n_neighbors=15, min_dist=0.1, random_state=42): """ Plot UMAP dimensionality reduction of the input DataFrame.

    Parameters:
    - df (pd.DataFrame): Input DataFrame
    - n_neighbors (int, optional): Number of neighbors to consider for each point. Defaults to 15.
    - min_dist (float, optional): Minimum distance between points in the embedding. Defaults to 0.1.
    - random_state (int, optional): Random state for reproducibility. Defaults to 42.
    """
    # Convert the DataFrame to binary (0/1) values
    df_binary = df.applymap(lambda x: 1 if x else 0)
    
    # Create a UMAP object
    reducer = umap.UMAP(n_neighbors=n_neighbors, min_dist=min_dist, random_state=random_state)
    
    # Fit the UMAP to the data and transform it
    embedding = reducer.fit_transform(df_binary)
    
    # Plot the UMAP
    plt.figure(figsize=(8, 8))
    plt.scatter(embedding[:, 0], embedding[:, 1], s=10, alpha=0.5)
    plt.title('UMAP Dimensionality Reduction')
    plt.xlabel('UMAP Dimension 1')
    plt.ylabel('UMAP Dimension 2')
    plt.show()   CopiedThis function takes the following parameters:
    

    df: the input DataFrame n_neighbors: the number of neighbors to consider for each point in the UMAP (default: 15) min_dist: the minimum distance between points in the embedding (default: 0.1) random_state: the random state for reproducibility (default: 42)

    The function first converts the input DataFrame to binary values using the same approach as before. Then, it creates a UMAP object with the specified parameters and fits it to the data using the fit_transform method. The resulting 2D embedding is then plotted using matplotlib. You can call this function by passing in your df DataFrame, like this: plot_umap(df) CopiedYou can also customize the plot by passing in additional parameters, such as changing the number of neighbors or the minimum distance. For example: plot_umap(df, n_neighbors=50, min_dist=0.05) Copied

  5. User

    Color all points by the binary value in the column "Evaluation"

  6. Assistant

    Here's an updated version of the plot_umap function that colors the points by the binary value in the "Evaluation" column: import umap import matplotlib.pyplot as plt

    def plot_umap(df, n_neighbors=15, min_dist=0.1, random_state=42): """ Plot UMAP dimensionality reduction of the input DataFrame.

    Parameters:
    - df (pd.DataFrame): Input DataFrame
    - n_neighbors (int, optional): Number of neighbors to consider for each point. Defaults to 15.
    - min_dist (float, optional): Minimum distance between points in the embedding. Defaults to 0.1.
    - random_state (int, optional): Random state for reproducibility. Defaults to 42.
    """
    # Convert the DataFrame to binary (0/1) values
    df_binary = df.applymap(lambda x: 1 if x else 0)
    
    # Create a UMAP object
    reducer = umap.UMAP(n_neighbors=n_neighbors, min_dist=min_dist, random_state=random_state)
    
    # Fit the UMAP to the data and transform it
    embedding = reducer.fit_transform(df_binary)
    
    # Plot the UMAP
    plt.figure(figsize=(8, 8))
    plt.scatter(embedding[:, 0], embedding[:, 1], c=df['Evaluation'], cmap='coolwarm', s=10, alpha=0.5)
    plt.title('UMAP Dimensionality Reduction')
    plt.xlabel('UMAP Dimension 1')
    plt.ylabel('UMAP Dimension 2')
    plt.colorbar(label='Evaluation')
    plt.show()   CopiedIn this updated version, we use the c parameter of the scatter function to color the points according to the binary value in the "Evaluation" column. We also use the cmap parameter to specify a color map (in this case, 'coolwarm', which maps 0 to blue and 1 to red). Finally, we add a color bar to the plot using the colorbar function.
    

    Note that this assumes that the "Evaluation" column is present in the original DataFrame df and contains binary values (0 or 1). If this is not the case, you may need to adjust the code accordingly.

  7. User

    Use TSNE instead

  8. Assistant

    Here is an updated version of the plot_umap function that uses t-SNE instead of UMAP: import matplotlib.pyplot as plt from sklearn.manifold import TSNE

    def plot_tsne(df, n_components=2, perplexity=30, random_state=42): """ Plot t-SNE dimensionality reduction of the input DataFrame.

    Parameters:
    - df (pd.DataFrame): Input DataFrame
    - n_components (int, optional): Number of dimensions for the embedding. Defaults to 2.
    - perplexity (int, optional): Perplexity for t-SNE. Defaults to 30.
    - random_state (int, optional): Random state for reproducibility. Defaults to 42.
    """
    # Convert the DataFrame to binary (0/1) values
    df_binary = df.applymap(lambda x: 1 if x else 0)
    
    # Create a t-SNE object
    tsne = TSNE(n_components=n_components, perplexity=perplexity, random_state=random_state)
    
    # Fit the t-SNE to the data and transform it
    embedding = tsne.fit_transform(df_binary)
    
    # Plot the t-SNE
    plt.figure(figsize=(8, 8))
    plt.scatter(embedding[:, 0], embedding[:, 1], c=df['Evaluation'], cmap='coolwarm', s=10, alpha=0.5)
    plt.title('t-SNE Dimensionality Reduction')
    plt.xlabel('t-SNE Dimension 1')
    plt.ylabel('t-SNE Dimension 2')
    plt.colorbar(label='Evaluation')
    plt.show()   CopiedThis function is very similar to the previous one, but it uses the TSNE class from scikit-learn instead of the umap library. The main differences are:
    

    The n_components parameter is used to specify the number of dimensions for the embedding, instead of n_neighbors and min_dist. The perplexity parameter is used to control the perplexity of the t-SNE, which affects the clustering of the points in the embedding. The fit_transform method is used to fit the t-SNE to the data and transform it, just like in the UMAP case.

    You can call this function in the same way as the previous one, passing in your df DataFrame and any desired parameters: plot_tsne(df) CopiedOr: plot_tsne(df, n_components=3, perplexity=50) Copied