Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0613· openaiHugging Face Suzhen/CodeChat

this code is working: # Check if the file with embeddings exists and load it try: with…

  1. User

    this code is working: # Check if the file with embeddings exists and load it try: with open("encoded_embeddings.pkl", "rb") as f: existing_embeddings = pickle.load(f) except FileNotFoundError: existing_embeddings = ''

    Find the number of already embedded rows

    existing_rows = len(existing_embeddings)

    Filter out the new rows from the closed_inq dataframe

    new_rows = closed_inq.iloc[existing_rows:]

    Encode new rows if needed

    if len(new_rows) > 0: new_embeddings = encoder.encode(closed_inq.iloc[:, 1].tolist(), show_progress_bar=True, device=torch_device,) existing_embeddings = np.concatenate((existing_embeddings, new_embeddings))

    Save the encoded embeddings to a file

    with open("encoded_embeddings.pkl", "wb") as f: pickle.dump(new_emeddings, f)

    closed_inq is a df that also has a column called inquiry_id in place zero. is it better to include those in the pickle file?

  2. Assistant

    It depends on your specific requirements and use case. Including the inquiry_id column in the pickle file can be useful if you need to associate the embeddings with their corresponding inquiry IDs later on.

    If you plan to use the embeddings for analysis or further processing where the inquiry IDs are not necessary, then it may not be necessary to include them in the pickle file. This can help reduce the file size and improve efficiency.

    Consider your future needs and use cases to determine whether including the inquiry_id column in the pickle file is necessary or not.

  3. User

    how would i add the inquiry_id for storage

  4. Assistant

    To include the inquiry_id column in the pickle file, you can modify the code as follows:

    try:
        with open("encoded_embeddings.pkl", "rb") as f:
            existing_embeddings = pickle.load(f)
    except FileNotFoundError:
        existing_embeddings = pd.DataFrame()
    
    # Find the number of already embedded rows
    existing_rows = len(existing_embeddings)
    
    # Filter out the new rows from the closed_inq dataframe
    new_rows = closed_inq.iloc[existing_rows:]
    
    # Encode new rows if needed
    if len(new_rows) > 0:
        new_embeddings = encoder.encode(closed_inq.iloc[:, 1].tolist(), show_progress_bar=True, device=torch_device)
        new_inquiry_ids = closed_inq.iloc[:, 0].tolist()  # Get the inquiry_id column values
        existing_embeddings = existing_embeddings.append(pd.DataFrame({"inquiry_id": new_inquiry_ids, "embeddings": new_embeddings}))
    
    # Save the encoded embeddings to a file
    with open("encoded_embeddings.pkl", "wb") as f:
        pickle.dump(existing_embeddings, f)
    

    In this modified code, a new DataFrame existing_embeddings is created to store both the inquiry_id and embeddings values. The existing_embeddings DataFrame is updated by appending the newly encoded rows along with their corresponding inquiry_id values.

    When saving the embeddings to the pickle file, the entire DataFrame existing_embeddings is stored, including both the inquiry_id and embeddings columns.

  5. User

    ValueError Traceback (most recent call last) c:\Users\harold.noble\Documents\apps\i2ms-analytics\working folder\test copy.ipynb Cell 12 line 2 1 new_inquiry_ids = closed_inq.iloc[:, 0].tolist() ----> 2 pd.DataFrame({"inquiry_id": new_inquiry_ids, "embeddings": new_embeddings})

    File ~\AppData\Roaming\Python\Python310\site-packages\pandas\core\frame.py:664, in DataFrame.init(self, data, index, columns, dtype, copy) 658 mgr = self._init_mgr( 659 data, axes={"index": index, "columns": columns}, dtype=dtype, copy=copy 660 ) 662 elif isinstance(data, dict): 663 # GH#38939 de facto copy defaults to False only in non-dict cases --> 664 mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager) 665 elif isinstance(data, ma.MaskedArray): 666 import numpy.ma.mrecords as mrecords

    File ~\AppData\Roaming\Python\Python310\site-packages\pandas\core\internals\construction.py:493, in dict_to_mgr(data, index, columns, dtype, typ, copy) 489 else: 490 # dtype check to exclude e.g. range objects, scalars 491 arrays = [x.copy() if hasattr(x, "dtype") else x for x in arrays] --> 493 return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)

    File ~\AppData\Roaming\Python\Python310\site-packages\pandas\core\internals\construction.py:118, in arrays_to_mgr(arrays, columns, index, dtype, verify_integrity, typ, consolidate) 115 if verify_integrity: 116 # figure out the index, if necessary ... --> 653 raise ValueError("Per-column arrays must each be 1-dimensional") 655 if not indexes and not raw_lengths: 656 raise ValueError("If using all scalar values, you must pass an index")

    ValueError: Per-column arrays must each be 1-dimensional

  6. Assistant

    Apologies for the confusion. It seems like the new_embeddings variable is a multi-dimensional array, which is causing the error when creating a DataFrame.

    To fix this issue, you can first create a DataFrame with just the inquiry_id column and then concatenate it with the new_embeddings array. Here's an updated version of the code:

    try:
        with open("encoded_embeddings.pkl", "rb") as f:
            existing_embeddings = pickle.load(f)
    except FileNotFoundError:
        existing_embeddings = pd.DataFrame()
    
    # Find the number of already embedded rows
    existing_rows = len(existing_embeddings)
    
    # Filter out the new rows from the closed_inq dataframe
    new_rows = closed_inq.iloc[existing_rows:]
    
    # Encode new rows if needed
    if len(new_rows) > 0:
        new_embeddings = encoder.encode(closed_inq.iloc[:, 1].tolist(), show_progress_bar=True, device=torch_device)
        new_inquiry_ids = closed_inq.iloc[:, 0].tolist()  # Get the inquiry_id column values
        new_rows_embeddings = pd.DataFrame({"inquiry_id": new_inquiry_ids})
        new_rows_embeddings["embeddings"] = new_embeddings.tolist()
        existing_embeddings = pd.concat([existing_embeddings, new_rows_embeddings], ignore_index=True)
    
    # Save the encoded embeddings to a file
    with open("encoded_embeddings.pkl", "wb") as f:
        pickle.dump(existing_embeddings, f)
    

    In this updated code, we create a separate DataFrame new_rows_embeddings by concatenating the inquiry_id values with the transformed new_embeddings array, converted to a list. Then, we use pd.concat to combine the existing_embeddings DataFrame and new_rows_embeddings.

    Now, the existing_embeddings DataFrame will include both the inquiry_id and embeddings columns, and it can be stored in the pickle file.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.