i have a df called: closedinq. i am embedding a column and saving the embedding as a…
- User
i have a df called: closed_inq. i am embedding a column and saving the embedding as a pickle file and reloading the file every run. i only want to run new rows to embed. how do i return closed_inq with len(embedding) only keeping new rows
- Assistant
To only embed new rows in the dataframe
closed_inqand keep only those rows, you can follow these steps:- Load the existing embedded rows from the pickle file.
- Find the number of already embedded rows.
- Filter out the new rows from the
closed_inqdataframe. - Embed the new rows.
- Concatenate the embedded new rows with the existing embedded rows.
- Save the updated embedded rows as a pickle file.
- Return the complete dataframe with only the new rows.
Here's the code that accomplishes this:
import pickle # 1. Load existing embedded rows from the pickle file with open('embedding.pkl', 'rb') as f: existing_embedding = pickle.load(f) # 2. Find the number of already embedded rows existing_rows = len(existing_embedding) # 3. Filter out the new rows from the closed_inq dataframe new_rows = closed_inq.iloc[existing_rows:] # Assuming you have a function 'embed' that embeds the rows new_embedding = embed(new_rows) # 5. Concatenate the embedded new rows with the existing embedded rows updated_embedding = pd.concat([existing_embedding, new_embedding]) # 6. Save the updated embedded rows as a pickle file with open('embedding.pkl', 'wb') as f: pickle.dump(updated_embedding, f) # 7. Return the complete dataframe with only the new rows return closed_inq.iloc[:existing_rows+len(new_rows)]Make sure to replace
'embedding.pkl'with the appropriate file path for your pickle file and modify theembedfunction according to your specific embedding method. - User
TypeError Traceback (most recent call last) c:\Users\harold.noble\Documents\apps\i2ms-analytics\working folder\test copy.ipynb Cell 12 line 1 ----> 1 pd.concat([existing_embeddings, new_embeddings])
File ~\AppData\Roaming\Python\Python310\site-packages\pandas\util_decorators.py:331, in deprecate_nonkeyword_arguments..decorate..wrapper(*args, **kwargs) 325 if len(args) > num_allow_args: 326 warnings.warn( 327 msg.format(arguments=_format_argument_list(allow_args)), 328 FutureWarning, 329 stacklevel=find_stack_level(), 330 ) --> 331 return func(*args, **kwargs)
File ~\AppData\Roaming\Python\Python310\site-packages\pandas\core\reshape\concat.py:368, in concat(objs, axis, join, ignore_index, keys, levels, names, verify_integrity, sort, copy) 146 @deprecate_nonkeyword_arguments(version=None, allowed_args=["objs"]) 147 def concat( 148 objs: Iterable[NDFrame] | Mapping[HashableT, NDFrame], (...) 157 copy: bool = True, 158 ) -> DataFrame | Series: 159 """ 160 Concatenate pandas objects along a particular axis. 161 (...) ... 462 # get the sample 463 # want the highest ndim that we have, and must be non-empty 464 # unless all objs are empty
TypeError: cannot concatenate object of type '<class 'numpy.ndarray'>'; only Series and DataFrame objs are valid
- Assistant
Apologies for the confusion. It seems that the variable names used in the example code were inconsistent. Here's an updated version that should resolve the issue:
import pickle # 1. Load existing embedded rows from the pickle file with open('embedding.pkl', 'rb') as f: existing_embedding = pickle.load(f) # 2. Find the number of already embedded rows existing_rows = len(existing_embedding) # 3. Filter out the new rows from the closed_inq dataframe new_rows = closed_inq.iloc[existing_rows:] # Assuming you have a function `embed` that embeds the rows new_embedding = embed(new_rows) # 5. Concatenate the embedded new rows with the existing embedded rows updated_embedding = np.concatenate((existing_embedding, new_embedding)) # 6. Save the updated embedded rows as a pickle file with open('embedding.pkl', 'wb') as f: pickle.dump(updated_embedding, f) # 7. Return the complete dataframe with only the new rows return closed_inq.iloc[:existing_rows+len(new_rows)]In this updated code, we use
np.concatenateto combine the existing and new embeddings instead ofpd.concat. Make sure that you have the necessary imports fornumpyand that you replace'embedding.pkl'with your actual file path.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.