39 21040-bge-m3 super().init(torch.C.PyTorchFileReader(nameorbuffer)) 39 21040-bge-m3…
- User
39|21040-bge-m3 | super().init(torch._C.PyTorchFileReader(name_or_buffer)) 39|21040-bge-m3 | RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory 39|21040-bge-m3 | During handling of the above exception, another exception occurred: 39|21040-bge-m3 | Traceback (most recent call last): 39|21040-bge-m3 | File "/data/app/anaconda3/envs/bge-m3/lib/python3.9/site-packages/transformers/modeling_utils.py", line 523, in load_state_dict 39|21040-bge-m3 | if f.read(7) == "version": 39|21040-bge-m3 | File "/data/app/anaconda3/envs/bge-m3/lib/python3.9/codecs.py", line 322, in decode 39|21040-bge-m3 | (result, consumed) = self._buffer_decode(data, self.errors, final) 39|21040-bge-m3 | UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte 39|21040-bge-m3 | During handling of the above exception, another exception occurred: 39|21040-bge-m3 | Traceback (most recent call last): 39|21040-bge-m3 | File "/data/app/ai_install/unify-openai-embeddding-api.py", line 24, in 39|21040-bge-m3 | model = BGEM3FlagModel('/data/app/ai/models/bge-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation 39|21040-bge-m3 | File "/data/app/ai/FlagEmbedding/FlagEmbedding/bge_m3.py", line 36, in init 39|21040-bge-m3 | self.model = BGEM3ForInference( 39|21040-bge-m3 | File "/data/app/ai/FlagEmbedding/FlagEmbedding/BGE_M3/modeling.py", line 40, in init 39|21040-bge-m3 | self.load_model(model_name, colbert_dim=colbert_dim) 39|21040-bge-m3 | File "/data/app/ai/FlagEmbedding/FlagEmbedding/BGE_M3/modeling.py", line 76, in load_model 39|21040-bge-m3 | self.model = AutoModel.from_pretrained(model_name) 39|21040-bge-m3 | File "/data/app/anaconda3/envs/bge-m3/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained 39|21040-bge-m3 | return model_class.from_pretrained( 39|21040-bge-m3 | File "/data/app/anaconda3/envs/bge-m3/lib/python3.9/site-packages/transformers/modeling_utils.py", line 3393, in from_pretrained 39|21040-bge-m3 | state_dict = load_state_dict(resolved_archive_file) 39|21040-bge-m3 | File "/data/app/anaconda3/envs/bge-m3/lib/python3.9/site-packages/transformers/modeling_utils.py", line 535, in load_state_dict 39|21040-bge-m3 | raise OSError( 39|21040-bge-m3 | OSError: Unable to load weights from pytorch checkpoint file for '/data/app/ai/models/bge-m3/pytorch_model.bin' at '/data/app/ai/models/bge-m3/pytorch_model.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
- Assistant
The error stack trace you're seeing suggests that there is an issue with loading a PyTorch model. Specifically, there is an issue with reading the model file that you specified (
/data/app/ai/models/bge-m3/pytorch_model.bin). Here are some potential causes and solutions for this problem:Potential Causes
- Corrupted Model File: The PyTorch model file you are trying to load might be corrupted.
- Incorrect File Path: The specified file path may be incorrect, or the file may not exist at the specified location.
- Wrong File Format: You might be trying to load a file that is not in the expected format (e.g., trying to load a TensorFlow checkpoint as a PyTorch model without proper conversion).
- Insufficient Permissions: There might be permission issues preventing the file from being accessed.
Steps to Diagnose and Resolve
-
Check the File Path: Ensure that the file indeed exists at the specified location and that the path is correct.
ls -l /data/app/ai/models/bge-m3/pytorch_model.bin -
Verify File Integrity: Ensure that the model file is not corrupted. One way is to re-download the model and ensure its checksum matches the expected value.
-
Permissions Check: Verify that the file permissions allow reading by the user running the script.
chmod 644 /data/app/ai/models/bge-m3/pytorch_model.bin -
Model Format: If the model was originally a TensorFlow checkpoint, ensure it has been correctly converted to PyTorch format or use the
from_tf=Trueflag if loading directly from a TensorFlow checkpoint.model = BGEM3ForInference.from_pretrained("/data/app/ai/models/bge-m3", from_tf=True) -
Test with a Small Example: Load a simple, known-good model from the transformers library to ensure your environment setup is correct.
from transformers import AutoModel model = AutoModel.from_pretrained("bert-base-uncased")If this works, your environment and the transformers library are likely set up correctly, pointing to an issue specific to your model file.
-
Verbose Error Logging: Enable more detailed logging to gather extra information about the loading process.
import logging logging.basicConfig(level=logging.DEBUG) from transformers import AutoModel model = AutoModel.from_pretrained("/data/app/ai/models/bge-m3")
Example Correction Code
Here is a modified snippet from your script to include logging and handle potential errors:
import logging from transformers import AutoModel # Enable detailed logging logging.basicConfig(level=logging.DEBUG) try: model = BGEM3ForInference.from_pretrained("/data/app/ai/models/bge-m3") except OSError as e: logging.error("Failed to load model", exc_info=True) raise # Optionally re-raise the error or handle it as needed except Exception as e: logging.error("An unexpected error occurred", exc_info=True) raise # Handle other exceptionsConclusion
By checking the file path, verifying the file integrity, ensuring proper permissions, and confirming the file format, you can diagnose and potentially fix the issue. Leveraging detailed logging can also provide more insights into what's going wrong.
- User
see the code below: @app.post("/embeddings/") async def create_embedding(item: Item): global model sentence = item.text embeddings = model.encode(sentence, batch_size=12, max_length=8192, # If you don't need such a long length, you can set a smaller value to speed up the encoding process. )['dense_vecs'] return {"embedding": embeddings.tolist()}
if name == "main": model = BGEM3FlagModel('/data/app/ai/models/bge-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation #uvicorn.run(app, host=args.server_name, port=args.server_port, workers=1) uvicorn.run(app, host="0.0.0.0", port=21040)
- Assistant
It looks like you are trying to run a FastAPI application that computes embeddings for a given text using a model instantiated from a custom
BGEM3FlagModelclass.However, it seems there was an issue related to loading the model file as indicated by the error messages you shared earlier.
To ensure that this part of the script works correctly, let's break down potential enhancements and checks that you can add to your code.
Improved Code with Error Handling
- Add Detailed Logging: This will help track where things may be going wrong.
- Error Handling in Model Load: Capture any errors when the model is being loaded and respond appropriately.
- Parameter Validation: Ensure input parameters are validated.
- Dependency Injection for Better Testing: While not required, it can make your code more modular.
Revised Code Example
from fastapi import FastAPI, HTTPException from pydantic import BaseModel import uvicorn import logging class Item(BaseModel): text: str app = FastAPI() logging.basicConfig(level=logging.DEBUG) # Ensure 'model' is available globally for the route model = None @app.post("/embeddings/") async def create_embedding(item: Item): global model sentence = item.text if model is None: raise HTTPException(status_code=500, detail="Model is not loaded") try: embeddings = model.encode( sentence, batch_size=12, max_length=8192 # Adjust as needed )['dense_vecs'] return {"embedding": embeddings.tolist()} except Exception as e: logging.error("Error during encoding", exc_info=True) raise HTTPException(status_code=500, detail="Internal server error while encoding") if __name__ == "__main__": try: model = BGEM3FlagModel('/data/app/ai/models/bge-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation except OSError as e: logging.error("Failed to load model", exc_info=True) raise except Exception as e: logging.error("Unexpected error while loading model", exc_info=True) raise # Run the server uvicorn.run(app, host="0.0.0.0", port=21040)Explanation
-
Detailed Logging:
logging.basicConfig(level=logging.DEBUG)initializes detailed logging.
-
Error Handling in Model Load:
- Wrapped the model loading in a try-except block to handle OSError and generic exceptions. This ensures any issues during the model initialization are logged, and the application exits gracefully if the model cannot be loaded.
-
Parameter Validation and Error Handling During Request:
- Added a check to raise an HTTP 500 error if the model is not loaded.
- Wrapped the model encoding inside a try-except block to log any encoding errors and return an HTTP 500 error.
-
Parameter Class:
- The
Itemclass derived fromBaseModelensures that the payload sent to the/embeddings/endpoint meets the expected format.
- The
Run the Server
Ensure the application is run with the correct permissions and paths, especially for accessing the model file. If you face any more issues with the file path or permissions, refer to the earlier suggestions about checking file existence, correctness, and permissions.
Testing
To test this script, you can start the server and then use tools like
curl, Postman, or a simple Python client to sendPOSTrequests tohttp://0.0.0.0:21040/embeddings/with a JSON payload like{"text": "your sentence here"}.curl -X POST "http://0.0.0.0:21040/embeddings/" -H "Content-Type: application/json" -d '{"text": "your sentence here"}'By adding these enhancements, your application should be more robust and easier to debug.