Skip to content
TheChatArchiveThe Chat Archive
GeminiGemini (model unspecified)· GoogleHugging Face ShareLM (browser-extension contributions)

DEBUG Traceback (most recent call last): File…

  1. User

    DEBUG Traceback (most recent call last):   File "/home/esther/PycharmProjects/edut710/pheonix_backup/edut710/holocaust_references/running_on_ivritai_trans/preprocess.py", line 152, in     segmented_df = preprocess_text_segments(data, max_words=500)   File "/home/esther/PycharmProjects/edut710/pheonix_backup/edut710/holocaust_references/running_on_ivritai_trans/preprocess.py", line 52, in preprocess_text_segments     nlp = spacy.load("he_ner_news_trf", disable=["tagger", "parser", "ner"])  # Requires GPU for best performance   File "/XXX.10/site-packages/spacy/init.py", line 51, in load     return util.load_model(   File "/XXX.10/site-packages/spacy/util.py", line 420, in load_model     return load_model_from_package(name, **kwargs)  # type: ignore[arg-type]   File "/XXX.10/site-packages/spacy/util.py", line 453, in load_model_from_package     return cls.load(vocab=vocab, disable=disable, exclude=exclude, config=config)  # type: ignore[attr-defined]   File "/XXX.10/site-packages/he_ner_news_trf/init.py", line 10, in load     return load_model_from_init_py(file, **overrides)   File "/XXX.10/site-packages/spacy/util.py", line 615, in load_model_from_init_py     return load_model_from_path(   File "/XXX.10/site-packages/spacy/util.py", line 489, in load_model_from_path     return nlp.from_disk(model_path, exclude=exclude, overrides=overrides)   File "/XXX.10/site-packages/spacy/language.py", line 2042, in from_disk     util.from_disk(path, deserializers, exclude)  # type: ignore[arg-type]   File "/XXX.10/site-packages/spacy/util.py", line 1299, in from_disk     reader(path / key)   File "/XXX.10/site-packages/spacy/language.py", line 2036, in     deserializers[name] = lambda p, proc=proc: proc.from_disk(  # type: ignore[misc]   File "/XXX.10/site-packages/spacy_transformers/pipeline_component.py", line 420, in from_disk     util.from_disk(path, deserialize, exclude)   File "/XXX.10/site-packages/spacy/util.py", line 1299, in from_disk     reader(path / key)   File "/XXX.10/site-packages/spacy_transformers/pipeline_component.py", line 394, in load_model     self.model.from_bytes(mfile.read())   File "/XXX.10/site-packages/thinc/model.py", line 594, in from_bytes     return self.from_dict(msg)   File "/XXX.10/site-packages/thinc/model.py", line 632, in from_dict     node.shims[i].from_bytes(shim_bytes)   File "/XXX.10/site-packages/spacy_transformers/layers/hf_shim.py", line 123, in from_bytes     self._model.load_state_dict(torch.load(filelike, map_location=map_location))   File "/XXX.10/site-packages/torch/nn/modules/module.py", line 2584, in load_state_dict     raise RuntimeError( RuntimeError: Error(s) in loading state_dict for BertModel: Unexpected key(s) in state_dict: "embeddings.position_ids".

  2. Gemini Assistant

    This RuntimeError is a classic version compatibility issue between the Hugging Face transformers library and the saved weights inside the he_ner_news_trf model.The ProblemThe error Unexpected key(s) in state_dict: "embeddings.position_ids" occurs because the saved model file (which likely uses an older BERT architecture) contains position_ids, but the version of the transformers library you are currently using does not expect them (it registers them differently in newer versions).Essentially, your transformers library is likely too new for the specific he_ner_news_trf model version you are trying to load.Solution 1: Downgrade transformers (Most Likely Fix)The most direct fix is to install a slightly older version of the transformers library that is compatible with the saved weights. Versions below 4.30 are usually safe for these types of errors.Run this in your terminal (make sure your virtual environment venv is active):Bashpip install "transformers<4.30" After the installation finishes, try running your script again.Solution 2: Update spacy-transformers and the ModelIf you prefer not to downgrade, you can try ensuring the bridge between spaCy and Hugging Face is fully up to date, which may include patches for this issue.Upgrade the libraries:Bashpip install -U spacy spacy-transformers Re-download the Hebrew model (this ensures you have the latest wrapper definitions):Bashpython -m spacy download he_ner_news_trf Summary of what to checkIf the above fixes don't work immediately, please verify your versions. You can run this command to see what is currently installed:Bashpip show spacy spacy-transformers transformers Target (Working) Configuration:spacy: >= 3.0spacy-transformers: >= 1.2transformers: ~4.28.1 (This is a very stable version for spaCy 3.x models)Next StepWould you like me to help you create a requirements.txt file to lock these versions so this error doesn't happen again in the future?