~/projects/BitNet-1.58-Instruct/bitnet$ CUDAVISIBLEDEVICES=0 python3 ../scripts/train.py…
- User
~/projects/BitNet-1.58-Instruct/bitnet$ CUDA_VISIBLE_DEVICES=0 python3 ../scripts/train.py -d ../train.jsonl -o ../outbitnet64vocab -m /home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab Loading model /home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab... Traceback (most recent call last): File "/home/user/projects/BitNet-1.58-Instruct/bitnet/../scripts/train.py", line 65, in main() File "/home/user/projects/BitNet-1.58-Instruct/bitnet/../scripts/train.py", line 33, in main model = BitNetLLM(args.model) ^^^^^^^^^^^^^^^^^^^^^ File "/home/user/projects/BitNet-1.58-Instruct/bitnet/models/bitnet/bitnet.py", line 12, in init super().init() File "/home/user/projects/BitNet-1.58-Instruct/bitnet/models/model.py", line 6, in init self._build() File "/home/user/projects/BitNet-1.58-Instruct/bitnet/models/bitnet/bitnet.py", line 16, in _build self.tokenizer = LlamaTokenizer.from_pretrained(self.model_name, trust_remote_code=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2255, in from_pretrained raise EnvironmentError( OSError: Can't load tokenizer for '/home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab' is the correct path to a directory containing all relevant files for a LlamaTokenizer tokenizer. how to make my tokenizer 64, model initalized and config.json both at 64, it should only use alphebet = my guess
- Assistant
To create a tokenizer with a vocabulary size of 64 and ensure it's properly loaded by your model, you'll need to follow these steps:
-
Create a Custom Vocabulary:
- Decide on the 64 tokens you want in your vocabulary. Since you mentioned it should only use the alphabet, you might include all lowercase letters, uppercase letters, digits, and some punctuation to reach 64 tokens.
- Example tokens could be:
['a', 'b', 'c', ..., 'A', 'B', 'C', ..., '0', '1', ..., '!', '?', '.', ',', ' ']
-
Build a Custom Tokenizer:
- Use the Hugging Face
tokenizerslibrary to create a tokenizer from scratch with your custom vocabulary. - Here's an example of how to create a
WordLeveltokenizer:
- Use the Hugging Face
from tokenizers import Tokenizer from tokenizers.models import WordLevel from tokenizers.pre_tokenizers import Whitespace from tokenizers.trainers import WordLevelTrainer # Define your custom vocabulary vocab = ['a', 'b', 'c', ..., 'A', 'B', 'C', ..., '0', '1', ..., '!', '?', '.', ',', ' '] # Initialize a tokenizer with the WordLevel model tokenizer = Tokenizer(WordLevel(unk_token="[UNK]")) # Set pre-tokenizer tokenizer.pre_tokenizer = Whitespace() # Set vocabulary to the tokenizer tokenizer.model = WordLevel(vocab={token: idx for idx, token in enumerate(vocab)}, unk_token="[UNK]") # Save the tokenizer tokenizer.save("path_to_save_tokenizer/tokenizer.json")-
Save the Tokenizer in the Model Directory:
- Save the tokenizer files (
tokenizer.json,tokenizer_config.json, etc.) in the directory/home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab. - Ensure that all necessary tokenizer files are present in that directory.
- Save the tokenizer files (
-
Modify the Model Code to Load the Custom Tokenizer:
- Since you're using
LlamaTokenizer, which expects specific tokenizer files (liketokenizer.modelfor SentencePiece models), you might need to adjust your approach. - If you stick with
LlamaTokenizer, you can use thetokenizer_classargument to specify a different tokenizer class or adjust the expected files.
- Since you're using
-
Example of Loading a Custom Tokenizer:
from transformers import PreTrainedTokenizerFast # Load your custom tokenizer tokenizer = PreTrainedTokenizerFast(tokenizer_file="path_to_save_tokenizer/tokenizer.json", unk_token="[UNK]") # Save tokenizer config if necessary tokenizer.save_pretrained("path_to_save_tokenizer") # Now, when loading the tokenizer in your model code: tokenizer = PreTrainedTokenizerFast.from_pretrained("path_to_save_tokenizer")- Adjust Your Model Initialization:
- Make sure your model's configuration (
config.json) reflects the vocabulary size of 64. - When initializing the model, ensure that it uses the correct vocabulary size:
- Make sure your model's configuration (
from transformers import AutoConfig, AutoModel config = AutoConfig.from_pretrained("path_to_model_directory", vocab_size=64) model = AutoModel.from_config(config)-
Ensure Consistency Between Tokenizer and Model:
- The tokenizer's vocabulary size must match the
vocab_sizeparameter in your model's configuration. - Verify that all tokens in your tokenizer are correctly mapped and that the model's embeddings are initialized accordingly.
- The tokenizer's vocabulary size must match the
-
Additional Tips:
- If using a custom tokenizer not compatible with
LlamaTokenizer, avoid usingLlamaTokenizer.from_pretrainedand instead usePreTrainedTokenizerFastor another suitable tokenizer class. - Check that the tokenizer files are correctly named and placed in the directory so that the
from_pretrainedmethod can find and load them.
- If using a custom tokenizer not compatible with
-
Debugging:
- If you continue to encounter errors, double-check the contents of
/home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab. Ensure it contains all necessary tokenizer files. - Use verbose logging or print statements to verify that the paths and files are correct.
- If you continue to encounter errors, double-check the contents of
Summary:
To resolve the error, you need to create a custom tokenizer with your desired vocabulary of 64 tokens, save it in the appropriate format and directory, and ensure your model code loads this tokenizer correctly. Adjust your model's configuration to match the tokenizer's vocabulary size, and ensure consistency throughout your code.
Example Resources:
- Building a tokenizer from scratch (Hugging Face documentation)
- Saving and loading tokenizers (Hugging Face documentation)
By following these steps, you should be able to create a tokenizer with a vocabulary size of 64 that matches your model's expectations, and successfully load it without encountering the
OSError. -