Skip to content
TheChatArchiveThe Chat Archive
o1-preview-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

~/projects/BitNet-1.58-Instruct/bitnet$ CUDAVISIBLEDEVICES=0 python3 ../scripts/train.py…

  1. User

    ~/projects/BitNet-1.58-Instruct/bitnet$ CUDA_VISIBLE_DEVICES=0 python3 ../scripts/train.py -d ../train.jsonl -o ../outbitnet64vocab -m /home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab Loading model /home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab... Traceback (most recent call last): File "/home/user/projects/BitNet-1.58-Instruct/bitnet/../scripts/train.py", line 65, in main() File "/home/user/projects/BitNet-1.58-Instruct/bitnet/../scripts/train.py", line 33, in main model = BitNetLLM(args.model) ^^^^^^^^^^^^^^^^^^^^^ File "/home/user/projects/BitNet-1.58-Instruct/bitnet/models/bitnet/bitnet.py", line 12, in init super().init() File "/home/user/projects/BitNet-1.58-Instruct/bitnet/models/model.py", line 6, in init self._build() File "/home/user/projects/BitNet-1.58-Instruct/bitnet/models/bitnet/bitnet.py", line 16, in _build self.tokenizer = LlamaTokenizer.from_pretrained(self.model_name, trust_remote_code=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/text-generation-webui/installer_files/env/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2255, in from_pretrained raise EnvironmentError( OSError: Can't load tokenizer for '/home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab' is the correct path to a directory containing all relevant files for a LlamaTokenizer tokenizer. how to make my tokenizer 64, model initalized and config.json both at 64, it should only use alphebet = my guess

  2. Assistant

    To create a tokenizer with a vocabulary size of 64 and ensure it's properly loaded by your model, you'll need to follow these steps:

    1. Create a Custom Vocabulary:

      • Decide on the 64 tokens you want in your vocabulary. Since you mentioned it should only use the alphabet, you might include all lowercase letters, uppercase letters, digits, and some punctuation to reach 64 tokens.
      • Example tokens could be: ['a', 'b', 'c', ..., 'A', 'B', 'C', ..., '0', '1', ..., '!', '?', '.', ',', ' ']
    2. Build a Custom Tokenizer:

      • Use the Hugging Face tokenizers library to create a tokenizer from scratch with your custom vocabulary.
      • Here's an example of how to create a WordLevel tokenizer:
    from tokenizers import Tokenizer
    from tokenizers.models import WordLevel
    from tokenizers.pre_tokenizers import Whitespace
    from tokenizers.trainers import WordLevelTrainer
    
    # Define your custom vocabulary
    vocab = ['a', 'b', 'c', ..., 'A', 'B', 'C', ..., '0', '1', ..., '!', '?', '.', ',', ' ']
    
    # Initialize a tokenizer with the WordLevel model
    tokenizer = Tokenizer(WordLevel(unk_token="[UNK]"))
    
    # Set pre-tokenizer
    tokenizer.pre_tokenizer = Whitespace()
    
    # Set vocabulary to the tokenizer
    tokenizer.model = WordLevel(vocab={token: idx for idx, token in enumerate(vocab)}, unk_token="[UNK]")
    
    # Save the tokenizer
    tokenizer.save("path_to_save_tokenizer/tokenizer.json")
    
    1. Save the Tokenizer in the Model Directory:

      • Save the tokenizer files (tokenizer.json, tokenizer_config.json, etc.) in the directory /home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab.
      • Ensure that all necessary tokenizer files are present in that directory.
    2. Modify the Model Code to Load the Custom Tokenizer:

      • Since you're using LlamaTokenizer, which expects specific tokenizer files (like tokenizer.model for SentencePiece models), you might need to adjust your approach.
      • If you stick with LlamaTokenizer, you can use the tokenizer_class argument to specify a different tokenizer class or adjust the expected files.
    3. Example of Loading a Custom Tokenizer:

    from transformers import PreTrainedTokenizerFast
    
    # Load your custom tokenizer
    tokenizer = PreTrainedTokenizerFast(tokenizer_file="path_to_save_tokenizer/tokenizer.json", unk_token="[UNK]")
    
    # Save tokenizer config if necessary
    tokenizer.save_pretrained("path_to_save_tokenizer")
    
    # Now, when loading the tokenizer in your model code:
    tokenizer = PreTrainedTokenizerFast.from_pretrained("path_to_save_tokenizer")
    
    1. Adjust Your Model Initialization:
      • Make sure your model's configuration (config.json) reflects the vocabulary size of 64.
      • When initializing the model, ensure that it uses the correct vocabulary size:
    from transformers import AutoConfig, AutoModel
    
    config = AutoConfig.from_pretrained("path_to_model_directory", vocab_size=64)
    model = AutoModel.from_config(config)
    
    1. Ensure Consistency Between Tokenizer and Model:

      • The tokenizer's vocabulary size must match the vocab_size parameter in your model's configuration.
      • Verify that all tokens in your tokenizer are correctly mapped and that the model's embeddings are initialized accordingly.
    2. Additional Tips:

      • If using a custom tokenizer not compatible with LlamaTokenizer, avoid using LlamaTokenizer.from_pretrained and instead use PreTrainedTokenizerFast or another suitable tokenizer class.
      • Check that the tokenizer files are correctly named and placed in the directory so that the from_pretrained method can find and load them.
    3. Debugging:

      • If you continue to encounter errors, double-check the contents of /home/user/Storage/1bitLLM_bitnet_b1_58-large/64vocab. Ensure it contains all necessary tokenizer files.
      • Use verbose logging or print statements to verify that the paths and files are correct.

    Summary:

    To resolve the error, you need to create a custom tokenizer with your desired vocabulary of 64 tokens, save it in the appropriate format and directory, and ensure your model code loads this tokenizer correctly. Adjust your model's configuration to match the tokenizer's vocabulary size, and ensure consistency throughout your code.

    Example Resources:

    By following these steps, you should be able to create a tokenizer with a vocabulary size of 64 that matches your model's expectations, and successfully load it without encountering the OSError.