Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-2024-05-13· OpenAIAi2 WildChat-4.8M (AI2)

How to load tokenizer using autotokenizer.from_preteained() but with the local folder…

  1. User

    How to load tokenizer using autotokenizer.from_preteained() but with the local folder where huggingface downloaded the files? Like ".cache/huggingface/hub/..."

  2. Assistant

    To load a tokenizer using the AutoTokenizer class from the Hugging Face Transformers library but from a local folder where the files have been previously downloaded, you can use the from_pretrained method and specify the path to the local directory. If the files have been downloaded to the default cache directory (usually ~/.cache/huggingface/hub/...), you can directly use that path.

    Here's a step-by-step example of how to do this:

    1. Locate the Local Folder: First, determine the exact path to the local folder where the tokenizer files are stored. This is typically under the ~/.cache/huggingface/hub/ directory followed by the specific subdirectories created by Hugging Face.

    2. Load the Tokenizer:

      • Import the necessary module.
      • Use the from_pretrained method with the local path.

    Here is a code snippet to illustrate this process:

    from transformers import AutoTokenizer
    
    # Example path to the local folder where the files are stored
    # Replace this with the actual path on your system
    local_path = "~/.cache/huggingface/hub/models--bert-base-uncased/snapshots/1234567890abcdef/"
    
    # Load the tokenizer from the local folder
    tokenizer = AutoTokenizer.from_pretrained(local_path)
    
    # Now you can use the tokenizer as usual
    print(tokenizer.tokenize("Hello, world!"))
    

    Notes:

    • Path Expansion: The ~ in the path will automatically be expanded to your home directory on Unix-based systems (Linux, macOS). If you are on Windows, you might need to specify the fully qualified path or use an equivalent environment variable.
    • Exact Subdirectory: Make sure to navigate to the specific subfolder where the tokenizer files (like config.json, tokenizer.json, etc.) are stored. Hugging Face usually organizes cached models and tokenizers under a nested structure within the .cache/huggingface/hub directory.

    Environment Variable (if needed):

    If you want to retrieve the path programmatically using an environment variable (e.g., if you've set custom cache directories), you can do something like this:

    import os
    from transformers import AutoTokenizer
    
    # Use an environment variable (example)
    local_path = os.path.join(os.getenv("HF_HOME", "~/.cache/huggingface"), "hub/models--bert-base-uncased/snapshots/1234567890abcdef/")
    
    # Load the tokenizer
    tokenizer = AutoTokenizer.from_pretrained(local_path)
    

    This approach will ensure your code is more portable and less dependent on hardcoded paths.