How to load tokenizer using autotokenizer.from_preteained() but with the local folder…
- User
How to load tokenizer using autotokenizer.from_preteained() but with the local folder where huggingface downloaded the files? Like ".cache/huggingface/hub/..."
- Assistant
To load a tokenizer using the
AutoTokenizerclass from the Hugging Face Transformers library but from a local folder where the files have been previously downloaded, you can use thefrom_pretrainedmethod and specify the path to the local directory. If the files have been downloaded to the default cache directory (usually~/.cache/huggingface/hub/...), you can directly use that path.Here's a step-by-step example of how to do this:
-
Locate the Local Folder: First, determine the exact path to the local folder where the tokenizer files are stored. This is typically under the
~/.cache/huggingface/hub/directory followed by the specific subdirectories created by Hugging Face. -
Load the Tokenizer:
- Import the necessary module.
- Use the
from_pretrainedmethod with the local path.
Here is a code snippet to illustrate this process:
from transformers import AutoTokenizer # Example path to the local folder where the files are stored # Replace this with the actual path on your system local_path = "~/.cache/huggingface/hub/models--bert-base-uncased/snapshots/1234567890abcdef/" # Load the tokenizer from the local folder tokenizer = AutoTokenizer.from_pretrained(local_path) # Now you can use the tokenizer as usual print(tokenizer.tokenize("Hello, world!"))Notes:
- Path Expansion: The
~in the path will automatically be expanded to your home directory on Unix-based systems (Linux, macOS). If you are on Windows, you might need to specify the fully qualified path or use an equivalent environment variable. - Exact Subdirectory: Make sure to navigate to the specific subfolder where the tokenizer files (like
config.json,tokenizer.json, etc.) are stored. Hugging Face usually organizes cached models and tokenizers under a nested structure within the.cache/huggingface/hubdirectory.
Environment Variable (if needed):
If you want to retrieve the path programmatically using an environment variable (e.g., if you've set custom cache directories), you can do something like this:
import os from transformers import AutoTokenizer # Use an environment variable (example) local_path = os.path.join(os.getenv("HF_HOME", "~/.cache/huggingface"), "hub/models--bert-base-uncased/snapshots/1234567890abcdef/") # Load the tokenizer tokenizer = AutoTokenizer.from_pretrained(local_path)This approach will ensure your code is more portable and less dependent on hardcoded paths.
-