what function is used to download models in the huggingface from_pretrained method
- User
what function is used to download models in the huggingface from_pretrained method
- Assistant
The
from_pretrainedmethod in the Hugging Face library uses theAutoTokenizer.from_pretrainedfunction to download and load the tokenizer for the specified pre-trained model. - User
which function inside from_pretrained does the downloading
- Assistant
The
from_pretrainedmethod in the Hugging Face library uses thetorch.hub.load_state_dict_from_urlfunction to download the pre-trained model weights. - User
that is only the case sometimes, what is it more generally
- Assistant
Apologies for the confusion. More generally, the specific function used for downloading the pre-trained model weights depends on the Hugging Face library used.
In the Transformers library, which is commonly used for downloading and using pre-trained models from Hugging Face, the
from_pretrainedmethod typically involves multiple steps. It can use either thetorch.hub.load_state_dict_from_urlfunction or thetensorflow.python.keras.utils.data_utils.get_filefunction to download the model weights, depending on the library's implementation.The
torch.hub.load_state_dict_from_urlfunction is used for models that are compatible with PyTorch, whileget_fileis used for models compatible with TensorFlow.