Skip to content
TheChatArchiveThe Chat Archive
gpt-3.5-turbo-0613· OpenAIAi2 WildChat-4.8M (AI2)

what function is used to download models in the huggingface from_pretrained method

  1. User

    what function is used to download models in the huggingface from_pretrained method

  2. Assistant

    The from_pretrained method in the Hugging Face library uses the AutoTokenizer.from_pretrained function to download and load the tokenizer for the specified pre-trained model.

  3. User

    which function inside from_pretrained does the downloading

  4. Assistant

    The from_pretrained method in the Hugging Face library uses the torch.hub.load_state_dict_from_url function to download the pre-trained model weights.

  5. User

    that is only the case sometimes, what is it more generally

  6. Assistant

    Apologies for the confusion. More generally, the specific function used for downloading the pre-trained model weights depends on the Hugging Face library used.

    In the Transformers library, which is commonly used for downloading and using pre-trained models from Hugging Face, the from_pretrained method typically involves multiple steps. It can use either the torch.hub.load_state_dict_from_url function or the tensorflow.python.keras.utils.data_utils.get_file function to download the model weights, depending on the library's implementation.

    The torch.hub.load_state_dict_from_url function is used for models that are compatible with PyTorch, while get_file is used for models compatible with TensorFlow.