Set up

1. Huggingface CLI and access token
   pip install huggingface-hub
2. Install ollama

Steps

Let’s use TAIDE for example.

1. Go to model page
   https://huggingface.co/taide/TAIDE-LX-7B-Chat-4bit
2. Pull the model
   huggingface-cli download taide/TAIDE-LX-7B-Chat-4bit --local-dir downloads --local-dir-use-symlinks False
3. Create a Modelfile with the following content
   Modelfile FROM ~/Downloads/taide-7b-a.2-q4_k_m.gguf
4. run ollama create <your-model-name> -f Modelfile
5. run the model using ollama as usual

Reference

• https://www.markhneedham.com/blog/2023/10/18/ollama-hugging-face-gguf-models/
• https://www.reddit.com/r/ollama/comments/1al30ut/how_to_run_gguf_model_in_ollama/