Any GGUF Model

Model ID

Use It in Python

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:5001/api/v1",
    api_key="YOUR_API_KEY",
)

response = client.chat.completions.create(
    model="koboldcpp/gguf",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Don't have a key yet? See how to get one — quick registration required.

Free Tier Limits (KoboldCpp)