Never Hit a Rate Limit Again: Build a Free LLM Fallback System

Every free LLM tier eventually says no. Groq caps you at 14,400 requests a day, Google AI Studio throttles Pro to 50 RPD, and OpenRouter's free models share a pool that gets tight during peak hours. The fix isn't picking "the best" free API — it's never depending on just one.

I run a few side projects that hit LLM APIs constantly, and the number one cause of 3am pages used to be a single provider's rate limit, not my own code. Once I put a fallback chain in front of every call, that class of outage disappeared entirely.

The idea

Most free-tier providers here expose an OpenAI-compatible /chat/completions endpoint. That means you can treat them as interchangeable — try provider A, and if it fails with a rate-limit or 5xx error, immediately retry the same request against provider B, then C.

A sensible order for general-purpose chat, ranked by daily quota generosity (see the provider directory for current numbers):

1. Groq — 14,400 RPD, extremely fast, good default first hop.
2. Cerebras — 1M tokens/day, similarly fast, different infra so failures rarely correlate with Groq's.
3. Google AI Studio — 1,500 RPD on Flash, reliable as a third layer.
4. OpenRouter free models — last resort, shared pool, but wide model selection.

Minimal Python implementation

Since they're all OpenAI-compatible, this is just a loop over base_url/api_key pairs:

from openai import OpenAI, APIStatusError

PROVIDERS = [
    {"name": "groq", "base_url": "https://api.groq.com/openai/v1", "api_key": GROQ_KEY, "model": "llama-3.3-70b-versatile"},
    {"name": "cerebras", "base_url": "https://api.cerebras.ai/v1", "api_key": CEREBRAS_KEY, "model": "llama-4-scout-17b-16e-instruct"},
    {"name": "google", "base_url": "https://generativelanguage.googleapis.com/v1beta/openai/", "api_key": GOOGLE_KEY, "model": "gemini-3-flash"},
    {"name": "openrouter", "base_url": "https://openrouter.ai/api/v1", "api_key": OPENROUTER_KEY, "model": "meta-llama/llama-4-maverick:free"},
]

def chat(messages):
    last_error = None
    for p in PROVIDERS:
        try:
            client = OpenAI(api_key=p["api_key"], base_url=p["base_url"])
            resp = client.chat.completions.create(model=p["model"], messages=messages, timeout=15)
            return resp.choices[0].message.content
        except APIStatusError as e:
            if e.status_code in (429, 500, 502, 503):
                last_error = e
                continue
            raise
    raise last_error

That's the whole pattern. No queueing, no circuit breaker library — just "catch the 429, try the next one." For most side projects that's already enough to stop rate limits from being your bottleneck.

Two things that will bite you

• Model behavior isn't identical across providers. Llama 4 Scout on Cerebras and Gemini 3 Flash on Google won't format JSON output the same way. If your app parses structured output, validate it after every call regardless of which provider answered — don't assume the fallback speaks in the same voice as your primary.
• Don't fall back on content policy rejections. Only retry on rate-limit and server errors (429/5xx). If a provider refuses a request for a policy reason, trying the same prompt on the next provider usually just wastes a call for another rejection — handle that error path separately.

Wrap up

A fallback chain across two or three free providers costs you maybe 20 lines of code and turns "free API rate limit" from an outage into a non-event. Start with whichever two providers you already have keys for — you can always add a third layer later.