Cloudflare AI Gateway can now act as a control layer between your application and multiple AI providers. Instead of hard-coding every request to one model, you can route traffic based on cost, task type, prompt content, user plan, or your own business rules.
That sounds like infrastructure for a large AI company, but the practical use case is simpler. A small team can send routine work to a fast, inexpensive model, reserve premium models for harder requests, and keep a fallback ready when a provider fails.
What AI model routing actually solves
Most AI projects begin with one provider and one model. The application sends a prompt, receives an answer, and moves on. That is easy to build, but it creates four operational problems as usage grows:
- Every request gets the same model, even when the task is simple.
- A provider outage becomes your outage.
- Model experiments require application changes and deployments.
- Cost controls get scattered across prompts, code, and provider dashboards.
A routing layer moves those decisions into one controlled gateway. Your application sends a request to the gateway. The gateway evaluates the request, selects a provider or model, applies limits, records the result, and returns the response.
Cloudflare describes AI Gateway as a proxy that adds observability, caching, rate limiting, retries, fallbacks, and model routing in front of AI providers. Its AI Gateway overview also explains that you can use provider-specific endpoints, OpenAI-compatible endpoints, or Cloudflare's Unified API.
The goal is not to use more models. The goal is to make model choice an explicit business rule instead of an accidental code decision.
Routing works best when each model has a defined job, not when every request is sent through a complicated decision tree.
Start with three routes, not thirty
A useful small-business setup usually needs only three model classes.
| Route | Typical work | Main priority |
|---|---|---|
| Fast | Classification, tagging, extraction, short summaries | Low latency and predictable cost |
| Quality | Customer-facing drafts, analysis, complex instructions | Better reasoning and writing |
| Fallback | Temporary replacement when the primary route fails | Availability and compatibility |
Consider a service business that uses AI to process incoming leads. The same workflow might need to:
- Classify the requested service
- Extract the address and preferred appointment time
- Score whether the lead matches the service area
- Draft a personalized reply
- Summarize unusual requests for a manager
The first three tasks are structured and repetitive. They can usually run on a fast model with strict output rules. The draft and unusual-case summary may deserve a more capable model. If the preferred provider times out, a fallback model can keep the intake process moving.
Define routing policy in plain English first
Before writing a rule, write the decision as a short policy:
- Use the fast route for classification and extraction.
- Use the quality route when the prompt requests analysis or a customer-facing draft.
- Use the quality route for paid accounts that explicitly request premium processing.
- Reject or queue requests when the daily budget is exhausted.
- Use the fallback only when the selected provider returns an eligible failure.
If your team cannot explain the rule without code, the rule is probably too complex to operate safely.
How Cloudflare dynamic routing works
Cloudflare's dynamic routing documentation describes a named, versioned route built from connected nodes. You can design the flow in a visual editor or manage its JSON configuration through the API.
The available nodes include a starting point, conditional branches, percentage splits, model calls, rate limits, budget limits, and an ending point. Conditions can reference the request body, headers, or metadata supplied by your application. Percentage nodes support controlled tests and gradual rollouts. Limit nodes can send traffic to a fallback path when a quota is reached.
Each edit creates a draft version that can be deployed or rolled back. That matters in production because the route can change without modifying the application that calls it. Cloudflare's dynamic route usage guide says the application uses dynamic/<route-name> as the model and names the gateway that owns the route.
Example routing plan
A simple production plan might look like this:
Rule 1: If x-task-type is classification, use Fast Model.
Rule 2: If x-customer-plan is premium, use Quality Model.
Rule 3: If the prompt contains a request for detailed analysis, use Quality Model.
Default: Use Fast Model.
Fallback: If the selected provider fails, try Backup Model.
The headers in this example should be set by your own trusted application after authentication. Do not let a public browser choose a premium route by sending an unverified header.
Cloudflare's documentation says dynamic routing requests use the OpenAI-compatible chat completions format and a gateway-level model identifier. This lets your application target the routing configuration while the gateway decides which underlying provider and model receives the request.
Keep security outside the prompt
Do not ask the model to choose its own route with instructions such as, "Decide whether this request deserves the expensive model." That can be useful during testing, but it is a weak production control because prompt content is user-controlled.
Use trusted metadata for high-impact choices:
- Authenticated account plan
- Internal task type
- Approved workflow ID
- Maximum cost per request
- Data sensitivity classification
- Whether the request is customer-facing
Prompt content can still be one signal, but it should not be the only authority for budgets, permissions, or access to sensitive tools.
Build fallback without hiding failures
Cloudflare supports fallbacks through its Unified API. You can provide an ordered list of provider and model options. If the first attempt fails with an eligible error, the gateway can try the next option.
A fallback is valuable, but it creates a new risk: the application may stay online while quality quietly changes.
Models do not behave identically. They may use different tool-call formats, safety policies, context limits, response styles, or structured-output behavior. A fallback that returns HTTP 200 is not automatically equivalent to the primary model.
Use these safeguards:
- Test the same evaluation set against primary and fallback models.
- Validate structured output after every route.
- Record which provider and model handled each request.
- Alert when fallback usage crosses a normal threshold.
- Keep prompts compatible with every model in the fallback chain.
- Require human review for high-risk actions after a fallback.
For example, a fallback may be acceptable for drafting a missed-call reply. It may not be acceptable for interpreting a contract clause or recommending a financial action without another review step.
Put cost and abuse controls in the gateway
Routing saves money only if the system also limits runaway usage. Cloudflare's rate limiting documentation supports limits based on requests or token counts over a defined period.
That gives a small team several practical controls:
- Limit anonymous trial traffic separately from authenticated customers.
- Cap tokens per user, location, or workflow.
- Apply tighter limits to expensive routes.
- Stop a broken automation from making thousands of repeated calls.
- Protect shared capacity from one unusually large customer.
Rate limits should match the business workflow. A support assistant may need many small requests throughout the day. A monthly report generator may need a few large requests. Applying one global request limit to both will produce bad results.
Use caching only for repeatable requests
AI Gateway also supports response caching. Cloudflare documents that cached responses can be served from its network and that custom cache keys can be used to control how requests are grouped.
Caching can help with repeatable tasks such as:
- Generating the same product description from unchanged source data
- Answering a fixed policy question
- Classifying an identical document hash
- Running a deterministic prompt against static content
Do not cache responses that contain personal data, account-specific recommendations, current inventory, changing prices, or time-sensitive instructions unless the cache key and retention policy isolate those cases correctly.
A good default is to leave caching off until you can describe exactly why two requests should receive the same response.
Fallbacks and alternate paths should be visible to the team. A successful response can still represent degraded service.
Measure routing as a product decision
The routing policy is not finished when requests return successfully. You need to compare the routes using business outcomes.
Cloudflare's logging and observability documentation describes logs for gateway requests and controls for viewing, exporting, and managing them. Cloudflare also documents that log fields can include provider, model, token, timing, and cost information when available.
Track at least these metrics:
| Metric | What it tells you |
|---|---|
| Success rate by route | Whether one provider or model fails more often |
| P50 and P95 latency | What normal and slow experiences feel like |
| Cost per completed task | Whether cheaper routing creates real savings |
| Fallback rate | Whether the primary route is unstable |
| Human correction rate | Whether lower-cost output creates more cleanup |
| Customer acceptance rate | Whether generated drafts are actually useful |
Cost per token is not the final metric. If a cheaper model saves 40 percent on API charges but doubles employee correction time, the route may cost more overall.
Run a controlled test
Start with shadow evaluation before changing production behavior.
- Collect 50 to 100 representative requests from the workflow.
- Remove or replace sensitive customer information.
- Run the set through the current model and proposed routes.
- Score accuracy, format compliance, latency, and correction time.
- Send a small percentage of live traffic through the new policy.
- Review failures and fallback events daily during the pilot.
- Increase traffic only after the business outcome stays stable.
Do not test only clean prompts. Include long requests, vague requests, malformed data, prompt-injection attempts, provider timeouts, and outputs that fail schema validation.
A practical 30-day rollout
Week 1: inventory the current calls
List every AI request your application makes. Record the task, current model, average volume, required output, maximum acceptable delay, and consequence of a bad answer.
Mark each task as low, medium, or high risk. A blog-title suggestion is low risk. Sending a customer refund or changing a legal record is high risk.
Week 2: create the fast and quality routes
Move one low-risk workflow behind AI Gateway. Define a fast default route and a quality route controlled by trusted application metadata. Keep the existing direct-provider path available for rollback.
Add schema validation and save the selected provider and model with each result.
Week 3: add limits and fallback
Set request or token limits that match the workflow. Add one fallback model. Test timeout, rate-limit, authentication, malformed-output, and provider-error scenarios.
Make fallback events visible in alerts or a daily operations report. Do not let fallback become a permanent hidden primary route.
Week 4: evaluate the business result
Compare the routed workflow with the previous setup. Review total provider cost, employee correction time, latency, task success, customer complaints, and fallback frequency.
Keep the policy if it improves the complete workflow. Simplify it if the operational burden is larger than the savings.
The practical recommendation
Cloudflare AI Gateway dynamic routing is useful when your business has more than one kind of AI task, needs provider resilience, or wants model experiments without repeated application deployments.
Start with a fast route, a quality route, and one tested fallback. Use trusted metadata for important decisions. Put budgets and rate limits at the gateway. Validate every structured response. Log the model that actually handled the request.
The best routing system is not the one with the most branches. It is the one your team can explain, test, monitor, and reverse when the data says it is wrong.