Tokens Don't Come With Cost Centres: Allocating AI Spend at Enterprise Scale
Ask most organisations which team drove their AI spend last month and you will usually get a shrug. The bill carries plenty of detail, all of it on dimensions the provider chose. These are the ideas that close that gap, and none of them need a programme behind them.
If a GPU burns 4.000 € overnight and no cost centre hears it, does it make a sound? Finance says yes. Finance has the invoice.
Try the practical version. Open last month's cloud bill and work out which team drove the most AI spend. Most people get as far as the service name and stop.
That is not a skills problem. The same organisations usually run a solid cloud cost practice, with tags enforced at provision time and a showback report that lands every Monday. AI spend broke the assumptions that practice was built on, and it happened quickly enough that nobody has stopped to say so out loud.
What follows is a set of ideas rather than a programme. There is no phase plan here and no eight week roadmap, because the right sequence depends on where you are starting and who is willing to engage. Take the pieces that fit.
One scoping note before any of them. At this scale the question is never whether attribution matters, it is where to start, because nobody instruments a whole estate at once. Three things are worth looking for first: a model endpoint provisioned centrally and called from across the estate, any capacity bought as a commitment, and any GPU running somewhere with no billing API attached to it. Those are the places where your existing allocation quietly stops working, and they tend to be where most of the money already is.
01
AI spend does not behave like infrastructure
Cloud cost management assumes you can point at a resource and name its owner. One VM, one team, one tag applied when it is provisioned. Every report downstream depends on that assumption holding.
One piece of vocabulary before the rest, because everything below leans on it. A token is a chunk of text, roughly four characters of English, or about three quarters of a word. You pay for the tokens you send and, at a higher rate, for the ones that come back.
AI breaks that assumption in several places at once.
What changes | Traditional cloud | AI workloads |
Billing unit | Provisioned hours | Tokens, inferences, GPU seconds |
Attribution | Tag the resource once | One endpoint serves everyone at once |
Predictability | Stable after provisioning | A single batch job can move the monthly number |
Tooling | Allocation built into every FinOps platform | Rich provider detail, thin support for business dimensions |
Vendors | One or two providers | Cloud plus model vendors plus self-hosted GPU |
The second row is the one that hurts. A shared model endpoint has no idea who is calling it, and neither does the invoice. Not because the bill is thin. A month of Bedrock usage runs to thousands of rows, split by model, region, usage type and API operation, and Azure and Vertex give you comparable depth. Every one of those dimensions belongs to the provider. None of them is a team, a product or a feature, and no amount of slicing gets you from one set to the other.
02
Five places the money disappears
Most unattributed AI bills come apart into some mix of these. Open the ones that sound familiar.
There is no cost centre on an API call
When your application calls a model, the invoice records the model, the region, the usage type and the token counts. It does not record which team or feature triggered the call, because the provider never saw that. Whatever you want to allocate by has to be attached before the request leaves your network.
Shared GPU infrastructure has no usage telemetry
A self-hosted model on a GPU cluster gets used by several teams. The bill is instance hours, which tells you the cluster ran and nothing about who used it. Unless something in front of it counts requests per caller, you are left splitting the bill on guesswork.
Dev experiments land in the same bucket as production
Engineers try things. That is the job. But a weekend of batch experiments shows up next to production serving, in the same budget line, and you cannot separate waste from value. Nineteen percent of spend sitting in dev is only a conversation once someone can see it.
Nobody watches the token profile
Output tokens cost several times more than input tokens on most commercial models. Claude Sonnet 4.6 on Bedrock is listed at 3 dollars per million input and 15 dollars per million output, a five to one ratio. Teams that track only total spend miss a prompt change that triples verbosity until the invoice arrives.
There is no loop back to the person who can fix it
Without visibility teams cannot optimise. Without any form of accountability they have no reason to. Without either, finance cannot forecast, so the budget conversation happens once a year and everyone guesses.
Visibility, attribution and
accountability. Most organisations
start with none of the three.
03
Put something in the middle
This is the idea that changes the most for the least effort, so it goes first.
Waiting for cloud billing data to solve attribution does not work. It arrives a day or two late, and the only dimensions it carries are the provider's own. You can slice it by model and region all month without getting any closer to a team. The alternative is a gateway: one hop that every AI call passes through on its way out. It stamps allocation metadata, decides which model the request deserves, counts the tokens, and emits an event you can query.
Easier to see than to describe. Switch the gateway off below and send a few requests, then switch it on and watch what changes.
Send 20 always sends the same 20 requests, so the figures do not shift between visits: 12 general questions and classifications, 6 drafting jobs, 2 code generation. One in five is a near-repeat, which is where the cache hit rate comes from.
The routing result is the one worth sitting with. With routing off, every request lands on the mid model, because that is what somebody picked once and nobody revisited. Asking "what is our refund window" costs exactly as much as generating a migration script.
Turn routing on and the gateway reads the request before it forwards it. General questions and classification go to the small model at roughly a fifth of the price. Drafting and summarising stay on the mid model. Code generation and debugging go up to the large model, which costs four times what the mid model does, and is worth it for that work.
That last part is why the number is modest. Routing does not halve your bill, because some work genuinely belongs on the expensive model and sending it there costs more than before. What it stops is paying mid-tier prices for questions the small model answers perfectly well. In this sample it takes 18 percent off, and 18 percent is a figure that holds up when an engineer checks the working.
We already allocate by account. Why is this different?
Fair question, and account structure is not the wrong answer. Most enterprises already vend accounts or subscriptions per team, and the existing FinOps tooling allocates infrastructure along exactly those lines. Where an AI workload runs on dedicated infrastructure inside one team's account, you are already done and none of this applies to it.
The gap opens in four specific places, and at this scale all four are usually present at once. A managed model endpoint gets provisioned once in a platform account and called by dozens of applications across the estate, so the spend lands on the platform team instead of the callers. Committed capacity, whether that is Bedrock Provisioned Throughput, Azure OpenAI PTUs or reserved GPU, is bought centrally against a forecast and consumed by everybody, which makes it a shared commitment that no account boundary can split. GPU running in private cloud or on-prem has no billing API to allocate from in the first place. And account structure answers which team, never which product, which feature or which use case tier, which is the granularity the funding conversation actually needs.
Credentials do not rescue this either. Once several thousand engineers and service principals hold access, the map from credential to cost centre is stale the week after somebody writes it, and a key still records who spent without ever recording what the request was for.
Keeping the gateway from becoming the outage
The obvious objection to all of this is that one hop every AI call passes through is a new single point of failure, and if it stops, every AI feature in the company stops with it. That objection is correct. It is also why most of these projects get quietly shelved after the architecture review, which is a shame, because the failure is avoidable and mostly comes down to what you build first.
The version that goes wrong starts as a proxy. Somebody stands up LiteLLM, points all traffic at it, and a cost reporting tool is now sitting in the request path of every customer-facing feature you have. When the outage arrives it will not even be the proxy's fault. It will be the event queue behind it filling up, or the tag validation timing out, and the proxy refusing requests it could have forwarded perfectly well.
So start with a library. A thin SDK wrapper that stamps the metadata and emits a usage event gives you attribution, which is the part you came for, with no new network hop and nothing to fall over. Teams adopt it by bumping a version. You can be reading cost per feature inside a sprint, and on the day a proxy does go in, every caller is already sending the right headers. Reach for the proxy when you need something the library cannot do from inside the caller, which means routing policy, a shared cache, key management, or enforcement a team cannot switch off.
When that day comes, make it fail open. If the gateway cannot reach its queue or its validation service, it forwards the request anyway and records that the tags are missing. Losing attribution on a few hundred calls costs you a gap in a report. Losing the feature costs you the meeting where you explain why the assistant was down. Dev and staging are the only places enforcement should reject outright, because a failed build there is the entire point.
Two smaller habits follow from the same instinct. Emit the usage event at a queue and return, so that nothing about writing to your cost data lake ever sits between a user and their answer, and budget single-digit milliseconds for it rather than assuming them. Then leave the client a way home without you, so a proxy that goes unreachable inside a short timeout means the SDK calls the provider directly and marks the request unattributed. After that, treat the gateway as the production dependency it has quietly become. Stateless, more than one instance, more than one zone, an SLO, somebody on call. If nobody will fund that, the library version is still there and it still answers the question you started with.
One more thing, since this is a post about splitting shared costs. The gateway is itself a shared cost. It is small next to the inference bill, but somebody pays for it, and a platform team absorbing it quietly is how these things get resented. The tidy answer is to allocate it by request volume using the data it generates, which makes it the only shared cost in your estate that can bill for itself.
The tags themselves
Six that do the work: team_id, product, feature, environment, model_id and request_type. Two more that make it useful to other people: cost_centre, so the numbers reach the finance system, and use_case_tier, so you can tell core workloads from experiments when deciding where to push.
If tagging lives in application code, it will drift within weeks. Enforce it at the gateway. Start by warning on missing headers, publish a date, then reject. A failed build is a cheaper conversation than a code review comment that everybody ignores.
04
Somebody still has to pay for the shared cluster
A shared GPU cluster is the tragedy of the commons with better cooling. Everybody grazes, nobody owns the field, and the invoice still arrives on the 3rd.
Once you can see usage, you have to decide how the shared cost gets split. There is no correct answer here, only trade-offs, and the choice matters more for how it feels than for how it calculates. A method the teams cannot explain to each other will not survive its first argument.
Sample figures. Usage for the month: research 410 GPU hours, product 180, support 60.
Showback before chargeback
Teams seeing their own spend, with no money moving, does most of the work. It builds the habit and surfaces the obvious waste without anyone having to defend a budget. Real chargeback, where a cost centre gets debited, is worth reaching eventually, but it needs stable data behind it and a notice period in front of it. Billing a team retroactively for a number they have never seen is the fastest way for a FinOps function to lose its credibility.
05
Ideas worth trying
These are the patterns that keep paying off. Some are cheap, some need engineering time, none of them require a programme.
Cache on meaning, not on string match
An embedding lookup in front of your cache catches prompts that mean the same thing but read differently. Worth most in features with repetitive query patterns, like a support assistant answering the same twenty questions all week.
Route by prompt type
A small classifier that reads what the request actually asks for and sends it to the smallest model that can handle it. The hard part is not the routing, it is agreeing what quality bar each tier has to clear. Do that first or the whole thing gets reverted after one bad output.
Measure your system prompt
The system prompt rides along with every single request, which makes it the quietest large cost in most applications. Count its tokens, then count how many calls per day carry it. That multiplication is usually the moment someone volunteers to trim it.
Move async work to batch
Amazon Bedrock prices batch inference at 50 percent of on-demand for supported models, with results typically back inside 24 hours. Embedding generation, document indexing and enrichment jobs do not need to run at 2pm on a Tuesday. Check your model is on the supported list before you plan around it.
Put a tax on the dev environment
Make the expensive models unavailable in dev by default, with a cheap tier served through gateway routing. Getting the full model needs an explicit override flag, and that flag shows up in the cost report. Engineers keep the ability to test properly, and the spend stops being invisible.
Alert on the token ratio, not just the total
A jump in output tokens divided by input tokens means someone changed a prompt and the model got chatty. The ratio moves days before the absolute number looks alarming, which makes it the more useful alert of the two.
Track cost per outcome
Cost per call is a number for engineers. Cost per resolved ticket, completed transaction or finished session is the number you can take to a business owner. A feature costing 2 € a session while driving real conversion is a different object entirely from one costing 10 cents that nobody uses.
The levers apply one after another to whatever spend is left, not added together, because they overlap. Caching removes calls that routing would otherwise have made cheaper. Treat the output as a shape to argue about, not a forecast.
06
What to put on the dashboard
Eight metrics, and you can probably start with three of them.
Metric | What it tells you | Rough target |
Attributed spend | Share of AI cost tagged to a team or product | The first one to fix |
Cost per 1K tokens | Blended rate across models and providers | Should drift down |
Cost per inference | Average per call, split by feature | Chase the outliers |
Cost per outcome | Per transaction, ticket or session | Your own north star |
Cache hit rate | Share of requests never reaching a model | Set it per feature |
Model mix | Share of calls on the cheapest model that clears the bar | Higher than you expect |
Budget utilisation | Share of budget spent by month end | Consistently over 100 means bad planning, not overspend |
Waste ratio | Share of spend in dev and test | Watch the trend |
A note on tooling
You do not need to buy anything to start. LiteLLM is an open source proxy that handles multi-provider routing and cost tracking, at the price of running it yourself. Azure API Management works well if you are already Azure-shaped. Helicone and LangSmith give you call-level tracing. Whatever you land on, send the events somewhere you can query them, which for most people means BigQuery or Snowflake.
Worth knowing about: FOCUS, the FinOps Foundation's open specification for normalising billing data across providers. Version 1.4 was ratified in June 2026 and now covers AI alongside cloud and SaaS, with columns for allocation methodology and invoice detail. Building your cost data lake on that schema saves you inventing your own and rewriting it in a year.
Where people usually start
Three things, none of which need a budget or a steering committee.
Pull the last three months of AI spend, filter to AI and ML services, and find out how much has no team tag on it. That number is your problem statement and you can get it this afternoon.
Then pick one team with high spend and a cooperative engineering lead, and instrument a single feature end to end. Get cost per request in front of their product manager. The reaction to that first number is what creates the momentum for everything else.
Then write down your mandatory tag set, one page, and get it signed off. Five tags required on every AI call from a stated date. Everything downstream depends on that decision, and it is the cheapest one on the list.
The goal was never to cut AI spend. It is to understand it well enough that every euro can be defended, and the waste can go somewhere useful. That is how you earn the right to spend more.
Comments