--- term: 'LLM API' seoTitle: 'LLM API: Tokens, Streaming, Tool Calling, Server-Side Keys' headline: 'What is an LLM API?' slug: llm-api category: ai-modern-stack shortDefinition: 'An LLM API is an HTTP endpoint to a hosted language model: send a prompt, get generated text back, billed per token.' relatedTerms: - api - api-key-security - cloud-code-serverless-functions - retrieval-augmented-generation-rag contrastsWith: - ai-agent aboutTerms: - 'Tokens' - 'Chat Completions' - 'Streaming (SSE)' - 'Tool Calling' faq: - question: 'What is an LLM API?' answer: 'An HTTP endpoint that lets your code send a prompt to a hosted large language model and get generated text back — without running the model, GPUs, or inference infrastructure yourself. You POST a JSON body, you receive an assistant message plus token-usage counts, and you pay per token.' - question: 'How do you call an LLM API?' answer: 'POST a JSON body — a messages array plus parameters — to a chat-completions endpoint, with your API key in the authorization header. The model returns an assistant message and a usage object counting the input and output tokens. The shape is portable: most providers accept the same chat-completions format.' - question: 'What is a token, and how is pricing calculated?' answer: 'A token is a chunk of text — roughly four characters or three-quarters of a word in English. You pay per million tokens, with separate rates for input (your prompt) and output (the generated reply). Output typically costs several times more than input, because generating each token is more compute than reading one.' - question: 'What is a context window?' answer: 'The maximum number of tokens — input plus output — a model can consider in one request. It caps how much conversation history or document you can include, and it is a direct cost driver: every call pays for the whole context you send, so a long history or a big retrieved passage is money spent on each request.' - question: 'What are the key parameters?' answer: 'temperature (0–2, controlling randomness; 0 is near-deterministic), top_p (nucleus sampling — tune this or temperature, not both), max_tokens (a cap on output length that bounds cost and latency), and stop (sequences that halt generation). The defaults are reasonable; max_tokens is the one you should almost always set.' - question: 'What is streaming, and why do chat UIs use it?' answer: 'Setting a stream flag returns the reply incrementally as Server-Sent Events instead of one final blob, so the UI renders tokens as they generate. It exists for perceived latency: time-to-first-token is a fraction of time-to-full-answer, so the user sees words immediately rather than a spinner for several seconds.' - question: 'What is function or tool calling?' answer: 'You describe available tools — a name and a JSON schema — in the request; the model, instead of answering in prose, returns a structured JSON call naming a tool and its arguments. Your code executes it and feeds the result back. The model never runs the function; it only requests it. This is the mechanism that turns an LLM API into an agent.' - question: 'Should you call an LLM API from the client or the server?' answer: 'The server, always. An API key shipped in a browser bundle or mobile binary is one network-tab inspection or one decompile from theft — and a stolen model key is a stranger spending your token budget. Every LLM call goes through your backend, with the key in server-side configuration.' - question: 'LLM API or self-host an open model?' answer: 'An API means zero infrastructure, per-call billing, and shipping today; self-hosting an open model (with tools like Ollama, vLLM, or llama.cpp) buys full control and data residency but only wins on cost at very high, sustained volume once GPUs and operations are counted. Most products start with the API and revisit at scale.' - question: 'How do you handle rate limits and failures?' answer: 'Limits come as requests-per-minute and tokens-per-minute; on a 429 or a 5xx, retry with exponential backoff plus jitter and honor any Retry-After header. Do not retry 4xx errors — bad auth, malformed requests, and content-policy refusals will fail identically the second time.' codeLanguages: [javascript, dart, swift, kotlin] externalAuthorities: - name: 'Server-Sent Events — WHATWG HTML Living Standard' url: 'https://html.spec.whatwg.org/multipage/server-sent-events.html' - name: 'How to call an LLM API with function calling — Martin Fowler' url: 'https://martinfowler.com/articles/function-call-LLM.html' - name: 'RFC 8259 — JSON' url: 'https://datatracker.ietf.org/doc/html/rfc8259' - name: 'Ollama — run open models locally' url: 'https://github.com/ollama/ollama' - name: 'Large language model — Wikipedia' url: 'https://en.wikipedia.org/wiki/Large_language_model' cta: title: 'Where the LLM call belongs' text: 'Put the model call in a Back4app Cloud Function: key server-side, cost caps enforced, result written to the database and streamed to clients over Live Queries — no key on the device, ever.' linkText: 'Start building free' linkUrl: 'https://www.back4app.com/signup' author: 'Back4app Engineering' publishedDate: '2026-07-30' translationKey: llm-api --- **An LLM API is an HTTP endpoint to a hosted language model: send a prompt, get generated text back, billed per token.** It is an ordinary [API](/glossary/api/) with three unusual properties — it's *stateless* (you resend the whole conversation each call), *metered by tokens* (input and output priced separately), and *non-deterministic* (the same prompt can return different text). Master those three and the rest is the operational discipline the ranking pages skip: where the call belongs, what it costs, and how it fails. ## Key takeaways | Question | Answer | | --- | --- | | What it is | POST a `messages` array → get an assistant reply + token counts | | The billing | Per token, input and output priced separately, output costs more | | The stateless catch | Memory is you resending prior turns — every call, every token paid | | The iron rule | Never call it from the client — the key leaks; proxy through the backend | | The bridge to agents | Tool calling — the model requests a function, your code runs it | ## A real call, server-side **JavaScript:** ```javascript // JavaScript — Cloud Code (cloud/main.js) // The LLM call belongs server-side — the key never reaches the client Parse.Cloud.define('summarize', async (req) => { const res = await Parse.Cloud.httpRequest({ method: 'POST', url: 'https://api.llm-provider.example/v1/chat/completions', headers: { Authorization: `Bearer ${process.env.LLM_KEY}`, // server-side secret 'Content-Type': 'application/json', }, body: { model: 'default-chat', messages: [ { role: 'system', content: 'Summarize in one sentence.' }, { role: 'user', content: req.params.text }, ], max_tokens: 80, // cost + latency guardrail, enforced by YOU }, }); return res.data.choices[0].message.content; }); ``` **Flutter:** ```dart // Flutter / Dart — Back4app Flutter SDK // The client calls YOUR function, never the LLM API directly final summary = await ParseCloudFunction('summarize') .execute(parameters: {'text': longArticle}); print(summary.result); // The model key stays on the server. If this app called the LLM API // itself, the key would ship in the binary — one decompile from theft. ``` **Swift:** ```swift // iOS / Swift — Back4app Swift SDK // The client calls YOUR function, never the LLM API directly let summary: String = try await Cloud.run( name: "summarize", parameters: ["text": longArticle]) print(summary) // The model key stays on the server. If this app called the LLM API // itself, the key would ship in the IPA — one decompile from theft. ``` **Kotlin:** ```kotlin // Android / Kotlin — Back4app Android SDK // The client calls YOUR function, never the LLM API directly val summary = ParseCloud.callFunction( "summarize", mapOf("text" to longArticle)) println(summary) // The model key stays on the server. If this app called the LLM API // itself, the key would ship in the APK — one decompile from theft. ``` The shape is portable — most providers accept the same chat-completions format, so the request and response below read the same whichever model you point at: ```text POST /v1/chat/completions Authorization: Bearer { "model": "default-chat", "messages": [ { "role": "system", "content": "You are concise." }, ← sets behavior { "role": "user", "content": "Explain tokens." } ], ← the instruction "max_tokens": 200, "temperature": 0.7, "stream": false } → { "choices": [ { "message": { "role": "assistant", "content": "A token is…" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 24, "completion_tokens": 118, "total_tokens": 142 } } ← what you're billed on ``` ## Tokens are the currency A **token** is a chunk of text — roughly four characters, or three-quarters of an English word. Everything about LLM API cost reduces to counting them, and two facts surprise people. **Output costs more than input** — often several times more — because reading your prompt is cheap while generating each reply token is compute-intensive, one prediction at a time across the whole vocabulary. And **context is paid every call**: the model is stateless, so "memory" is you resending prior turns, and a long history or a big [retrieved passage](/glossary/retrieval-augmented-generation-rag/) is billed *again* on every request. The rough monthly formula worth internalizing: ```text monthly cost ≈ ( requests/day × (avg_input_tokens × input_price_per_M / 1_000_000 + avg_output_tokens × output_price_per_M / 1_000_000) ) × 30 The levers that move it: cap max_tokens, trim the context you resend, pick a smaller model for easy tasks, and cache reused prompt prefixes. ``` ## The parameters that matter | Parameter | Controls | Practical guidance | | --- | --- | --- | | `temperature` | Randomness (0–2) | 0 for extraction/classification; ~0.7 for prose | | `top_p` | Nucleus sampling | Tune this *or* temperature — not both | | `max_tokens` | Output length cap | Almost always set it — bounds cost and latency | | `stop` | Halt sequences | End generation at a delimiter you control | ## Streaming: why chat UIs feel fast Set a `stream` flag and the reply arrives incrementally as [Server-Sent Events](https://html.spec.whatwg.org/multipage/server-sent-events.html) — `data:` lines, one token-chunk at a time, ending in a `[DONE]` marker — instead of one blob after several seconds. The reason is perceptual: **time-to-first-token** is a fraction of time-to-full-answer, so the user watches words appear rather than a spinner spin. It's one-directional server-to-client text over plain HTTP, which is exactly [SSE's shape](/glossary/sse-vs-websockets-vs-polling/) and precisely why LLM streaming uses it rather than WebSockets. The backend consequence: your proxy must stream *through* — reading the provider's event stream and forwarding it — rather than buffering the whole reply and defeating the point. ## Tool calling and structured output Two features turn "text in, text out" into something programmable. **Tool calling:** you describe available functions (name + JSON schema) in the request, and the model — instead of prose — returns a structured JSON call naming a tool and its arguments; your code executes it and feeds the result back ([Fowler's walkthrough](https://martinfowler.com/articles/function-call-LLM.html) is the clear reference). The model never runs the function; it only *asks*. **Structured output**, distinct from tool calling and often confused with it, is three things worth separating: | Mechanism | Guarantee | Use for | | --- | --- | --- | | JSON mode | Valid JSON — but any shape | Loose "give me JSON" needs | | Structured Outputs | Matches *your* schema exactly | Extraction, classification, typed data | | Tool calling | A call to *your* function | Doing things, not just formatting | Tool calling is the mechanism that makes an [AI agent](/glossary/ai-agent/) possible — the same request/response, run in a loop until the model stops asking for tools. ## LLM API vs. self-hosting an open model | | Hosted LLM API | Self-hosted open model | | --- | --- | --- | | Infrastructure | None — the provider runs it | GPUs, serving stack, ops | | Billing | Per token, pay-as-you-go | Fixed capacity, yours to fill | | Time to ship | Minutes | Days to weeks | | Data residency | The provider's terms | Fully yours | | Cost at low/spiky volume | Cheapest | Idle GPUs burn money | | Cost at extreme sustained volume | Can exceed self-hosting | Wins past the break-even | The honest read: an API wins for almost everyone almost always — zero infrastructure, instant start, and cheaper until volume is genuinely large and steady. Self-hosting an open model (with [Ollama](https://github.com/ollama/ollama), vLLM, or llama.cpp) earns its operational cost only past a high break-even, or when data residency is a hard requirement. Most products start on the API and revisit the question if scale ever forces it. ## Reliability: LLM endpoints are flaky Treat the LLM API as a slow, rate-limited, occasionally-failing remote dependency, because it is. Limits arrive as **requests-per-minute and tokens-per-minute**; exceed either and you get a `429`. The discipline is the same one every [rate-limited API](/glossary/api-rate-limiting-throttling/) demands: on a `429` or a `5xx`, retry with **exponential backoff plus jitter** and honor `Retry-After`; on a `4xx` — bad auth, malformed request, content-policy refusal — *don't* retry, because it will fail identically. Add per-request timeouts (generation can stall), and remember that a retried non-idempotent call costs tokens twice. None of this is LLM-specific; all of it is skipped by the tutorials that stop at the happy-path `curl`. ## The rule the tutorials bury: never call it from the client The single most important operational fact, and the one glossary pages omit: **the LLM API call belongs on your server, never in the browser or app.** A model API key shipped to the client is one network-tab glance or one binary decompile from theft — and unlike a leaked publishable key, a stolen *model* key is a stranger with your token budget and no rate limit but your bill. The pattern is the [same proxy discipline](/glossary/api-key-security/) every secret-keyed service needs: the client calls *your* endpoint, your backend holds the key in server-side configuration and calls the model, and the reply comes back through you. That proxy is also where every other control in this article lives — cost caps, retries, streaming, prompt trimming — which is why "where does the call go?" has one answer. ## Common use cases - **Summarization and extraction** — turn long text into short structured data, `temperature: 0`. - **Chat and assistants** — streamed replies over a resent conversation history. - **Classification and tagging** — Structured Outputs enforcing your label schema. - **[RAG answers](/glossary/retrieval-augmented-generation-rag/)** — generation grounded in retrieved context, cost-managed by trimming that context. - **Agentic actions** — tool calling in a loop, each tool a permissioned backend function. ## Should you use an LLM API? A decision matrix | Situation | Reach for | | --- | --- | | Shipping an AI feature now | An LLM API — zero infrastructure | | Any client-facing app | The API, called **server-side** — never from the device | | Extreme, sustained volume | Evaluate self-hosting (Ollama, vLLM) past the break-even | | Strict data residency | Self-host, or a provider with the right guarantees | | Deterministic, rule-based task | Maybe no LLM at all — plain code is cheaper and reliable | | Model must *do* things | Tool calling → an [agent](/glossary/ai-agent/) | ## Limitations and trade-offs - **Non-determinism is the default.** The same prompt varies run to run; anything requiring exact repeatability needs `temperature: 0` and, often, validation on the output. - **Cost scales with tokens, silently.** A generous context or an unbounded `max_tokens` turns a cheap feature expensive at volume — meter usage, don't assume it. - **Latency is seconds, not milliseconds.** LLM calls are slow by web standards; design for it with streaming, async patterns, and honest loading states. - **The dependency is external and rate-limited.** Provider outages and throttling are your outages; retries, fallbacks, and caching are resilience, not polish. - **Outputs can be wrong and confident.** The API returns fluent text regardless of truth; grounding ([RAG](/glossary/retrieval-augmented-generation-rag/)) and validation are how you trust it. ## LLM APIs on Back4app Back4app is an open-source Backend-as-a-Service (BaaS) platform that combines a managed database, auto-generated REST and GraphQL APIs, authentication, file storage, and Cloud Code serverless functions. The "where does the call belong" question answers itself here: inside a [Cloud Function](/glossary/cloud-code-serverless-functions/), exactly as the code tabs show — the client calls your `summarize`, the function holds the model key in [server-side config](/glossary/api-key-security/) and calls the LLM API, and the key never touches the device. That one placement solves the whole gap list at once: cost caps (the function sets `max_tokens` and picks the model), retries and backoff (in the function, against the flaky endpoint), and delivery two ways — return the reply synchronously for a single answer, or write it to the database and let clients watch it fill in over [Live Queries](/glossary/real-time-live-queries/), which is streaming without a socket you had to build. The LLM API stops being an integration risk and becomes one more server-side call your backend already knows how to make safely.