When can a simple LLM call be replaced with conventional code?

A stable rule has an owner and a boundary

The first step is to write the behavior down without referring to the prompt. Specify the input fields, valid values, outputs, precedence rules, error handling, and who changes the rule when business policy changes. If a product owner cannot say what should happen for a missing field, an unknown code, a conflicting signal, or a new input variant, the LLM may be hiding unresolved requirements rather than doing work that code can safely take over.

Code is a strong fit when the task is a lookup, calculation, parsing rule, normalization, fixed decision tree, schema validation, or policy mapping. The input should be structured or parseable with known rules, and the output should be checkable without asking a model whether it “seems right.” A model may still be useful before that boundary, for example to extract fields from an unstructured message. The deterministic decision that follows can then be conventional code.

Avoid turning an LLM’s sampled answers into an invented policy. The useful process is to recover the actual rule from the domain owner, existing specification, or system of record, then compare the model’s behavior with that rule. A call that classifies free-form customer intent may look trivial in a dashboard while still depending on ambiguity, context and vocabulary that no one has defined. Replacing it with a small list of keywords would change the product, not compile it.

Compilation, distillation, caching and wrapping solve different problems

Distillation is a training technique, not ordinary code generation. The original knowledge-distillation paper describes transferring behavior from a cumbersome model or ensemble to a smaller model that is easier to deploy. The result is still a learned model with an approximation problem. Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network.

A cache also has a narrower job. RFC 9111 describes a cache as storing a response and reusing it for a later request when its reuse rules and cache key allow it. RFC 9111, HTTP Caching. A semantic cache may use an embedding or similarity threshold instead of an exact key, but it still reuses an earlier answer. It does not reveal the rule behind that answer, and a bad key can apply a response to the wrong request.

The distinction matters operationally. A compiled routing rule can be inspected, changed by policy version, and exercised by ordinary unit tests. A cached or distilled system still needs model-aware monitoring. A wrapper may improve output shape or reliability, but the model remains a dependency and its uncertainty remains part of the product.

Worked example with a fulfillment routing rule

Suppose a service receives a structured order record and needs one output: standard or manual_review. The actual policy is: standard fulfillment is available only for an approved country code, a numeric parcel weight from zero through 20 kilograms, and a product explicitly marked as not requiring special handling. The approved-country list is maintained by the operations team.

An early version may have sent the record to an LLM with instructions to “choose the correct fulfillment route.” Its call site might have been this small:

const route = await llm.classify({
  instruction: "Return standard or manual_review using the fulfillment policy.",
  input: order,
});

Once the policy is confirmed, the service can replace the decision with code and a versioned country list:

function fulfillmentRoute(order: {
  countryCode?: string;
  weightKg?: number;
  specialHandling?: boolean;
}): "standard" | "manual_review" {
  const country = order.countryCode?.trim().toUpperCase();
  const allowedCountries = new Set(["AT", "BE", "DE", "NL"]);

  if (!country || !allowedCountries.has(country)) return "manual_review";
  const weight = order.weightKg;
  if (typeof weight !== "number" || !Number.isFinite(weight)) return "manual_review";
  if (weight < 0 || weight > 20) return "manual_review";
  if (order.specialHandling !== false) return "manual_review";
  return "standard";
}

This example is hypothetical. Its important feature is not the particular countries or weight limit. The policy is explicit, testable and owned by operations. Missing weight or an unknown special-handling flag sends the order to manual review. The country list should be a reviewed configuration artifact with an effective date, not a hidden constant that engineers forget to update. If incoming data later changes from weightKg: 12.5 to a free-text field such as weight: "about twelve kilos", the deterministic rule still exists, but the parsing step needs its own specification or a separate extraction system. Do not silently feed the new text into the old code and claim equivalence.

Test the replacement against exceptions and change

Use the written rule as the oracle. The goal is product-equivalent behavior on the defined contract, not matching every word or mistake the LLM happened to produce.

1. Build a labeled set with ordinary cases, boundaries, invalid values, missing values, conflicting fields and historical incidents. For the example, include lowercase country codes, exactly 20 kilograms, 20.01 kilograms, NaN, negative weight, a newly approved country and an unknown code.

2. Run both the current LLM configuration and the proposed code in shadow mode against the set. Classify each disagreement: a code bug, an unclear requirement, a model error, a data-format change, or a case that should retain the model or go to review.

3. Add every resolved disagreement to regression tests. When policy changes, update the rule and tests together, with a version and effective date. This prevents a new country or exception from becoming an unexplained production regression.

4. Compare the two paths on a protected sample of real traffic without changing user-visible outcomes. Measure unknown-input rate, rule fallback rate, latency, cost and the rate at which a person corrects the decision. Do not send duplicated side effects while shadowing.

5. Release behind a flag, retain a rollback path, and keep an explicit fallback for inputs outside the rule. The fallback might be manual review, a structured form that requests the missing field, or the LLM with additional logging. Which option fits depends on the cost of delay and the consequence of a wrong result.

Current evaluation tooling can support this comparison, but the tool does not supply the specification. For example, OpenAI’s Evals API defines an evaluation as testing criteria plus a data-source schema and supports runs with different model configurations. OpenAI Evals reference, accessed 2026-09-05. A conventional test suite, a spreadsheet of reviewed cases, or an internal evaluation harness can use the same idea: expected behavior must be explicit before a comparison is meaningful.

Inputs that change need a separate decision

Replacing a call is safest when the input contract is stable. Review these changes before expanding code coverage:

• A new field may carry information the original model inferred from prose. Decide whether the field becomes authoritative, optional or a trigger for review.

• A changed unit, locale, enum name or upstream schema can alter a deterministic result even when the business rule has not changed. Validate and version the parser at that boundary.

• A new product, country, customer tier or policy exception may require configuration and tests rather than a code change. Name the owner who approves it.

• A request that appears in the fallback bucket repeatedly is evidence that the rule’s domain is too narrow or that the input format needs redesign. It is not evidence that the code should guess.

The same caution applies to model-generated structured data. JSON schema validation can establish that a field has the expected shape. It cannot establish that a model correctly understood the customer’s language. Keep a deterministic rule downstream only when the fields it receives have adequate validation and the product can handle uncertainty through a confidence threshold, a request for clarification or review.

When leaving the LLM in place is the better choice

Keep the LLM for a task whose value comes from interpreting novel language, synthesizing varied documents, recognizing an open-ended intent, or explaining a result in the user’s own words. In those cases, make the boundary explicit: use conventional code for authentication, authorization, calculations, database changes, policy enforcement and irreversible actions; use the model for the language work that remains.

There is also a middle ground. A rule can handle known cases and send only unknown or ambiguous inputs to the LLM or a human reviewer. This reduces model calls without pretending that the remaining language problem has a fixed rule. Track the fallback reasons. If one class recurs, it may become the next candidate for a specified code path.