LLM Classification Is Feature Engineering
The framing that LLM classification is feature engineering hit 81 points on Hacker News today, and it deserves more than a upvote. Most teams building AI integration pipelines treat LLM classifiers as black-box oracles: you throw text at the model, it returns a label, and you ship it. That mental model is costing people real money in API costs, classification errors, and systems that are nearly impossible to debug. The correct mental model is that every prompt you write is a feature engineering decision, and once you see it that way, you start making better tradeoffs.
This post compares two approaches to text classification: prompt-as-feature (using an LLM directly for classification) versus the traditional ML pipeline (embeddings plus a trained classifier, or hand-engineered features fed into something like XGBoost). Both are legitimate. Neither is universally better. But engineers who conflate them, or who reach for the LLM by default without understanding what they're actually doing, end up with systems that are expensive, brittle, and hard to improve.
Here's how to think about the choice.
The Old Way: Traditional Machine Learning Pipelines
Before LLMs were cheap enough to call per-document, the standard approach to text classification was explicit feature engineering. You'd tokenize text, compute TF-IDF scores, extract domain-specific signals (word counts, presence of certain phrases, metadata like document length or source), and feed those into a logistic regression, SVM, or gradient-boosted tree.
The defining characteristic of this approach is that features are explicit and inspectable. You know exactly what the model is looking at. When a classifier mislabels something, you can trace the error back to a specific feature or a gap in training data. You can add a new feature, retrain in minutes, and measure the delta.
Fine-tuned embeddings sit somewhere in the middle. You take a pre-trained transformer, generate dense vector representations of your text, and train a shallow classifier on top of those embeddings. The features are no longer hand-crafted, but the classification head is still trained on your labeled data, and the whole system is optimizable end-to-end with standard ML tooling.
The tradeoffs of traditional pipelines are well understood:
- Requires labeled data. You need enough examples to train a classifier. For rare categories or fast-moving domains, getting that data is expensive.
- Feature engineering is manual labor. Someone has to decide what signals matter, which requires domain knowledge and iteration.
- Brittle to distribution shift. When the input distribution changes, accuracy degrades and you need to retrain.
- Fast and cheap at inference time. Once trained, a logistic regression over embeddings costs almost nothing per call.
The New Way: Prompt-as-Feature with LLMs
When you write a prompt like "Classify the following support ticket as billing, technical, or account issue," you are making feature engineering decisions. You're deciding which categories exist, how they're described, what context the model receives, and implicitly, what signals the model will weight when it generates a label.
The difference from traditional feature engineering is that the features are encoded in natural language and the "model" is a frozen, general-purpose system you don't control. The LLM has already learned from an enormous corpus. Your prompt is steering that learned knowledge toward your specific classification problem.
This is genuinely powerful. For tasks where:
- Labeled training data doesn't exist yet
- The categories require nuanced reasoning that's hard to encode as features
- The classification problem is exploratory and the schema is changing
- You need to ship something in days, not weeks
...the prompt-as-feature approach wins on time-to-value, often by a large margin.
But here's what the "black-box oracle" framing causes people to miss: the prompt is not magic. It's a feature specification written in English, and it has all the same failure modes as traditional feature engineering, plus some new ones.
Where the Two Approaches Diverge
Debuggability
In a traditional ML pipeline, a mislabeled example gives you somewhere to look. You can inspect feature values, examine decision boundaries, and check if the training data contains similar examples. The system is transparent by construction.
With an LLM classifier, a mislabeled example gives you almost nothing. You can guess that the prompt was ambiguous, or that the model's training data had conflicting examples, or that the input hit some edge case in the tokenizer. But you can't inspect the internal features the model used. You can only modify the prompt and re-run, which is iterating blind.
This is the hidden cost of LLM classification feature engineering that most teams don't account for upfront. Debugging becomes prompt archaeology.
Reproducibility
A logistic regression over embeddings is deterministic. Given the same input and the same model weights, you get the same output every time. You can write unit tests, pin a model version, and trust that your CI pipeline catches regressions.
LLM classifiers with temperature > 0 are stochastic. Even at temperature 0, model providers sometimes update their models in ways that change outputs without changing the model name. OpenAI's model versioning documentation acknowledges this explicitly: dated model snapshots exist precisely because the base alias (like gpt-4o) may be updated over time.
If your classification pipeline depends on an LLM, you need to pin to a dated snapshot and build an eval harness that catches label drift when you upgrade. Most teams don't do this until something breaks in production.
Cost and Latency at Scale
At low volumes, LLM classification is often fine on cost. At high volumes, the math changes fast. A typical GPT-4o call for a classification task might cost $0.002 to $0.01 depending on prompt length and output tokens. A fine-tuned embedding model plus a logistic regression classifier costs a fraction of a cent per thousand documents, at lower latency.
For a pipeline processing 100,000 documents per day, that's a difference of potentially $200-$1,000 per day versus pennies. Over a year, the choice of approach is a significant budget line.
The counterargument is that you don't need GPT-4o for classification. Smaller models like gpt-4o-mini or open-source alternatives like Llama 3 bring costs down substantially. But the cost-at-scale argument still generally favors the traditional pipeline once you have enough labeled data to train one.
Adaptability
This is where LLMs win clearly. Adding a new category to a traditional classifier means collecting labeled examples, retraining, and evaluating. Adding a new category to an LLM classifier means updating the prompt. For fast-moving domains where the classification schema evolves frequently, that flexibility is worth a lot.
The Aclif agent CLI framework (posted on HN today with 26 points) is a good example of a system that needs this kind of adaptability: when you're building a grammar layer across dozens of SaaS tools, the categories of intent are constantly expanding. A prompt-based classifier can track that expansion without a retraining cycle.
Direct Comparison on Key Dimensions
| Dimension | LLM Classifier (Prompt-as-Feature) | Traditional ML Pipeline |
|---|---|---|
| Time to first working classifier | Hours | Days to weeks |
| Labeled data required | None (zero-shot) or few examples | Hundreds to thousands |
| Debuggability | Low | High |
| Reproducibility | Moderate (pin model snapshots) | High |
| Cost at scale | High | Low |
| Latency | Higher (network + inference) | Lower |
| Adaptability to schema changes | High | Low |
| Accuracy ceiling on domain tasks | Moderate to high | High (with sufficient data) |
| Feature inspection | None | Full |
The Hybrid Approach Most Teams Should Actually Use
The framing of LLM vs. traditional ML is a bit of a false choice in practice. The pattern that works well at scale is using LLMs to bootstrap labeled data for a traditional pipeline.
Here's the flow: you write a prompt-based classifier, run it over a sample of your data, have humans review and correct a subset of the outputs, and use the resulting labeled dataset to train a fine-tuned embedding classifier. The LLM does the expensive exploration work upfront. The traditional pipeline handles production volume at low cost with high reproducibility.
This is exactly what the original minimallysufficient.com post is gesturing at when it reframes LLM classification as feature engineering: the prompt is a hypothesis about what signals matter. Once you've validated that hypothesis, you can operationalize it in a cheaper, more inspectable form.
Amber Case's interview on why AI has it backwards makes a related point: AI tools often substitute for human judgment in ways that make systems harder to understand and control, rather than augmenting human decision-making. The prompt-as-oracle pattern is a specific instance of this. Treating the prompt as a feature specification, and building evaluation infrastructure around it, keeps humans in the loop in a meaningful way.
When to Use Each Approach
Use LLM classification when:
- You have no labeled data and need to ship quickly
- The classification schema is actively evolving
- The task requires nuanced reasoning that's hard to encode in features
- Volume is low enough that API costs are acceptable
- You're using the LLM output to generate training data for a future traditional classifier
Use a traditional ML pipeline when:
- You have sufficient labeled data (rough threshold: a few hundred examples per class minimum)
- You need deterministic, inspectable outputs
- Volume is high enough that API costs matter
- Latency requirements are strict
- The classification schema is stable
Use a hybrid approach when:
- You're building something new and expect the schema to stabilize over 3-6 months
- You can afford LLM costs during the bootstrapping phase
- You have the engineering capacity to build an eval harness
The Discipline This Requires
Reframing LLM classification as feature engineering has a practical implication: you need to treat prompt changes with the same discipline you'd apply to feature changes in a traditional ML system. That means versioning prompts, running evals before deploying changes, and tracking accuracy metrics over time.
Most teams skip this because prompts feel informal. They're just text. But a prompt change is a feature change, and a feature change can silently degrade accuracy on a subset of inputs while improving it on the cases you're actively looking at. The eval harness is mandatory, not optional.
Engineers who've built production ML systems know this instinctively from the traditional pipeline world. The discipline transfers directly to LLM classification. The mistake is assuming that because LLMs feel different, the engineering rigor around them can be lower.
It can't. The failure modes are just less obvious until something goes wrong in production.