
> "Every Java developer eventually ends up adding AI to their Spring Boot app. The question is whether they understand what they're actually doing."

Let me be honest with you.

Three weeks ago, if someone asked me to "integrate an LLM into a backend service," I would've Googled it, copy-pasted some OpenAI SDK code, and called it a day. I'd used LLMs as tools, but I hadn't really *understood* them.

Then I had an interview coming up for a role that explicitly said: *"Experience integrating LLMs into backend services, RAG pipelines, AI agent workflows."*

So I did what any slightly panicked developer does — I went deep. Really deep. For a week straight.

This is what I actually learned. Not the hype. Not the tutorials that hand you a `ChatGPT` wrapper and call it "AI engineering." The real stuff.

---

## First, the mental model that changes everything

Here's the thing nobody says out loud:

**An LLM is just a stateless REST endpoint.**

That's it. You send text in. You get text out. No memory. No state. Every single request is independent — if you want the model to "remember" your previous messages, you literally re-send them every time.

```java
// This is ALL an LLM call is, underneath everything:
POST https://api.openai.com/v1/chat/completions
{
  "model": "gpt-4o",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user",   "content": "What is Kafka?" }
  ]
}
// → { "choices": [{ "message": { "content": "Kafka is a distributed..." } }] }
```

Once I got this, everything else clicked. All the "magic" of ChatGPT remembering your conversation? Your client is just re-sending the full chat history on every message. The context window limit everyone talks about? That's the maximum size of that history array.

This is very Kafka-like, actually. The LLM is a stateless consumer. You're the one managing state.

---

## The question I was getting wrong: RAG vs Fine-tuning

I used to think these were the same category of thing. They're not.

**Fine-tuning** = surgery on the model. You take a pre-trained model and continue training it on your data. The knowledge gets baked into the weights. Expensive, slow, and now your data is permanently inside a model you can't easily update.

**RAG (Retrieval-Augmented Generation)** = Google search + copy-paste, but for AI. When the user asks a question, you search your own documents for relevant chunks, then paste them into the prompt alongside the question.

```
User: "What's the total AWS spend in Q3?"

Without RAG:
Prompt: "What's the total AWS spend in Q3?"
LLM: *makes something up* ← HALLUCINATION

With RAG:
Step 1: Search your invoice database → find 5 relevant invoices
Step 2: Prompt: "Given these invoices: [invoice data]... What's the AWS spend in Q3?"
LLM: "Based on the provided invoices, AWS spend in Q3 was $18,420." ← GROUNDED
```

The killer analogy: **RAG is to LLMs what a search engine is to a student writing an essay.** You don't memorize the entire internet — you look things up when you need them.

For anything with private, changing data — invoices, logs, customer records, your codebase — RAG is almost always the right answer. Fine-tuning is for when you want the model to write in a specific style or master a very niche task format.

I learned this the hard way when I built VendorIQ, my invoice automation side project. I initially thought about fine-tuning. Then I realized: invoices change every month. Fine-tuning would require retraining every month. RAG let me just add new invoices to a vector database and they were instantly queryable. Done.

---

## Vector embeddings — the part that sounds scary but isn't

"Embeddings" sounds intimidating. It's not.

An embedding is just a list of numbers that represents the *meaning* of text. Similar meanings → similar numbers.

```
"dog"   → [0.2, 0.8, 0.1, 0.9, ...]  // 1536 numbers
"puppy" → [0.21, 0.79, 0.11, 0.88, ...] // very close to "dog"
"car"   → [0.9, 0.1, 0.7, 0.2, ...]  // completely different
```

When you do a vector search, you're asking: "which documents have numbers closest to my query's numbers?" Cosine similarity measures the angle between two vectors — the smaller the angle, the more similar the meaning.

Why not just use `LIKE '%query%'` in SQL? Because that matches *words*, not *meaning*. A user asking "vehicle repair cost" won't match a document that says "car maintenance expense" with SQL. Vector search catches it because both phrases end up with similar embeddings.

In Java with Spring AI, this is actually straightforward:

```java
@Service
public class RagService {
    @Autowired VectorStore vectorStore;
    @Autowired ChatClient chatClient;

    public String answer(String question) {
        // 1. Find relevant docs (vector similarity search)
        List<Document> docs = vectorStore.similaritySearch(
            SearchRequest.query(question).withTopK(5)
        );

        // 2. Build context string
        String context = docs.stream()
            .map(Document::getContent)
            .collect(Collectors.joining("\n---\n"));

        // 3. Ask LLM with context
        return chatClient.prompt()
            .system("Answer ONLY based on the provided context. If unsure, say so.")
            .user("Context:\n" + context + "\n\nQuestion: " + question)
            .call()
            .content();
    }
}
```

That's a working RAG service. Not much more to it.

---

## AI agents — where things get interesting (and where they break)

An "agent" sounds like it should be some complex autonomous robot. In practice, it's a loop.

**ReAct pattern:** the agent alternates between *Reasoning* (thinking about what to do) and *Acting* (calling a tool), then observes the result and repeats.

```
User: "Is our Cloud SQL spend above budget this month?"

REASON: "I need to get current Cloud SQL cost"
ACT:    call getCost(service="CloudSQL") → $4,200
OBSERVE: "Got $4,200. Now I need the budget."
REASON: "I need to check the budget"
ACT:    call getBudget(category="CloudSQL") → $5,000
REASON: "$4,200 < $5,000. Within budget."
ANSWER: "Cloud SQL is at $4,200, within your $5,000 budget."
```

The model doesn't "know" how to call your functions. You describe them in JSON schemas, the model decides which one to call and with what arguments, your code actually executes them, and you feed the result back to the model.

```java
// Spring AI function calling
@Bean
@Description("Get monthly cost for a GCP service")
public Function<CostRequest, CostResponse> getGcpCost() {
    return req -> new CostResponse(req.service(), billingService.getCost(req.service()));
}

// The LLM will call this when it decides it needs cost data
String answer = chatClient.prompt()
    .user("Is Cloud SQL over budget?")
    .functions("getGcpCost", "getBudget")
    .call()
    .content();
```

**But here's where most tutorials stop and real life begins.**

Agents fail. A lot. The most common failure modes I ran into:

1. **Infinite loops** — the model keeps calling tools, never concludes. Fix: set a max iterations limit.
2. **Hallucinated function arguments** — the model invents parameter values that don't exist in your system. Fix: strict JSON schema validation.
3. **Context window overflow** — long conversations + tool results fill up the context. Fix: summarize older turns, truncate tool outputs.
4. **Non-determinism** — same input produces different tool call sequences. Fix: `temperature=0` for agents doing real work.

The lesson: for anything touching production systems, agents need guardrails. The agent can *suggest* a remediation action. A human should *approve* it before it runs.

---

## LangChain vs LangGraph — why the difference actually matters

LangChain is great for simple pipelines: step 1 → step 2 → step 3 → done. Linear flows.

But what if step 2 fails and you need to retry? What if the result of step 3 determines whether you go back to step 1 or continue to step 4? LangChain can't do this cleanly. **Chains are linear. Real problems aren't.**

LangGraph treats your agent workflow as a directed graph where nodes are functions and edges are transitions — and crucially, edges can loop back. This enables:

- Retry logic (failed analysis → try again)
- Conditional branching (P1 severity → page on-call, P2 → Slack notification)
- Multi-step reasoning with cycles

For the cloud incident triage use case I've been studying: you need an agent that reads logs, classifies severity, potentially re-reads with more context if classification is ambiguous, then routes to the right action. That's a graph, not a chain.

---

## The Kafka + AI combo nobody talks about

This is where my background actually becomes an advantage.

Most AI tutorials show synchronous request-response: user asks question → LLM answers → done. But at enterprise scale with thousands of tenants, you can't synchronously block a thread waiting 3-5 seconds for an LLM response on every operation.

The pattern I'd use:

```
Incident alert fires
    ↓
Kafka topic: "cloud-incidents"
    ↓
AI triage consumer (Spring Boot)
    ├── Calls LLM to analyze logs → structured JSON result
    └── Publishes to "remediation-actions" topic
         ↓
    Remediation consumer
    ├── P1 → request human approval → wait for approval event
    └── P2/P3 → auto-execute safe actions
```

Every step is async. The LLM call is just another microservice in your event-driven architecture. Failures are handled by Kafka's consumer retry mechanism. High load? Scale the consumer group. It's the same patterns we already know — just with an LLM in the middle.

This is why backend engineers who understand distributed systems have an edge in AI engineering. The AI part is honestly the easier part once you understand it. The hard part is building the infrastructure around it correctly.

---

## On hallucinations — the problem that never fully goes away

LLMs hallucinate. They generate plausible-sounding text that's factually wrong. This isn't a bug that will be fixed — it's a fundamental property of how they work. They're predicting the most likely next token, not looking up facts in a database.

You can't eliminate hallucinations. You can *reduce and contain* them:

1. **Grounding** — always provide the actual data (RAG). Don't ask "what's the AWS cost" — provide the actual cost data and ask the model to interpret it.
2. **Force citations** — instruct the model to always cite the source document. If it can't cite, it should say so.
3. **Structured output + validation** — force JSON output, validate the schema. If parsing fails, retry or use a fallback.
4. **Temperature = 0** for factual tasks. Higher temperature = more creativity = more hallucination.

The mindset shift: **treat LLM output as untrusted user input**. Validate it before acting on it, exactly like you validate API request bodies.

---

## Token economics — why this matters at SAP scale

One LLM call at 2,000 tokens costs basically nothing. Ten thousand tenant requests per day at 2,000 tokens each is $100+/day — $3,000/month — before output tokens.

The optimization that surprised me most: **prompt caching**. If your system prompt is 800 tokens and you send it on every request, that's 800 × 10,000 = 8 million input tokens/day just for the system prompt. With Anthropic's prompt caching, repeated static content is cached at 10% of the normal cost. That alone cuts a significant chunk of cost.

The other one: **model routing**. Use a cheap model (Claude Haiku, GPT-4o mini) for simple tasks like severity classification or intent detection. Use the expensive model only for the actual synthesis/reasoning step. Most requests hit the cheap model; only complex ones escalate.

```java
// Smart routing
public String route(String task, String content) {
    if (isSimpleClassification(task)) {
        return cheapClient.prompt().user(content).call().content();  // $0.25/1M
    }
    return smartClient.prompt().user(content).call().content();      // $3/1M
}
```

Combined, these two patterns can cut 60-70% of LLM costs without any quality degradation.

---

## What I actually think about all this

AI engineering is not magic. It's backend engineering with a probabilistic component in the middle.

The engineers who will do this well aren't the ones who read the most papers on transformers or memorized every benchmark. They're the ones who understand distributed systems, know how to design for failure, and treat LLM calls the same way they treat any external API call — with retries, timeouts, circuit breakers, fallbacks, and validation.

The LLM is just another service. An expensive, occasionally unreliable, sometimes brilliant service that you need to engineer around carefully.

If you've been building microservices with Kafka and Spring Boot, you're more prepared for AI engineering than you probably realize. The mental models are the same. The tools are just newer.

That's the thing nobody tells you when they post about "AI developers" making 10x salaries. The actual skill isn't prompt engineering or knowing which model benchmarks highest. It's building reliable systems — which is what we've been doing all along.

---

*I've been documenting all of this as I learn — the concepts, the code patterns, the production gotchas. If any of this resonated, follow along. More coming.*

*Built something with LLMs recently? I'd actually love to hear what broke first.*


Archive: 2026-8

I Spent a Week Going Deep on LLMs. Here's What Actually Matters.