
> "Every Java developer eventually ends up adding AI to their Spring Boot app. The question is whether they understand what they're actually doing."

Let me be honest with you.

Three weeks ago, if someone asked me to "integrate an LLM into a backend service," I would've Googled it, copy-pasted some OpenAI SDK code, and called it a day. I'd used LLMs as tools, but I hadn't really *understood* them.

Then I had an interview coming up for a role that explicitly said: *"Experience integrating LLMs into backend services, RAG pipelines, AI agent workflows."*

So I did what any slightly panicked developer does — I went deep. Really deep. For a week straight.

This is what I actually learned. Not the hype. Not the tutorials that hand you a `ChatGPT` wrapper and call it "AI engineering." The real stuff.

---

## First, the mental model that changes everything

Here's the thing nobody says out loud:

**An LLM is just a stateless REST endpoint.**

That's it. You send text in. You get text out. No memory. No state. Every single request is independent — if you want the model to "remember" your previous messages, you literally re-send them every time.

```java
// This is ALL an LLM call is, underneath everything:
POST https://api.openai.com/v1/chat/completions
{
  "model": "gpt-4o",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user",   "content": "What is Kafka?" }
  ]
}
// → { "choices": [{ "message": { "content": "Kafka is a distributed..." } }] }
```

Once I got this, everything else clicked. All the "magic" of ChatGPT remembering your conversation? Your client is just re-sending the full chat history on every message. The context window limit everyone talks about? That's the maximum size of that history array.

This is very Kafka-like, actually. The LLM is a stateless consumer. You're the one managing state.

---

## The question I was getting wrong: RAG vs Fine-tuning

I used to think these were the same category of thing. They're not.

**Fine-tuning** = surgery on the model. You take a pre-trained model and continue training it on your data. The knowledge gets baked into the weights. Expensive, slow, and now your data is permanently inside a model you can't easily update.

**RAG (Retrieval-Augmented Generation)** = Google search + copy-paste, but for AI. When the user asks a question, you search your own documents for relevant chunks, then paste them into the prompt alongside the question.

```
User: "What's the total AWS spend in Q3?"

Without RAG:
Prompt: "What's the total AWS spend in Q3?"
LLM: *makes something up* ← HALLUCINATION

With RAG:
Step 1: Search your invoice database → find 5 relevant invoices
Step 2: Prompt: "Given these invoices: [invoice data]... What's the AWS spend in Q3?"
LLM: "Based on the provided invoices, AWS spend in Q3 was $18,420." ← GROUNDED
```

The killer analogy: **RAG is to LLMs what a search engine is to a student writing an essay.** You don't memorize the entire internet — you look things up when you need them.

For anything with private, changing data — invoices, logs, customer records, your codebase — RAG is almost always the right answer. Fine-tuning is for when you want the model to write in a specific style or master a very niche task format.

I learned this the hard way when I built VendorIQ, my invoice automation side project. I initially thought about fine-tuning. Then I realized: invoices change every month. Fine-tuning would require retraining every month. RAG let me just add new invoices to a vector database and they were instantly queryable. Done.

---

## Vector embeddings — the part that sounds scary but isn't

"Embeddings" sounds intimidating. It's not.

An embedding is just a list of numbers that represents the *meaning* of text. Similar meanings → similar numbers.

```
"dog"   → [0.2, 0.8, 0.1, 0.9, ...]  // 1536 numbers
"puppy" → [0.21, 0.79, 0.11, 0.88, ...] // very close to "dog"
"car"   → [0.9, 0.1, 0.7, 0.2, ...]  // completely different
```

When you do a vector search, you're asking: "which documents have numbers closest to my query's numbers?" Cosine similarity measures the angle between two vectors — the smaller the angle, the more similar the meaning.

Why not just use `LIKE '%query%'` in SQL? Because that matches *words*, not *meaning*. A user asking "vehicle repair cost" won't match a document that says "car maintenance expense" with SQL. Vector search catches it because both phrases end up with similar embeddings.

In Java with Spring AI, this is actually straightforward:

```java
@Service
public class RagService {
    @Autowired VectorStore vectorStore;
    @Autowired ChatClient chatClient;

    public String answer(String question) {
        // 1. Find relevant docs (vector similarity search)
        List<Document> docs = vectorStore.similaritySearch(
            SearchRequest.query(question).withTopK(5)
        );

        // 2. Build context string
        String context = docs.stream()
            .map(Document::getContent)
            .collect(Collectors.joining("\n---\n"));

        // 3. Ask LLM with context
        return chatClient.prompt()
            .system("Answer ONLY based on the provided context. If unsure, say so.")
            .user("Context:\n" + context + "\n\nQuestion: " + question)
            .call()
            .content();
    }
}
```

That's a working RAG service. Not much more to it.

---

## AI agents — where things get interesting (and where they break)

An "agent" sounds like it should be some complex autonomous robot. In practice, it's a loop.

**ReAct pattern:** the agent alternates between *Reasoning* (thinking about what to do) and *Acting* (calling a tool), then observes the result and repeats.

```
User: "Is our Cloud SQL spend above budget this month?"

REASON: "I need to get current Cloud SQL cost"
ACT:    call getCost(service="CloudSQL") → $4,200
OBSERVE: "Got $4,200. Now I need the budget."
REASON: "I need to check the budget"
ACT:    call getBudget(category="CloudSQL") → $5,000
REASON: "$4,200 < $5,000. Within budget."
ANSWER: "Cloud SQL is at $4,200, within your $5,000 budget."
```

The model doesn't "know" how to call your functions. You describe them in JSON schemas, the model decides which one to call and with what arguments, your code actually executes them, and you feed the result back to the model.

```java
// Spring AI function calling
@Bean
@Description("Get monthly cost for a GCP service")
public Function<CostRequest, CostResponse> getGcpCost() {
    return req -> new CostResponse(req.service(), billingService.getCost(req.service()));
}

// The LLM will call this when it decides it needs cost data
String answer = chatClient.prompt()
    .user("Is Cloud SQL over budget?")
    .functions("getGcpCost", "getBudget")
    .call()
    .content();
```

**But here's where most tutorials stop and real life begins.**

Agents fail. A lot. The most common failure modes I ran into:

1. **Infinite loops** — the model keeps calling tools, never concludes. Fix: set a max iterations limit.
2. **Hallucinated function arguments** — the model invents parameter values that don't exist in your system. Fix: strict JSON schema validation.
3. **Context window overflow** — long conversations + tool results fill up the context. Fix: summarize older turns, truncate tool outputs.
4. **Non-determinism** — same input produces different tool call sequences. Fix: `temperature=0` for agents doing real work.

The lesson: for anything touching production systems, agents need guardrails. The agent can *suggest* a remediation action. A human should *approve* it before it runs.

---

## LangChain vs LangGraph — why the difference actually matters

LangChain is great for simple pipelines: step 1 → step 2 → step 3 → done. Linear flows.

But what if step 2 fails and you need to retry? What if the result of step 3 determines whether you go back to step 1 or continue to step 4? LangChain can't do this cleanly. **Chains are linear. Real problems aren't.**

LangGraph treats your agent workflow as a directed graph where nodes are functions and edges are transitions — and crucially, edges can loop back. This enables:

- Retry logic (failed analysis → try again)
- Conditional branching (P1 severity → page on-call, P2 → Slack notification)
- Multi-step reasoning with cycles

For the cloud incident triage use case I've been studying: you need an agent that reads logs, classifies severity, potentially re-reads with more context if classification is ambiguous, then routes to the right action. That's a graph, not a chain.

---

## The Kafka + AI combo nobody talks about

This is where my background actually becomes an advantage.

Most AI tutorials show synchronous request-response: user asks question → LLM answers → done. But at enterprise scale with thousands of tenants, you can't synchronously block a thread waiting 3-5 seconds for an LLM response on every operation.

The pattern I'd use:

```
Incident alert fires
    ↓
Kafka topic: "cloud-incidents"
    ↓
AI triage consumer (Spring Boot)
    ├── Calls LLM to analyze logs → structured JSON result
    └── Publishes to "remediation-actions" topic
         ↓
    Remediation consumer
    ├── P1 → request human approval → wait for approval event
    └── P2/P3 → auto-execute safe actions
```

Every step is async. The LLM call is just another microservice in your event-driven architecture. Failures are handled by Kafka's consumer retry mechanism. High load? Scale the consumer group. It's the same patterns we already know — just with an LLM in the middle.

This is why backend engineers who understand distributed systems have an edge in AI engineering. The AI part is honestly the easier part once you understand it. The hard part is building the infrastructure around it correctly.

---

## On hallucinations — the problem that never fully goes away

LLMs hallucinate. They generate plausible-sounding text that's factually wrong. This isn't a bug that will be fixed — it's a fundamental property of how they work. They're predicting the most likely next token, not looking up facts in a database.

You can't eliminate hallucinations. You can *reduce and contain* them:

1. **Grounding** — always provide the actual data (RAG). Don't ask "what's the AWS cost" — provide the actual cost data and ask the model to interpret it.
2. **Force citations** — instruct the model to always cite the source document. If it can't cite, it should say so.
3. **Structured output + validation** — force JSON output, validate the schema. If parsing fails, retry or use a fallback.
4. **Temperature = 0** for factual tasks. Higher temperature = more creativity = more hallucination.

The mindset shift: **treat LLM output as untrusted user input**. Validate it before acting on it, exactly like you validate API request bodies.

---

## Token economics — why this matters at SAP scale

One LLM call at 2,000 tokens costs basically nothing. Ten thousand tenant requests per day at 2,000 tokens each is $100+/day — $3,000/month — before output tokens.

The optimization that surprised me most: **prompt caching**. If your system prompt is 800 tokens and you send it on every request, that's 800 × 10,000 = 8 million input tokens/day just for the system prompt. With Anthropic's prompt caching, repeated static content is cached at 10% of the normal cost. That alone cuts a significant chunk of cost.

The other one: **model routing**. Use a cheap model (Claude Haiku, GPT-4o mini) for simple tasks like severity classification or intent detection. Use the expensive model only for the actual synthesis/reasoning step. Most requests hit the cheap model; only complex ones escalate.

```java
// Smart routing
public String route(String task, String content) {
    if (isSimpleClassification(task)) {
        return cheapClient.prompt().user(content).call().content();  // $0.25/1M
    }
    return smartClient.prompt().user(content).call().content();      // $3/1M
}
```

Combined, these two patterns can cut 60-70% of LLM costs without any quality degradation.

---

## What I actually think about all this

AI engineering is not magic. It's backend engineering with a probabilistic component in the middle.

The engineers who will do this well aren't the ones who read the most papers on transformers or memorized every benchmark. They're the ones who understand distributed systems, know how to design for failure, and treat LLM calls the same way they treat any external API call — with retries, timeouts, circuit breakers, fallbacks, and validation.

The LLM is just another service. An expensive, occasionally unreliable, sometimes brilliant service that you need to engineer around carefully.

If you've been building microservices with Kafka and Spring Boot, you're more prepared for AI engineering than you probably realize. The mental models are the same. The tools are just newer.

That's the thing nobody tells you when they post about "AI developers" making 10x salaries. The actual skill isn't prompt engineering or knowing which model benchmarks highest. It's building reliable systems — which is what we've been doing all along.

---

*I've been documenting all of this as I learn — the concepts, the code patterns, the production gotchas. If any of this resonated, follow along. More coming.*

*Built something with LLMs recently? I'd actually love to hear what broke first.*


I Spent a Week Going Deep on LLMs. Here's What Actually Matters.


> "The paradox remains: We build in the dark, hoping to create light."

There's something absurd about our situation as developers in 2025. We're building systems to solve human problems while we're still figuring out what it means to be human. It's like trying to write a manual for something you've never fully understood yourself.

I write code, push commits, ship features. But in quiet moments—usually around 3 AM when debugging some API that refuses to behave—I wonder: What are we really building? And more importantly, why?

The existentialists had it figured out, sort of. Camus wrote about the absurd—our desperate search for meaning in a universe that refuses to give us clear answers. We keep building, keep coding, keep shipping, but the "why" remains frustratingly unclear.

Here's what I've noticed: Technology promises to solve everything, but it amplifies everything instead. Social media was supposed to connect us—now we're more isolated than ever. AI was supposed to make us smarter—instead, it's making us question what intelligence even means.

But maybe that's the point.

Maybe the value isn't in having all the answers. Maybe it's in the questioning itself. Every line of code is a small bet against meaninglessness. Every project is saying, "I believe this matters, even if I can't prove it."

The hard work isn't just learning React or understanding algorithms. The hard work is continuing to create while accepting that you might never understand why you're creating. It's coding with conviction while holding uncertainty.

Most programming philosophy talks about clean code, design patterns, best practices. But the deeper question is: How do you write code that feels true to who you are, not just what the market demands?

I don't have the answer. But I know this: authenticity in code, like authenticity in life, isn't about perfection. It's about being honest about the process—messy, uncertain, human.

The universe doesn't care about your deployment pipeline. But the people using what you build do. And sometimes, that's enough reason to keep going.

The paradox remains: We build in the dark, hoping to create light.


The Paradox of Building in the Dark


> "Why do Java developers wear glasses? Because they don’t C#!” — a classic, but still gold.

Hello fellow developer/human-in-tech!

Have you ever merged PDFs manually like some sort of tech caveman, then built and pushed Docker images like a robot stuck in a time loop?

When you’re merging PDF’s manually but pretend it’s all under control.

Same. But fear not — automation is here to save your sanity and weekends too :).

In this blog, I’ll show you how I built a simple Spring Boot PDF Merger, and then automated the build and Docker push process using Jenkins. It’s easy, it’s powerful, and yes, it might just make you feel like a DevOps wizard.

## Part 1: Meet Our Simple PDF Merger

Imagine your manager drops a folder of 50 PDFs on your desk and says, “Hey man, could you merge these PDFs?”

Or you’re doing an M.Tech in Software Engineering at BITS through WILP, and you need to merge 30–40 lecture PDFs together, and you don’t have time.

That’s when you build this:

### PDF Merger (Spring Boot)
A Spring Boot REST API that merges multiple PDF files into one.

**Tech Stack:**
- Java 17
- Spring Boot
- Apache PDFBox
- Maven

**Sample Endpoint:**
POST `/merge` (with multipart/form-data containing `List<MultipartFile>` as input)

**Sample Controller Snippet:**
<div style={{overflowX: 'auto', width: '100%'}}>

```java
@PostMapping("/merge")
public ResponseEntity<byte[]> mergePdfs(@RequestParam("files") List<MultipartFile> files) {
    ByteArrayOutputStream mergedOutput = new ByteArrayOutputStream();
    PDFMergerUtility pdfMerger = new PDFMergerUtility();
    pdfMerger.setDestinationStream(mergedOutput);
    for (MultipartFile file : files) {
        pdfMerger.addSource(file.getInputStream());
    }
    pdfMerger.mergeDocuments(null);
    return ResponseEntity.ok()
        .contentType(MediaType.APPLICATION_PDF)
        .body(mergedOutput.toByteArray());
}
```

</div>
Easy-peasy. Upload some files, get merged PDF. But now we go full DevOps ninja.

## Part 2: Automating the Docker Build & Push

If you automate a mess, you get an automated mess. But at least it’s a fast mess :)

Let’s automate our Docker builds and image push using Jenkins.

**Our Goal:**
- Build Docker image of our PDF Merger API.
- Tag image based on Git branch.
- Push image to Docker Hub.

**Pre-requisites:**
- Jenkins set up on your machine (use Docker or install natively)
- Docker installed
- Docker Hub account (with repo created)
- Jenkins Docker plugin installed

### Step 1: Dockerfile for Spring Boot PDF Merger
```dockerfile
FROM openjdk:17-jdk-slim AS build
WORKDIR /app
COPY . .
RUN ./mvnw package -DskipTests

FROM openjdk:17-jdk-slim
WORKDIR /app
COPY --from=build /app/target/pdf-merger.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]
```
This multi-stage Dockerfile keeps the final image small by copying only the JAR from the build stage.

### Step 2: Jenkinsfile Magic
```groovy
pipeline {
  agent any
  environment {
    IMAGE = "your-dockerhub-username/pdf-merger"
    BRANCH = "${env.GIT_BRANCH}".replaceAll("origin/", "")
  }
  stages {
    stage('Checkout') {
      steps {
        git url: 'https://github.com/your-repo/pdf-merger.git'
      }
    }
    stage('Build') {
      steps {
        sh './mvnw clean package -DskipTests'
      }
    }
    stage('Docker Build & Push') {
      steps {
        sh '''
          docker build -t $IMAGE:$BRANCH .
          echo "Pushing Docker image..."
          docker push $IMAGE:$BRANCH
        '''
      }
    }
  }
}
```
This Jenkins pipeline checks out the repo, builds the project, and pushes the image — all automatically.

### Step 3: Shell Script to Run Anytime
Tired of typing commands? Just run this smart script:

```bash
#!/bin/bash
# Automating Docker build and push with a single command
# Saves you the hassle of doing it manually every time!
DOCKER_IMAGE="your-dockerhub-username/your-repo"
BRANCH_NAME=$(git rev-parse --abbrev-ref HEAD)
echo "Building Docker image for branch $BRANCH_NAME..."
docker build -t $DOCKER_IMAGE:$BRANCH_NAME .
echo "Pushing Docker image to Docker Hub..."
docker push $DOCKER_IMAGE:$BRANCH_NAME
if [ $? -eq 0 ]; then
  echo "Docker image successfully pushed to Docker Hub!"
else
  echo "Something went wrong. Docker image push failed."
  exit 1
fi
```
Place this in a `deploy.sh`, give it `chmod +x` and never worry again.

## Step 4: Automate Everything
Now, just run the Jenkins pipeline or execute the shell script and voila — Jenkins takes care of the rest. The Docker image will be built and pushed, and you can rest easy knowing that this task is no longer your responsibility.

## Final Thoughts
Automation is the best thing to happen to DevOps since the coffee break. It’s not just about saving time; it’s about making your workflow smoother, reducing errors, and freeing up mental space for the fun stuff — like chilling with friends or playing games!

Ready to try this out yourself? Give it a shot, and let me know how it goes. Automation is the future, and I am pretty sure you will never go back to the days of manually pushing Docker images.

Haapppyy Codinngg, and may Jenkins always build and push your Docker images without fail!!

To the Jenkins builds running while I sleep. Cheers.

**Repo:**  
Clone it, fork it, break it, automate it.  
[GitHub - nileshkr17/jenkins-pdf-merger](https://github.com/nileshkr17/jenkins-pdf-merger)

And if you liked this blog, give a double tap and do leave a comment!!

**UI:** <a href="https://jenkins-pdf-merger-latest.onrender.com/api/pdf/home" target="_blank" rel="noopener noreferrer">https://jenkins-pdf-merger-latest.onrender.com/api/pdf/home</a>

**API to merge:** <a href="https://jenkins-pdf-merger-latest.onrender.com/api/pdf/merge" target="_blank" rel="noopener noreferrer">https://jenkins-pdf-merger-latest.onrender.com/api/pdf/merge</a>


Blog

I Spent a Week Going Deep on LLMs. Here's What Actually Matters.

The Paradox of Building in the Dark

Automate All the Things: Docker + Jenkins + Spring Boot Made Easy

Archives