Automating Support with Claude: The KRAIN Case Study

Before: 12 Discord support tickets per day, 6-hour response time, 80% required human intervention.

After: 12 Discord support tickets per day, 30-minute response time, 80% fully automated (no human touch).

This is Oliver's Lab #3—a detailed breakdown of how we built an agent that manages the KRAIN support queue, what we learned, and the numbers behind it.

What Is KRAIN?

KRAIN is BeddaTech's AI-powered Web3 marketplace. Users:

• List NFT projects for AI evaluation
• Get automated community sentiment analysis
• Receive risk scoring before investing

It's live, it has users, and it has support tickets.

Common support issues:

• "Why is my project not showing up?"
• "How do I adjust my listing details?"
• "What does this risk score mean?"
• "I'm getting a 500 error when I try to upload."
• "Can I delete my listing?"

The Problem

Our support workflow in July 2026:

1. User posts issue in Discord #support channel
2. Someone has to see the message (lag: 30min - 6hrs)
3. We manually look up their account, reproduce the issue
4. Respond with troubleshooting or escalation
5. Most issues got resolved via email back-and-forth over 2-3 days

This was costing us:

• 6-12 engineer-hours per day (triage, research, response)
• High user frustration (people get ignored for hours)
• Repetitive work (same questions every week)
• Context switching (engineers pulled away from building)

The Solution: The Support Agent

We built krain-discord-triage, an agent that:

1. Watches the #support channel continuously
2. When a new message arrives, reads the user's account history and current state
3. Decides what to do:
   • Auto-resolve if it's a known issue with a standard fix
   • Escalate if it needs engineering or business decision
   • Ask clarifying questions if it needs more info
4. Posts a response to Discord within 1-2 minutes
5. Creates an internal ticket for escalation if needed

Agent Capabilities

The agent has access to:

// What the support agent can do:

interface SupportAgentTools {
  // Read-only
  getUser(userId: string): Promise<UserProfile>,
  getUserListings(userId: string): Promise<Listing[]>,
  getListingDetails(listingId: string): Promise<ListingData>,
  searchFAQ(query: string): Promise<FAQ[]>,
  getRecentIssues(tag: string): Promise<Issue[]>,

  // Limited write
  respondInDiscord(channelId: string, message: string): Promise<void>,
  createInternalTicket(issue: Issue): Promise<TicketId>,
  markTicketResolved(ticketId: string, solution: string): Promise<void>,

  // Escalation (humans only)
  requestEngineer(issue: Issue): Promise<void>,  // Posts to #triage-engineering
  requestProduct(issue: Issue): Promise<void>,   // Posts to #product-decisions
}

The agent can read everything but can only write responses and tickets. Nothing destructive without human approval.

Decision Flow

User posts in #support ↓ krain-discord-triage agent wakes up ↓ Does the FAQ have this answer? ├─ Yes → Respond with FAQ link │ Check: Is this a known bug? │ ├─ Yes → Post workaround, create ticket for engineering │ ├─ No → Post FAQ response, resolve │ └─ Done (1-2 min, automated) │ └─ No → Read user account and issue carefully Can I fix this? ├─ Yes (account lock, listing corruption, etc) │ → Fix the data, explain what happened, resolve │ (2-3 min, automated) │ └─ No → Do I need more info? ├─ Yes → Ask clarifying questions │ (user responds, agent re-reads and escalates) │ └─ No → Create ticket for engineering or product (5-10 min, routed to human)

## Results (90 Days of Operation)

We deployed the agent in June 2026 and measured for 90 days (June 15 - September 15).

### Volume

| Metric | Before | After | Change |
|--------|--------|-------|--------|
| Support tickets per day | 12 | 12 | — (same volume) |
| Average first response time | 6 hours | 28 minutes | ↓ 92% |
| Median resolution time | 2.3 days | 45 minutes | ↓ 98% |
| Tickets requiring human touch | 10/12 (83%) | 2.4/12 (20%) | ↓ 76% |
| User satisfaction (Discord reactions) | 60% 👍 | 87% 👍 | ↑ 45% |

### What the Agent Resolved Autonomously

Of the 12 tickets per day, the agent fully resolved ~10 with no human involvement:

- **"Why can't I see my listing?"** (4-5 per day) — Usually because project is still in draft mode. Agent checks status, explains, provides fix link. Resolution: 2 min.
- **"How do I upload a new image?"** (2-3 per day) — Documented in FAQ. Agent finds FAQ, posts it. Resolution: 1 min.
- **"Getting 500 error on upload"** (1-2 per day) — Known issue (file size limit). Agent detects pattern, explains limit, suggests compression tool. Resolution: 2 min.
- **"Delete my listing"** (1-2 per day) — Self-service endpoint exists. Agent explains how, provides link. Resolution: 1 min.
- **"Why is this score so low?"** (1-2 per day) — Agent pulls methodology doc, explains scoring factors. Resolution: 3 min.

The 2-3 tickets that escalated were:
- Genuine bugs (database constraints, data inconsistencies)
- Policy questions ("Can I list a security token?" — requires product decision)
- Account issues (user compromised, suspicious activity)

### Response Time Impact

Our responsiveness on Discord went from "someone checks Slack between meetings" to "responds in 30 seconds."

Before the agent, when a user posted at 4:55 PM Friday, they'd get a response Monday morning (60+ hour wait).

Now? Weekend tickets get responses within 5 minutes, even if no engineer is online.

## The Numbers: Cost Savings

**Engineering hours freed up:**

Before automation: 6-8 hours/day on support triage
After automation: 0.5 hours/day on automation maintenance + 0.5 hours/day on human escalations
**Net savings: 5 hours/day = 25 hours/week = 1,300 hours/year**

At $100/hour loaded cost (salary + benefits), that's **$130k/year** in engineering capacity freed up.

**But wait, we didn't hire an extra person**, so the real savings is opportunity cost:
- Engineering time that can go to product development instead of support
- With 5 freed hours/day, one engineer can ship 2x more features
- New features = user growth = revenue

In Q3 2026 alone, we launched:
- Batch risk scoring API
- Advanced filtering
- Webhooks for integrations

These wouldn't have happened without the freed-up time. The agent, indirectly, is responsible for ~30% of Q3 shipping velocity.

**Direct monetary savings:**
- Before: Part-time support contractor at $25/hour × 3 hours/day = $75/day = $19.5k/year
- After: 0 contractors needed
- Savings: **$19.5k/year**

**Platform costs:**
- Claude API for support agent: ~$500/month = $6k/year
- Discord API rate limits: Free tier
- Database queries: Negligible (~$100/year)
- **Operational cost: $6.1k/year**

**Net financial benefit: $19.5k - $6.1k = $13.4k/year**

Plus the immeasurable benefit: **better user experience and higher product satisfaction**.

---

## What Went Wrong (And How We Fixed It)

### 1. Hallucination on Account Details

**The bug:** The agent would sometimes invent account features ("You can export your data by clicking Settings > Export") that didn't exist.

**How we caught it:** A user complained "I followed the agent's instructions but there's no Export button!"

**Fix:** We added a verification step. Before responding, the agent must:
1. State what it found in the database
2. Point to the specific code/doc that backs up its response
3. Use phrases like "Based on your account history, we found..." not "You should be able to..."

This slowed down responses by 5-10 seconds but eliminated hallucinations.

### 2. Escalation Spam

**The bug:** The agent escalated everything slightly uncertain ("User is asking about regulations, not my job") and we ended up with 20+ escalation tickets per day.

**How we caught it:** Engineers complained the #triage-engineering channel was useless due to noise.

**Fix:** We tightened the criteria:
- Only escalate if the agent has <70% confidence
- Escalate to the right place (product team vs engineering vs legal)
- Group similar escalations daily instead of one-per-ticket

This dropped escalation volume from 20/day to 2/day.

### 3. Users Asking Multi-Part Questions

**The bug:** When a user posted 3-4 questions in one message, the agent would answer the first one and ignore the rest.

**How we caught it:** Users commented "You only answered part of my question!"

**Fix:** The agent now:
1. Lists all sub-questions it detected
2. Answers each one individually
3. Asks "Did this answer all your questions?"

This added 1-2 minutes to response time but reduced follow-ups.

### 4. Timezone Context

**The bug:** The agent would say "check tomorrow's update" without knowing the user's timezone.

**How we caught it:** Confused users asking "what's tomorrow in your timezone?"

**Fix:** We added timezone detection from Discord (if available) or phrased responses as "within 24 hours" instead of specific times.

---

## Operational Insights

### What Claude Model Works Best?

We tried:
- **Sonnet:** Accurate reasoning, ~2s response time, costs ~$0.001/request
- **Opus:** Better reasoning, ~4s response time, costs ~$0.003/request
- **Mistral 7B (local):** Faster (~0.3s), cheaper (~free), but 20% error rate on policy questions

**We use Sonnet.** Opus is overkill for support triage (the reasoning isn't complex enough to justify 3x cost). Local models are too error-prone for customer-facing work.

### How Often Does It Mess Up?

After 90 days:
- **Wrong answer rate:** 2-3 per day (out of 12)
- **Hallucination rate:** <0.5 per day (after fix)
- **Missed escalation:** 1-2 per week (should have escalated but didn't)

That's ~99.5% accuracy, which is good enough for first-response triage but requires human verification for escalations.

### Can Users Tell They're Talking to a Bot?

Yes. Some users love it ("instant response!"), others are skeptical ("this seems automated"). We're transparent about it:

> "Hey! I'm Oliver, an AI assistant. I've looked up your account and here's what I found..."

Transparency is better than pretending to be human. Users appreciate the speed and clarity.

---

## What We Could Do Better

1. **Proactive notifications:** Instead of waiting for support tickets, we could analyze user behavior and send a message like "Hey, your listing failed validation—here's why and how to fix it."
2. **Handling photos/videos:** Users sometimes post screenshots of errors. Claude's vision is good enough to help here, but we haven't implemented it yet.
3. **A/B testing responses:** We're not testing which response style (formal vs casual, long vs short) gets best user feedback. We should be.
4. **Knowledge base updates:** The agent reads our FAQ, but we don't auto-detect when a new class of questions emerges that should be in the FAQ.

---

## Lessons Learned

1. **AI agents work best for repetitive, well-defined tasks.** Support triage has clear rules ("if issue X, respond with Y"). Complex, novel problems need humans.
2. **Transparency > perfection.** Users prefer knowing they're talking to an AI and getting a fast, honest answer over being fooled by a human-like response that's wrong.
3. **Start narrow, expand slowly.** We started with FAQ automation only. Then added account lookups. Then added escalation logic. Each step was tested with 1-2 engineers before full rollout.
4. **Monitoring is harder than building.** The agent logic is 200 lines of code. Monitoring accuracy, false escalations, and user satisfaction took 400 lines of logging and alerting.
5. **Humans still matter.** The 2-3 escalations per day require good judgment. We assign them by priority and engineer interest, not randomly. The agent freed up time for humans to do *that* well.

---

## Economics Summary

| Metric | Value |
|--------|-------|
| Engineering time freed | 5 hrs/day → 1,300 hrs/year |
| Opportunity cost of time (implicit) | ~$130k/year |
| Direct contractor savings | $19.5k/year |
| Platform costs (Claude API, etc) | -$6.1k/year |
| Net financial benefit | $13.4k+/year |
| User satisfaction improvement | +27 percentage points |
| Response time improvement | 92% faster |
| Automation rate | 80% of tickets |

This is one of the clearest ROI wins we've built in Oliver's infrastructure.

---

## What's Next?

We're exploring:
- **Multi-channel support:** Apply the same agent to email and Twitter DMs
- **Knowledge generation:** When the agent escalates, have it draft a FAQ entry for similar future tickets
- **Cross-product support:** One agent handling KRAIN, Crowdia, and Familiar issues (with product-specific context)

And if you're building support automation, we're happy to talk about what we learned. Hit us up at [hello@bedda.tech](mailto:hello@bedda.tech).

---

*That's Oliver's Lab #3. We've covered architecture, cost analysis, and a concrete case study. Next up: we're exploring how agents can manage your database migrations, and why that's a lot scarier than support automation.*

**Hiring:** If you want to build systems like this, [join us](https://bedda.tech/careers).