AI/LLM Integrations in Production Apps: What They Don't Tell You

Everyone's adding AI features to their apps. Most are doing it wrong. After shipping LLM-powered features to production serving thousands of users, I've learned that integrating AI into real applications is less about the model and more about the infrastructure around it. The demo is easy. Production is where you find out if you know what you're doing.

The Cost Problem Nobody Talks About

Your first surprise will be the bill. GPT-4 costs about $0.03 per 1K input tokens and $0.06 per 1K output tokens. That sounds cheap until you realize a typical conversation can easily hit 5K-10K tokens with context. If 1,000 users each have 10 conversations per month, you're looking at $1,500-$3,000 just in API costs. This scales fast.

The solution isn't to avoid AI features—it's to instrument everything and make cost a first-class metric. Track token usage per request, per user, per feature. Set up alerts when costs spike. Use cheaper models (GPT-3.5, Claude Haiku) for simpler tasks and reserve GPT-4 for complex reasoning.

// Track costs in real-time with custom metrics
import { trackMetric } from './monitoring';

async function callLLM(prompt: string, userId: string) {
  const startTime = Date.now();
  
  const response = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [{ role: 'user', content: prompt }],
  });

  const inputTokens = response.usage?.prompt_tokens || 0;
  const outputTokens = response.usage?.completion_tokens || 0;
  const cost = (inputTokens * 0.00003) + (outputTokens * 0.00006);

  await trackMetric({
    metric: 'llm.cost',
    value: cost,
    tags: { userId, model: 'gpt-4' },
  });

  await trackMetric({
    metric: 'llm.latency',
    value: Date.now() - startTime,
    tags: { userId },
  });

  return response.choices[0].message.content;
}

Streaming Is Non-Negotiable

LLM responses can take 5-15 seconds. That's an eternity in user experience terms. If you're waiting for the full response before showing anything, your users will think your app is broken. Streaming responses is not optional for production applications.

// Server-side streaming endpoint
export async function POST(req: Request) {
  const { prompt } = await req.json();
  
  const stream = await openai.chat.completions.create({
    model: 'gpt-4-turbo-preview',
    messages: [{ role: 'user', content: prompt }],
    stream: true,
  });

  const encoder = new TextEncoder();
  
  const readableStream = new ReadableStream({
    async start(controller) {
      try {
        for await (const chunk of stream) {
          const text = chunk.choices[0]?.delta?.content || '';
          controller.enqueue(encoder.encode(`data: ${JSON.stringify({ text })}\n\n`));
        }
        controller.enqueue(encoder.encode('data: [DONE]\n\n'));
        controller.close();
      } catch (error) {
        controller.error(error);
      }
    },
  });

  return new Response(readableStream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Connection': 'keep-alive',
    },
  });
}

Prompt Versioning and Management

Here's a mistake I see constantly: hardcoding prompts in your application code. When you need to tweak the prompt (and you will), you're redeploying your entire app. Worse, you have no history of what changed or why. Treat prompts like database migrations—version them, store them separately, and make them easy to roll back.

I store prompts in the database with versioning. Each prompt has a key, version number, and the actual template. The application fetches the active version at runtime. This lets product managers iterate on prompts without engineering deploys, and gives you an audit trail when something goes wrong.

CREATE TABLE prompts (
  id UUID PRIMARY KEY,
  key VARCHAR(255) NOT NULL,
  version INTEGER NOT NULL,
  template TEXT NOT NULL,
  variables JSONB,
  is_active BOOLEAN DEFAULT false,
  created_at TIMESTAMP DEFAULT NOW(),
  created_by VARCHAR(255),
  UNIQUE(key, version)
);

CREATE INDEX idx_prompts_active ON prompts(key, is_active) WHERE is_active = true;

Error Handling and Fallbacks

LLM APIs fail. They rate limit, timeout, return malformed responses, or just have bad days. Your error handling needs to be more sophisticated than try/catch and showing a generic error message. Implement exponential backoff, circuit breakers, and meaningful fallbacks.

• Rate limiting: OpenAI has per-minute token limits. Queue requests and implement backpressure instead of failing immediately.
• Timeouts: Set aggressive timeouts (30s max) and fail fast. A slow response is worse than an error message with a retry button.
• Fallback responses: For non-critical features, have pre-written fallback content. Better to show something useful than an error.
• Model fallback: If GPT-4 fails, automatically retry with GPT-3.5. Most users won't notice the quality difference.
• User communication: Be honest. 'AI is experiencing high demand' is better than 'Something went wrong'.

Observability Is Everything

You need visibility into your LLM usage that goes beyond basic logging. Track token usage, latency, error rates, and cost per user. But also track quality metrics—are users regenerating responses? Editing them heavily? Abandoning the feature? These signals tell you if your prompts are actually working.

The hardest part of production LLM features isn't the integration—it's building the infrastructure to make them reliable, cost-effective, and observable. Start with these patterns, measure everything, and iterate based on real usage data. The companies winning with AI aren't using better models; they're using better infrastructure around those models.