Building a RAG Chatbot with Claude API and Supabase pgvector

RAG (Retrieval-Augmented Generation) is the most practical pattern for building AI assistants over your own data — documentation, support articles, product catalogues, legal documents. Instead of fine-tuning Claude on your data, you retrieve relevant content at query time and include it in the context. This guide builds a complete, production-ready RAG pipeline with Supabase pgvector and Claude.

Architecture Overview

The RAG pipeline has two phases. Ingestion (runs once per document): chunk the document, generate embeddings for each chunk, store chunks and embeddings in Supabase. Query (runs on every chat message): embed the user's query, retrieve the top-k most similar chunks, pass the chunks as context to Claude with prompt caching enabled, stream the response to the user.

Setting Up Supabase pgvector

Enable the pgvector extension in your Supabase project and create the documents and chunks tables. The embedding column stores the vector — set the dimension to match your embedding model (1024 for voyage-3, 1536 for OpenAI text-embedding-3-small).

-- Enable pgvector
create extension if not exists vector;

-- Documents table
create table documents (
  id uuid primary key default gen_random_uuid(),
  title text not null,
  source_url text,
  created_at timestamptz default now()
);

-- Chunks table with embeddings
create table document_chunks (
  id uuid primary key default gen_random_uuid(),
  document_id uuid references documents(id) on delete cascade,
  content text not null,
  embedding vector(1024), -- voyage-3 dimension
  chunk_index integer not null,
  created_at timestamptz default now()
);

-- IVFFlat index for approximate nearest-neighbour search
create index on document_chunks
  using ivfflat (embedding vector_cosine_ops)
  with (lists = 100);

-- Similarity search function
create or replace function match_chunks(
  query_embedding vector(1024),
  match_count int default 5,
  min_similarity float default 0.5
)
returns table (
  id uuid,
  content text,
  similarity float
)
language sql stable as $$
  select
    id,
    content,
    1 - (embedding <=> query_embedding) as similarity
  from document_chunks
  where 1 - (embedding <=> query_embedding) > min_similarity
  order by embedding <=> query_embedding
  limit match_count;
$$;

Document Chunking

Split documents into overlapping chunks before embedding. Overlap ensures that content spanning a chunk boundary isn't lost in retrieval. A recursive character splitter that respects paragraph and sentence boundaries produces better chunks than splitting at fixed character counts.

// lib/chunker.js
export function chunkText(text, chunkSize = 500, overlap = 50) {
  const words = text.split(/\s+/);
  const chunks = [];

  for (let i = 0; i < words.length; i += chunkSize - overlap) {
    const chunk = words.slice(i, i + chunkSize).join(' ');
    if (chunk.trim()) chunks.push(chunk);
    if (i + chunkSize >= words.length) break;
  }

  return chunks;
}

// Prefer splitting at paragraph boundaries
export function chunkByParagraph(text, maxWords = 500) {
  const paragraphs = text.split(/\n{2,}/);
  const chunks = [];
  let current = [];
  let wordCount = 0;

  for (const para of paragraphs) {
    const paraWords = para.split(/\s+/).length;
    if (wordCount + paraWords > maxWords && current.length > 0) {
      chunks.push(current.join('\n\n'));
      current = [];
      wordCount = 0;
    }
    current.push(para);
    wordCount += paraWords;
  }

  if (current.length > 0) chunks.push(current.join('\n\n'));
  return chunks;
}

Generating and Storing Embeddings

Use Voyage AI's voyage-3 model for embeddings — it's optimised for retrieval tasks and pairs well with Claude. Batch embed chunks to stay within rate limits. Store each chunk with its document reference and chunk index for attribution.

// lib/embed.js
import { createClient } from '@supabase/supabase-js';

const supabase = createClient(
  process.env.SUPABASE_URL,
  process.env.SUPABASE_SERVICE_KEY,
);

async function embedBatch(texts) {
  const res = await fetch('https://api.voyageai.com/v1/embeddings', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.VOYAGE_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ model: 'voyage-3', input: texts }),
  });
  const { data } = await res.json();
  return data.map((d) => d.embedding);
}

export async function ingestDocument(title, text, sourceUrl) {
  // Create document record
  const { data: doc } = await supabase
    .from('documents')
    .insert({ title, source_url: sourceUrl })
    .select()
    .single();

  const chunks = chunkByParagraph(text);
  const BATCH_SIZE = 20;

  for (let i = 0; i < chunks.length; i += BATCH_SIZE) {
    const batch = chunks.slice(i, i + BATCH_SIZE);
    const embeddings = await embedBatch(batch);

    await supabase.from('document_chunks').insert(
      batch.map((content, j) => ({
        document_id: doc.id,
        content,
        embedding: embeddings[j],
        chunk_index: i + j,
      })),
    );
  }

  return doc.id;
}

Retrieval: Finding Relevant Chunks

At query time, embed the user's question and run a cosine similarity search against the chunks table. The match_chunks function returns the top-k most relevant chunks above a similarity threshold. Retrieve 5–8 chunks for most queries — enough context without exceeding what Claude can use effectively.

// lib/retrieve.js
export async function retrieveContext(query, matchCount = 6) {
  // Embed the query using the same model as the documents
  const [queryEmbedding] = await embedBatch([query]);

  const { data: chunks, error } = await supabase.rpc('match_chunks', {
    query_embedding: queryEmbedding,
    match_count: matchCount,
    min_similarity: 0.5,
  });

  if (error) throw error;

  // Join chunks into a single context string
  return chunks
    .map((c, i) => `[Source ${i + 1}]\n${c.content}`)
    .join('\n\n---\n\n');
}

Generating with Claude + Prompt Caching

The retrieved context goes into a cached system prompt block. Because the same document chunks are frequently retrieved for similar queries, prompt caching means you pay full price for the first retrieval of a context window and 90% less for subsequent hits within the 5-minute cache window — essential for cost control on a high-traffic chatbot.

// lib/rag.js
import Anthropic from '@anthropic-ai/sdk';
import { retrieveContext } from './retrieve';

const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

export async function ragQuery(userMessage, conversationHistory = []) {
  const context = await retrieveContext(userMessage);

  const stream = await client.messages.stream({
    model: 'claude-sonnet-4-6',
    max_tokens: 1024,
    system: [
      {
        type: 'text',
        text:
          'You are a helpful assistant. Answer questions using ONLY the ' +
          'provided context. If the context does not contain the answer, ' +
          'say "I don\'t have information about that in my knowledge base." ' +
          'Cite the source number when referencing specific information.',
      },
      {
        type: 'text',
        text: `CONTEXT:\n\n${context}`,
        cache_control: { type: 'ephemeral' }, // cache the retrieved context
      },
    ],
    messages: [
      ...conversationHistory,
      { role: 'user', content: userMessage },
    ],
  });

  return stream;
}

The Complete Route Handler

Wire retrieval and generation together in a streaming Route Handler. The client gets streamed tokens as Claude generates the response — latency is dominated by the Supabase vector search (typically 50–150ms) plus the Claude streaming start time.

// app/api/rag-chat/route.js
import { ragQuery } from '@/lib/rag';

export async function POST(req) {
  const { message, history = [] } = await req.json();

  const stream = await ragQuery(message, history);

  const readable = new ReadableStream({
    async start(controller) {
      try {
        for await (const chunk of stream) {
          if (
            chunk.type === 'content_block_delta' &&
            chunk.delta.type === 'text_delta'
          ) {
            controller.enqueue(
              new TextEncoder().encode(chunk.delta.text),
            );
          }
        }
      } finally {
        controller.close();
      }
    },
  });

  return new Response(readable, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

Improving Retrieval Quality

Pure semantic search misses exact keyword matches — product codes, error messages, and proper nouns. Add PostgreSQL full-text search alongside vector search and combine the scores with Reciprocal Rank Fusion (RRF). This hybrid approach consistently outperforms either search method alone and requires no additional infrastructure beyond what pgvector already provides.

Monitoring and Evaluation

Log every RAG query with: the user message, the retrieved chunk IDs, the Claude response, and response.usage.cache_read_input_tokens versus cache_creation_input_tokens. Monitoring cache hit rates tells you whether similar queries are clustering (good — low cost) or each query hits completely different chunks (may indicate your chunking or retrieval needs tuning). A cache hit rate above 60% on a production chatbot is a good target.