ask the lyrics active

A retrieval-augmented Q&A system over a corpus of hip-hop lyrics. You ask a natural-language question ("which artist mentions guns the most?", "what song does Nas rap from the perspective of a gun?") and it retrieves the relevant lyrics and synthesizes a sourced answer. Live at askthelyrics.com.

Terminology

A few terms used throughout:

• RAG (retrieval-augmented generation): rather than answering from the model's memory, the system first retrieves relevant source text and answers from it, so responses are grounded and citable.
• Embedding: a piece of text turned into a vector (a list of numbers) that captures its meaning; texts with similar meaning land near each other in that space.
• Vector / semantic search: finding text by meaning instead of exact keywords, by comparing embeddings.
• Chunking: splitting long text (here, song lyrics) into smaller passages so each can be embedded and retrieved on its own.
• Query expansion: rewriting a plain-English question into the words the source actually uses (here, hip-hop slang and coded language) so search matches more.
• Semantic cache: caching answers keyed by meaning, so a question close to one already asked returns the stored answer with no recompute.
• Upsert: "update or insert"; write a record, replacing what's there only when the new data scores better.

Pipeline

┌──────────────────────────────────────────────┐
│ DISCOVERY  ·  catalog API                    │
│   resolve an artist's discography            │
└──────────────────────────────────────────────┘
                        │
                        ▼  what songs exist
┌──────────────────────────────────────────────┐
│ FETCHER  ·  httpx + BeautifulSoup            │
│   priority: local cache -> external sources  │
└──────────────────────────────────────────────┘
                        │
                        ▼  store raw lyrics
┌──────────────────────────────────────────────┐
│ PostgreSQL  ·  canonical store               │
│   lyrics, metadata, source + quality scores  │
└──────────────────────────────────────────────┘
                        │
                        ▼  embed
┌──────────────────────────────────────────────┐
│ ChromaDB  ·  vector index                    │
│   chunked-lyric embeddings (local model)     │
└──────────────────────────────────────────────┘
                        │
                        ▼  semantic search
┌──────────────────────────────────────────────┐
│ FastAPI  ·  REST                             │
│   POST /ask   GET /search   GET /stats       │
│   query expansion + semantic cache           │
└──────────────────────────────────────────────┘
                        │
                        ▼
   sanitized Markdown answer, rendered in the browser

Ingestion

Discovery and fetching are separated so "what songs exist" is decided independently of "where do we get the words." A discovery layer queries a music-catalog API to resolve an artist's discography, then a fetcher (httpx + BeautifulSoup, with a Playwright/Chromium path for harder pages) pulls lyrics using a strict source priority (local cache first, then external sources), so re-runs are cheap and the cleanest copy wins.

Everything lands in PostgreSQL as the canonical store: artists, songs, the source each lyric came from, and per-source quality scores.

Quality-ranked sources

When the same song shows up from more than one source, a quality score decides which copy wins, and also determines whether a new fetch is even allowed to overwrite the stored one. Points for lyric length and for structural [Verse]/[Chorus] markers, album metadata, and a release year; penalties for truncation (text that ends in ... or comes back suspiciously short). An upsert only replaces the existing row if the new copy scores higher, so a complete, well-structured version automatically beats a bare or cut-off one, requiring no manual curation.

Indexing & retrieval

Lyrics are chunked on verse/section boundaries (capped ~500 chars), and every chunk is prefixed with its Artist: / Song: context before embedding, so the vector captures attribution as well as content, which sharply improves retrieval when a question names a specific artist. Chunks are embedded with a local embedding model (all-MiniLM-L6-v2 via ChromaDB), which requires no embedding API calls, so indexing is free and offline.

Query routing

It isn't a single RAG loop. An incoming question is first classified into one of 11 types by a regex fast-path for unambiguous cases or a lightweight model for the rest. Each type dispatches to a structurally different retrieval strategy with its own prompt and context budget. On a cache miss, spam-checking and classification run concurrently.

One bit of deliberate prompt engineering: the classifier distinguishes a "concept" question (what an artist says about themselves, such as bragging or claiming greatness) from a "who" question (which artist most references some external topic), because those want completely different retrieval.

Analytical queries

Questions like "who mentions guns the most?" or "who has the biggest vocabulary?" aren't handed to the model at all. They're computed:

• Aggregates rank by match density (songs-with-matches ÷ total songs) instead of raw counts, so a 200-song catalog doesn't automatically dominate. A per-song minimum-distinct-terms threshold, auto-scaled to how many slang/expansion terms the query produced, filters out incidental one-off matches.
• Vocabulary questions read a precomputed artist_stats table (unique words, vocab diversity, words-per-song) and return a ranked Markdown table straight from SQL, defaulting to unique-words-per-song so catalog size is normalized out, and excluding very small catalogs. No model is in the loop, so it's instant and exact.

For abstract "concept" questions ("which rapper thinks they're the best?"), it generates 5–8 short first-person phrases ("I'm the greatest", "can't nobody see me") and runs each as a separate vector search, then dedupes and groups hits by artist before composing the answer.

Caching, three ways

• Query expansion (rewriting a plain question into hip-hop slang / coded language) is fully model-driven (no static slang lists) and cached to disk (LRU, ~10k entries, debounced writes) keyed on a normalized question: punctuation stripped, artist-name variants canonicalized (Jay-Z → jayz, 50 Cent → 50cent) to maximize hits. The query classifier reuses the same store.
• Query embeddings get an in-process LRU. Each embed takes ~350 ms, so repeats are free.
• Answered Q&A pairs live in a separate cosine-space ChromaDB collection; a new question ≥ 0.85 similar to a previous one returns the cached answer with no retrieval and no model call. Even "unanswerable" questions are cached (flagged) so repeats don't re-spend.

API & frontend

A FastAPI service exposes the REST surface (POST /ask, GET /search, GET /stats) plus a CLI for one-off queries. The web UI renders answers as sanitized Markdown in the browser: a strict allowlist preserves basic formatting (bold/italic, lists, tables, code, blockquotes) while raw HTML is escaped and links/images are stripped. There is no frontend framework, just a small bundled Markdown renderer.

Operations

Production runs entirely in Docker as two compose stacks behind a shared web network: an app stack (Caddy with automatic TLS + the FastAPI app + PostgreSQL) and a separate analytics stack. Indexing is deliberately a local-only step: scraping and embedding happen on a dev machine, then a sync_to_prod step backs up the production vector store, diffs local vs prod by both source_url and catalog id, and inserts only the genuinely new songs before pushing the index (production never indexes, as it would OOM). The chroma push intentionally avoids rsync --delete so prod-only data like the question cache survives, stops/restarts the app in a try/finally, and health-checks /stats before declaring success.

Tested like a product

A golden suite of hand-curated cases asserts must-include artists/terms and must-NOT-include guards (hallucination tripwires) across every query type. A snapshot test compares result signatures over time with a tolerance band for model non-determinism, and Locust load tests model realistic traffic mixes (mostly cache hits, some fresh) while tracking p50/p95/p99 across commits.

Stack: Python 3.11, FastAPI, PostgreSQL, ChromaDB, httpx/BeautifulSoup/Playwright, uv, Docker Compose, Caddy.