RAG Is a Multi-Prompt System
A RAG (Retrieval-Augmented Generation) system is not one prompt — it is three or four distinct prompts working together:
- Query transformation prompt: Converts the user's question into an optimal search query
- Retrieval grading prompt: Evaluates whether retrieved documents are actually relevant
- Generation prompt: Synthesises retrieved context into an answer
- Citation and verification prompt: Adds sources and checks for hallucination
Each needs to be engineered separately. Getting one wrong breaks the whole pipeline.
---
Users rarely ask questions in the format that retrieves the best results. Their language is conversational. The database expects precise, information-rich queries.
javascript
async function transformQuery(userQuestion) {
const response = await client.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{
role: 'user',
content: 'Transform this conversational question into an optimal search query for a vector database.\n\nRules:\n- Remove filler words (what, how, can you, please, etc.)\n- Keep only the information-bearing terms\n- Add synonyms or related terms that might help retrieve relevant documents\n- If the question contains multiple sub-questions, produce multiple queries\n\nOutput JSON: { "queries": string[] }\n\nQuestion: ' + userQuestion
}],
response_format: { type: 'json_object' },
temperature: 0.0,
});
const { queries } = JSON.parse(response.choices[0].message.content);
return queries;
}
// Example transformation:
// Input: "Hey, can you tell me what the refund policy is if I cancel my subscription?"
// Output: ["subscription cancellation refund policy", "cancel subscription money back", "subscription refund terms conditions"]
Multi-Query Retrieval
javascript
async function multiQueryRetrieval(userQuestion, limit = 5) {
const queries = await transformQuery(userQuestion);
const allResults = await Promise.all(
queries.map(query => searchVectorDb(query, limit))
);
// Deduplicate by document ID, keeping highest similarity score
const seen = new Map();
for (const results of allResults) {
for (const result of results) {
if (!seen.has(result.id) || seen.get(result.id).similarity < result.similarity) {
seen.set(result.id, result);
}
}
}
return Array.from(seen.values())
.sort((a, b) => b.similarity - a.similarity)
.slice(0, limit);
}
---
Prompt 2: Retrieval Grading
Not all retrieved documents are relevant. Add a relevance grading step that filters out poor matches before passing context to the generation model:
javascript
async function gradeRelevance(question, documents) {
const gradingPromises = documents.map(doc =>
client.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{
role: 'user',
content: 'Is this document relevant to answering the question?\n\nQuestion: ' + question + '\n\nDocument: ' + doc.content + '\n\nOutput JSON: { "relevant": boolean, "reason": string }'
}],
response_format: { type: 'json_object' },
temperature: 0.0,
})
);
const grades = await Promise.all(gradingPromises);
return documents.filter((doc, i) => {
const grade = JSON.parse(grades[i].choices[0].message.content);
return grade.relevant;
});
}
This filter step reduces hallucination significantly — the model cannot be led astray by irrelevant documents it is forced to cite.
---
Prompt 3: Grounded Generation
The generation prompt must enforce strict grounding:
javascript
function buildGenerationPrompt(question, relevantDocs) {
const context = relevantDocs.map((doc, i) =>
'[Document ' + (i+1) + '] (Source: ' + (doc.metadata.source || 'Unknown') + ')\n' + doc.content
).join('\n\n---\n\n');
return [
'You are a precise question-answering assistant. Answer the user\'s question based ONLY on the provided context documents.',
'',
'STRICT RULES:',
'1. Only use information explicitly stated in the provided documents',
'2. If the documents do not contain enough information, say: "The available documents do not contain sufficient information to answer this question fully."',
'3. Do not use any knowledge from your training data — only the provided documents',
'4. For every factual claim, cite the document number in brackets like [1] or [2]',
'5. If documents contain conflicting information, acknowledge the conflict explicitly',
'6. Do not speculate beyond what the documents say',
'',
'Context Documents:',
context,
'',
'Question: ' + question,
'',
'Answer (with citations):'
].join('\n');
}
---
Prompt 4: Hallucination Detection
After generation, add a verification step that checks the output against the source documents:
javascript
async function verifyAnswer(question, answer, sourceDocs) {
const verification = await client.chat.completions.create({
model: 'gpt-4o',
messages: [{
role: 'user',
content: 'Verify this answer against the source documents. Identify any claims in the answer that are not supported by the provided documents.\n\nQuestion: ' + question + '\n\nAnswer to verify:\n' + answer + '\n\nSource documents:\n' + sourceDocs.map((d, i) => '[' + (i+1) + '] ' + d.content).join('\n\n') + '\n\nOutput JSON: { "verified": boolean, "unsupported_claims": string[], "confidence": "high"|"medium"|"low", "recommendation": "use_as_is"|"add_caveats"|"regenerate" }'
}],
response_format: { type: 'json_object' },
temperature: 0.0,
});
return JSON.parse(verification.choices[0].message.content);
}
---
Handling Knowledge Gaps Gracefully
When the system cannot answer a question from the available documents, the response matters. A RAG system that gracefully admits knowledge gaps is far more trustworthy than one that hallucinates answers.
javascript
function handleKnowledgeGap(question, context) {
if (context.length === 0 || allDocumentsLowRelevance(context)) {
return {
answer: 'I don\'t have information about this topic in my current knowledge base. ' + getSuggestion(question),
sources: [],
knowledge_gap: true,
};
}
return generateWithCaveats(question, context);
}