Here's what happens when you deploy an AI assistant on top of your internal knowledge base:
A new engineer asks: "How do I set up the staging environment?"
The AI searches your documentation, finds three relevant documents, synthesises an answer, and presents it with confidence. The engineer follows the instructions. The first two steps work. Step three references a CLI tool that was deprecated six months ago. Step four describes an infrastructure setup that was replaced during a migration nobody documented.
The engineer is stuck. They message the team channel. Someone says: "Oh, that doc is really old." The AI didn't know that. It can't know that. It just fetched everything it found and presented it as truth.
This is the default behaviour of every RAG system, every AI search tool, and every LLM-powered assistant you've ever used on internal docs. They fetch everything. They don't discriminate. They can't tell fresh from stale.
And it's destroying trust in AI tools faster than those tools can build it.
Why AI assistants are blind to quality
Large language models and retrieval-augmented generation (RAG) systems work by finding text that's semantically relevant to a query, then using that text to generate an answer. The relevance matching is usually excellent — vector search and embeddings are genuinely good at finding content that relates to a question.
But relevance is not the same as reliability.
A document written in 2023 about your Kubernetes deployment process is highly relevant to the question "how do I deploy to production?" It's also completely wrong if you migrated to a different platform in 2024. The AI sees relevant text. It does not see a document that's 18 months out of date with broken links and zero readership.
Most AI systems have exactly one ranking signal: semantic similarity to the query. That's it. They don't check:
- When was this document last reviewed?
- Are the links inside it still valid?
- Is anyone actually reading this document?
- Has the content been flagged by readers as outdated?
- Is this a draft, an archived page, or a current document?
- If this exists in multiple languages, are the translations current?
Without these signals, the AI is doing keyword matching with extra steps. Impressive keyword matching, yes — but fundamentally incapable of telling you whether the answer it's giving is based on content you can trust.
The confidence problem
This wouldn't be as dangerous if AI tools presented uncertain answers with appropriate caveats. But they don't. That's not how LLMs work. They generate fluent, confident text regardless of whether the source material is current or ancient.
A human reading a wiki article might notice it looks dated. The page layout is old. The screenshots show a UI that no longer exists. There's a comment at the bottom saying "this is outdated." A human can apply judgement.
An AI can't. It reads the text, processes it as equivalent to any other text, and generates an answer that sounds authoritative. The user — especially a new hire who doesn't know what the current process looks like — has no reason to doubt it.
The more confident the AI sounds, the more damage stale source material does.
What the AI actually needs
For an AI assistant to give trustworthy answers from your knowledge base, it needs more than text and embeddings. It needs metadata that tells it which documents are worth using as sources. Specifically:
1. Freshness score
A numeric signal that represents how healthy a document is right now. Not when it was last edited — that's just one input. A true freshness score combines review status, link health, readership, translation alignment, and contextual drift into a single number.
When a document scores above a threshold (say, 70 out of 100), it's eligible to be used as a source for AI answers. Below that threshold, it's excluded. No exceptions.
This single mechanism eliminates the most dangerous class of AI errors: confidently wrong answers based on stale sources.
2. Expiry status
Is this document currently within its review window, or has it expired without re-approval? An expired document should be heavily deprioritised or excluded entirely, regardless of how relevant its content might be to the query.
In Rasepi, expired documents are flagged and their freshness scores drop automatically. An AI system querying the knowledge base can see this status and act on it.
3. Classification labels
Not every document serves the same purpose. A draft shouldn't be used as a source. An archived document shouldn't appear in AI answers. An internal-only document shouldn't surface in queries from external-facing tools.
Classification labels give the AI context about what kind of document it's looking at:
- Published — current, approved, safe to use
- Draft — work in progress, should not be cited
- Under review — expiry triggered, awaiting re-approval
- Archived — no longer active, kept for reference only
- Internal / External — controls visibility scope
When an AI assistant processes a query, it can filter by classification before even looking at content relevance. A draft document that perfectly matches the query should never be served as an answer.
4. Language-level signals
If your knowledge base is multilingual, the AI needs to know whether the version it's pulling from is current. A French translation that's three months behind the English source is technically relevant in French, but the information might be outdated.
Rasepi tracks freshness at the language level. Each translation has its own score based on whether its source blocks have changed since the translation was last updated. An AI querying the French knowledge base can see that the French version of a document is stale and either:
- Fall back to the English source (which is current)
- Include a caveat that the French version may be outdated
- Exclude the document entirely
5. Reader signals
If multiple readers have flagged a document as outdated, that signal should reduce the document's weight in AI responses. Crowdsourced quality signals are noisy, but they're valuable — especially when combined with other freshness metrics.
How this works in practice
Let's walk through what happens when an AI assistant queries a Rasepi knowledge base:
Query: "What's our process for handling a P1 incident at 2am?"
Step 1 — Retrieval with filtering. The system searches for semantically relevant documents. Before ranking, it filters out:
- Documents with freshness score below the threshold
- Expired documents that haven't been re-approved
- Drafts and archived content
- Documents whose language version is stale (if the query is in a non-primary language)
Step 2 — Freshness-weighted ranking. Among the remaining documents, those with higher freshness scores rank higher. A document scoring 94 outranks one scoring 72, even if the 72-scored document has slightly higher semantic similarity.
Step 3 — Answer generation. The AI generates an answer from the filtered, freshness-ranked sources. Every source is cited with its freshness score visible.
Step 4 — Staleness warnings. If the best available source has a borderline freshness score, the AI includes a caveat: "Note: The primary source for this answer was last reviewed 60 days ago. You may want to verify with the team."
Compare this to the default behaviour: find relevant text, generate confident answer, hope for the best.
What happens when you don't do this
The consequences of AI systems operating on unfiltered knowledge bases are predictable and expensive:
New hire confusion. The most common AI use case for internal docs is onboarding. New hires, by definition, don't know what's current and what's stale. They trust the AI. The AI trusts everything. Stale docs get served with confidence.
Compliance exposure. If your AI assistant provides guidance on regulatory processes using outdated documents, the advice might not just be wrong — it might be non-compliant. "The AI told me to" doesn't hold up in an audit.
Erosion of trust. Every time the AI gives a wrong answer, users trust it a little less. After three or four bad experiences, they stop using it. The investment in AI tooling delivers no value because the underlying content wasn't trustworthy.
Shadow knowledge. When people lose trust in the official knowledge base (and the AI built on top of it), they create their own: Slack messages, personal notes, tribal knowledge shared in meetings. The fragmentation that the wiki was supposed to prevent happens anyway, just differently.
The fix is at the source, not at the model
There's a temptation to solve this at the AI layer — better prompts, more sophisticated RAG pipelines, fine-tuned models that can somehow detect staleness from text alone. This is the wrong approach.
The fix is at the source. If your documents carry rich, accurate metadata about their current state — freshness score, expiry status, classification, language alignment, reader signals — then any AI system can use that metadata to make better decisions. You don't need a smarter model. You need smarter documents.
This is what Rasepi provides:
- Every document has a live freshness score that updates continuously based on link health, readership, review status, and more
- Every document has an expiry date that triggers review when it arrives
- Every document has a classification (published, draft, under review, archived)
- Every language version has its own freshness signal so stale translations are detected independently
- Reader flags and cross-reference tracking add additional quality signals
When an AI system queries Rasepi's knowledge base, all of this metadata is available. The AI doesn't have to guess whether a document is trustworthy. The document tells it.
A practical starting point
If you have an AI assistant running on your knowledge base today, you can start assessing the problem in 30 minutes:
-
Ask your AI assistant 10 questions you know the answers to. Note which answers use stale sources. You'll probably find at least 2-3 out of 10 are based on outdated content.
-
Check the source documents. For each answer the AI gave, look at the source document. When was it last reviewed? Are the links valid? Would you trust it if you read it yourself?
-
Look for the worst case. Find your oldest, most neglected document that still appears in search results. Ask the AI a question that would surface it. Does the AI use it? It almost certainly does.
-
Estimate the impact. How many queries per day does your AI assistant handle? If 20-30% of answers are based on stale content, what's the cost in terms of wasted time, wrong decisions, and lost trust?
AI assistants are only as good as the content they're built on. Right now, most of them treat every document in your knowledge base as equally valid. They fetch everything — the doc that was reviewed yesterday and the one nobody has touched in two years — and present it all with the same confidence.
That's not a model problem. It's a data quality problem. And the solution is straightforward: give your documents metadata that tells AI tools what to trust.
Your AI assistant shouldn't sound confident about an answer sourced from a document nobody has reviewed in 18 months. With the right signals, it won't.
Rasepi makes every document carry its own trust score — freshness, expiry status, classification, language alignment. AI tools query the knowledge base and get not just content, but context. Trusted sources surface. Stale ones don't. That's how AI-powered documentation should work.