We started building Inkeep in mid-2024 with the premise that a support agent trained on a company's own documentation would answer developer questions more accurately than a general-purpose AI. That premise is correct, but the implementation details turned out to be harder than we expected, and the hardest parts were not the ones we predicted.
Private documentation has a different failure profile than public documentation. Public docs are written and edited for external audiences, they go through review processes, they tend to be reasonably consistent. Private docs, by which we mean internal knowledge bases, Confluence spaces, Notion docs, closed support tickets, were often written by different people for different immediate purposes and stored without the expectation that an AI would ever reason over them. The gap between "this knowledge exists somewhere in our systems" and "this knowledge is usable by a support agent" is larger than it appears.
The Access Control Problem
The first challenge we hit was access control. Private documentation exists in private systems for a reason. Some content is internal only: pricing strategies, roadmaps, personnel matters. Some content is tier-specific: documentation for enterprise customers that should not be visible to free-tier users. Some content is time-limited: a runbook for handling an incident that is no longer relevant after the incident is resolved.
When you train a support agent on private documentation without explicit access control at the retrieval layer, you create a system where a developer with a free account could potentially surface information intended only for enterprise customers, or where a support agent could surface internal operational procedures to external users. This is not a theoretical risk. We saw it happen in early testing when we indexed a team's Confluence space without proper scope filtering.
Our approach is to require explicit source scoping before indexing. Teams specify which spaces, which channels, and which document types are in scope for each deployment context. We do not index from a source by default; every source is an intentional inclusion. That slows down the onboarding process slightly but prevents the access control errors from showing up in production.
Stale Content Is the Hardest Problem
Public docs teams typically have review cycles because the docs are customer-facing. Private documentation has no such forcing function. An internal runbook from eighteen months ago is still in the knowledge base because no one thought to remove it. A Slack thread that got pinned as an explanation of how something works is still there even though the behavior changed in a subsequent release. A Confluence page with an outdated architecture diagram is still indexed because no one updated it.
When a support agent draws from stale private content, the failure mode is worse than drawing from stale public content. Stale public docs are visible to the developer; they might notice a version number mismatch or a date on the page. Stale private content surfaces in an answer without any visible provenance. The developer has no way to cross-check it.
We built two mechanisms to address this. First, every indexed document has a freshness signal derived from its last-modified timestamp. When a document has not been updated in more than a configurable window (we default to 180 days), it gets a lower retrieval weight. It does not drop out of the index entirely, because old content can still be correct, but it is not preferred over recently updated content on the same topic.
Second, we surface content age in the citation when the source document is more than 90 days old. The answer might cite "internal runbook (last updated: 8 months ago)." That gives the developer and the support team a signal to verify the answer before acting on it. It adds friction, but it adds honest friction.
Formatting Inconsistency and Its Effect on Retrieval
Public documentation is usually written in a consistent format: headings, structured sections, clear topic scoping per page. Private documentation is not. A Confluence space might have pages written as prose essays, pages written as bullet lists, pages that are tables with no narrative text, and pages that are embedded images with no searchable text at all.
When the content structure is inconsistent, chunking strategies that work well on structured docs produce poor results on unstructured ones. A chunk that starts mid-sentence in an unstructured page lacks the contextual framing that makes retrieval accurate. The semantic embedding of that chunk is muddier than it would be if the chunk started at a natural section boundary.
We spent meaningful engineering time on adaptive chunking for unstructured content: detecting whether a document has clear section structure, falling back to semantic boundary detection when it does not, and tagging low-confidence chunks so they are retrieved with more context padding. The result is not as clean as indexing well-structured docs, but it is substantially better than applying a fixed-window chunking strategy to content that was never written to be chunked.
What Surprised Us
Two things surprised us about building on private documentation.
The first was how much useful knowledge lives in closed support tickets. Teams that had been running Zendesk or Linear for a year or more had a body of resolved tickets where the answer had been carefully written out by a support engineer. That content is often better than the official documentation for the specific question patterns that real developers ask, because it was written in response to those exact questions. Closed tickets became one of our highest-value private knowledge sources, which we did not anticipate when we started building.
The second was that teams often did not know what was in their own knowledge bases until the agent surfaced it. Indexing a large Confluence space and running queries against it revealed documentation that the team had forgotten existed, including some that answered questions they had been handling manually for months. The process of setting up the agent also became a documentation audit process, which was an unexpected benefit but a genuinely useful one.
What We Still Have Not Solved
We are honest about the limits. Handwritten or image-based content is not indexed usefully. Documents with complex formatting, heavy use of embedded spreadsheets or diagrams with data in image form, require manual remediation before they are useful as knowledge sources. We have not built a solution for that yet.
The deeper challenge is content governance. Inkeep can surface stale content more explicitly and weight it lower, but it cannot fix a knowledge base where teams are not maintaining the content. Stale content is not an AI problem. It is an organizational process problem that AI tooling can reveal and surface but cannot solve. If a team has not built a habit of updating their internal documentation when the product changes, no indexing architecture makes up for that. The tool is as good as the content it draws from, and private documentation quality varies more than most teams initially realize.