Cerebras Enterprise Knowledge Base Reference Architecture

Cerebras provides a concrete production reference for a source-native knowledge system: small connectors emit one evidence shape, ingestion refreshes complete semantic units, retrieval fuses independent signals, and reusable project scopes constrain what a query may search.

Why This Is A First-Class Reference

Cerebras reports that its internal knowledge base reached more than 15,000 questions per day within three months and is used by employees, automations, and agents. Its most important decision is organizational as much as technical: do not make a new knowledge-base UI the place where everyone must migrate their work. Extract directly from the native systems where the work already lives. Source: Cerebras, 2026-07-16

This makes the article a strong implementation reference for Brain Source Fabric. It supplies a practical connector and retrieval design that the local plan previously described mostly as contracts and lifecycle states.

Reference System

The design has three layers:

1. collection and storage;
2. scoped query and evidence assembly;
3. authentication, authorization, auditing, and analytics across both.

Cerebras normalizes heterogeneous sources into a common Postgres row interface containing embeddings, summaries, and metadata. Source-specific behavior remains in connectors; the query side can therefore fuse evidence without knowing every upstream API.

Connector Contract To Adopt

Every connector should ship a manifest, adapter, fixture corpus, contract test, and doctor. A minimal manifest is:

connector_id: slack-channel-engineering
source_type: slack
scope_ids: [project-agent-systems]
auth_ref: secret://slack/socket-mode
mode: event
cursor_strategy: stable-event-id
semantic_unit: complete-thread
refresh_strategy: refetch-and-upsert
raw_retention: restricted-immutable
normalizer_version: 1
lexical_fields: [question, transcript, systems, code_refs]
semantic_fields: [normalized_question, summary, resolution]
privacy: private
acl_strategy: source-and-row
freshness_slo: 60s
retry_policy: exponential-with-dead-letter
health_checks: [auth, cursor, queue-age, fetch, normalize, index]

Required behavior:

1. acknowledge an event quickly, then process it asynchronously;
2. deduplicate using a stable upstream event identity;
3. resolve the complete semantic unit rather than indexing an isolated delta;
4. preserve the restricted raw artifact and its source metadata;
5. make raw text lexically searchable as soon as policy allows;
6. normalize or distill the semantic unit into consistent retrieval fields;
7. apply ACL and project scope metadata before indexing;
8. upsert idempotently and record schema, cursor, freshness, and receipt state;
9. expose capture, enrichment, indexing, and review failures separately;
10. replay fixtures without duplicate canonical effects.

This pattern is especially important for conversational sources. A new Slack reply is not a standalone document; it changes the meaning and resolution state of the entire thread. The adapter should refetch and recompile that thread as one semantic unit.

Connector SDK And Onboarding

Cerebras lets teams add custom sources through small Python modules that emit the common row shape. The local equivalent should be a reviewed connector package rather than a special-case script:

connector/
├── manifest.yaml
├── adapter.ts or adapter.py
├── fixtures/
│   ├── event.json
│   ├── source-response.json
│   └── expected-envelope.json
├── contract.test.*
└── doctor.*

The pull-request gate verifies identity, cursor semantics, raw preservation, privacy defaults, normalized schema, idempotency, replay, rate limits, retry/dead-letter behavior, ACL propagation, freshness, and uninstall/rollback. A connector is not active because its module exists; it becomes active only after configuration, authorization, health, workflow binding, and authority checks pass.

Retrieval Is An Ensemble

Cerebras does not trust one score. Its Slack search combines:

• full-text matching for exact errors, flags, hosts, file paths, and identifiers;
• embeddings for paraphrases and conceptual similarity;
• inverse-document-frequency rarity so distinctive terms outrank filler;
• age decay where recent operational knowledge should win;
• conversation-burst structure and social signals where useful.

Candidate lists are combined with reciprocal-rank fusion:

rrf_score(document) = Σ weight(retriever) / (60 + rank)

Then the system consolidates duplicate chunks into sources, caps domination by one file, reranks against the actual question, keeps a bounded result set, and expands each winner to neighboring context. The final model receives a rich evidence packet, not a bag of embedding hits.

For this wiki, the equivalent path is:

privacy + project filter
→ qmd semantic candidates + exact/lexical candidates + live-source tools
→ weighted RRF
→ source dedupe and diversity caps
→ question-specific rerank
→ canonical-page / neighboring-section expansion
→ cited evidence packet

Query Planning And Project Scope

A short planner should choose only from a compact source/tool inventory and an already-authorized scope. Cerebras exposes source-specific tools such as subsystem search, Slack search, code search, recent pull requests, and expertise discovery, then runs independent retrieval paths in parallel.

Its Projects abstraction is the stronger design lesson: a project is a named, reusable bundle of Slack channels, repositories, databases, and documents. The same source may belong to several projects without data duplication. Kevin's workflow manifest should play the same role, bundling knowledge routes, source scopes, capabilities, privacy, freshness, and proof requirements. The user may select a project or workflow; the planner may narrow it, but must not silently widen authority.

MCP Boundary

Cerebras exposes narrow retrieval building blocks over MCP instead of one opaque answer endpoint. That is the right default:

• tools are structured, stable, and as LLM-free as practical;
• clients own planning, parallelism, synthesis, citation, and caveats;
• direct retrieval remains independently testable;
• a new agent client can compose the same primitives without inheriting a hidden orchestration policy.

The web application can own a richer planner → executor → synthesizer loop because it also owns the interaction, citation, and audit experience.

What To Adopt, Adapt, And Reject

Implementation Slice

The first build should prove three adapters against one connector SDK:

1. X or arbitrary URL/article: immutable post plus mutable linked artifact;
2. repository: versioned files, changed-chunk refresh, allow/deny paths, exact code search;
3. AI session or Discord/Slack-style thread: complete semantic-unit refresh, private ACLs, attachments, decisions, and corrections.

Each adapter must pass:

• stable identity and duplicate-event fixtures;
• raw replay and schema-migration fixtures;
• incremental update without duplicate rows or canonical writes;
• lexical availability before semantic distillation finishes;
• ACL and cross-project leakage tests;
• stale cursor, rate-limit, partial fetch, and dead-letter recovery;
• retrieval tests covering exact identifiers, paraphrases, freshness, source diversity, and neighboring context;
• removal/retention behavior and a complete incorporation receipt.

Architecture Position

---

Timeline

• 2026-07-16 | Promoted Cerebras's production knowledge-base write-up into a first-class architecture reference and adopted its source-native connector, semantic-unit refresh, ensemble retrieval, RRF, context expansion, narrow MCP, and reusable project-scope patterns for the local source fabric. Source: Cerebras; User request, 2026-07-16