Event-Driven RAG: ownCloud, NATS, Qdrant and MCP

In the landscape of artificial intelligence and personal productivity assistants, Retrieval-Augmented Generation (RAG) has established itself as the standard for connecting large language models to private knowledge bases. However, the vast majority of current implementations rely on outdated ingestion patterns: periodic filesystem cron polling, blind full re-indexing of massive directories, or manual trigger buttons in administrative dashboards.

This batch-driven approach suffers from critical design flaws: it wastes CPU cycles and disk I/O, introduces a latency of tens of minutes between uploading a document and having it available to AI models, and frequently creates duplicate vector chunks during ordinary file renames or edits.

To equip my self-hosted cloud with an instantaneous and secure document memory, I designed an event-driven and natively multi-user ingestion pipeline built around ownCloud Infinite Scale (oCIS), the NATS JetStream distributed message broker, the Qdrant vector database, and a Model Context Protocol (MCP) server exposed via Streamable HTTP.

Here is the technical deep dive into this architecture, from intercepting internal NATS packets to strict per-user vector collection partitioning.

Architecture Overview

Rather than running a scheduled crawler, the architecture taps into the native microservice event bus of ownCloud Infinite Scale. oCIS services coordinate asynchronously over an embedded NATS broker. Connecting a lightweight sidecar container (ocis-qdrant-indexer) on the same internal Docker network handles ingestion, event routing, and Model Context Protocol exposure.

flowchart TB
    subgraph StorageLayer["oCIS Storage Layer"]
        User["User / WebDAV Client"] -->|"Upload / Edit"| OCIS["ownCloud Infinite Scale (oCIS)<br>decomposedfs Engine"]
        OCIS -->|"NATS Event (SpaceId)"| NATS["NATS JetStream Broker<br>(Internal Endpoint :9233)"]
    end

    subgraph SidecarLayer["Sidecar Container (ocis-qdrant-indexer)"]
        Consumer["NATS Event Consumer<br>(UploadReady, ItemTrashed, ItemMoved)"]
        UserStore[("Encrypted SQLite User Store<br>(AES-256-GCM / PATs & Spaces)")]
        UserIndexer["Multi-User Indexer<br>(SpaceId Routing & Background Sync)"]
        WebDAVClient["Authenticated WebDAV Client<br>(Dedicated Per-User PAT)"]
        Chunker["Extraction Tika / PaddleOCR<br>Tokenizer Cache & SHA-256"]
    end

    subgraph VectorAndAI["Vector Layer & Inference"]
        CloudflareAI["Cloudflare Workers AI<br>Model @cf/baai/bge-m3 (1024 dims)"]
        Qdrant[("Qdrant Vector DB<br>Per-User Collections: user_hash")]
    end

    subgraph MCPLayer["AI Agent Surface"]
        MCPServer["MCP Streamable HTTP Server<br>(HTTP Basic Auth with ownCloud)"]
        Hermes["AI Agents (Hermes, Pi, Claude)<br>search_owncloud_docs / sync_owncloud_docs"]
    end

    NATS -->|"Event with SpaceId"| Consumer
    Consumer --> UserIndexer
    UserIndexer <-->|"User & PAT Resolution"| UserStore
    UserIndexer --> WebDAVClient
    WebDAVClient -->|"Download via /dav/spaces/"| OCIS
    WebDAVClient --> Chunker
    Chunker -->|"Vector Calculation 1024 dims"| CloudflareAI
    CloudflareAI -->|"Embeddings"| Chunker
    Chunker -->|"Upsert / Filtered Purge (space_id)"| Qdrant

    Hermes <-->|"MCP Session & Basic Auth"| MCPServer
    MCPServer <-->|"Graph API Validation /me"| OCIS
    MCPServer <-->|"Isolated Vector Search"| Qdrant
    MCPServer -->|"Trigger Background Sync"| UserIndexer

1. Event Interception: NATS JetStream within oCIS

Unlike legacy ownCloud 10’s PHP monolith, ownCloud Infinite Scale is a distributed Go application whose services communicate through asynchronous events on an embedded NATS JetStream broker.

By default, this broker listens exclusively on 127.0.0.1:9233 inside the oCIS container. To allow our sidecar container to consume events across the dedicated Docker network (resources), the listener host is explicitly configured on the container’s network interface:

services:
  owncloud:
    image: owncloud/ocis:latest
    environment:
      - "NATS_NATS_HOST=0.0.0.0"
      - "NATS_NATS_PORT=9233"
    networks:
      - resources

Monitored Event Lifecycle

The sidecar subscribes to NATS subjects emitted by oCIS’s storage-users component:

1. UploadReady: Emitted once a file upload has fully completed and settled into storage. The payload carries the ResourceID (SpaceId + OpaqueId) and the relative FileRef.Path.
2. ItemTrashed & ItemPurged: Dispatched when an item is moved to the trash bin or permanently deleted.
3. ItemMoved: Fired whenever a file or directory is moved or renamed across the hierarchy.

Because ingestion is event-driven, indexing begins within milliseconds of the decomposed filesystem (decomposedfs) confirming the write.

2. Multi-User Architecture & Zero-Trust Authentication

A common pitfall when integrating with oCIS is its Zero Trust security model. In oCIS, each user operates within a strictly isolated Personal Space.

The Rejection of Global Admins and Service Accounts

In traditional architectures, developers commonly reuse an administrative service account to read user files. In oCIS, this approach is both ineffective and insecure:

• Absence of cross-space rights: The system admin user has zero default read permissions inside personal spaces. Any WebDAV request dispatched by the admin user against /remote.php/dav/files/<user>/... returns an HTTP 401 Unauthorized or 403 Forbidden error.
• Principle of Least Privilege: Global administrative impersonation bypasses tenant isolation boundaries and compromises all user data if the ingestion service is breached.

Native HTTP Basic Authentication with PATs

To maintain uncompromising isolation, the MCP server adopts an HTTP Basic authentication scheme where each user supplies their own ownCloud username and personal access token (Personal Access Token / PAT):

1. Immediate oCIS Graph API Validation: Upon first MCP connection, credentials are validated directly against the Graph API (/me and /me/drives). Tenant identity is derived strictly from the verified ownCloud user ID, never from arbitrary client request parameters.
2. Encrypted Persistent Storage (users.sqlite): User PATs and authorized space lists (space_id) are stored locally in an SQLite database encrypted with authenticated AES-256-GCM. The 32-byte encryption key (USER_STORE_ENCRYPTION_KEY) is injected via container environment variables.
3. Zero Secret Leakage: Cleartext tokens are never logged in application traces or exposed to AI agents.

3. Qdrant Collection Partitioning & Space-Level Filtering

The conventional alternative of grouping all documents into a single vector collection filtered by tenant metadata poses severe operational risks: a single omitted query filter or point ID collision across identical paths could leak confidential documents.

The architecture enforces a two-tier isolation model:

flowchart TD
    UserQuery["MCP Request (search_owncloud_docs)"] --> AuthCheck["Validate Basic Auth<br>& Derive Tenant ID"]
    AuthCheck --> CollectionSelect["Select Dedicated Qdrant Collection<br>user_{hash(user_id)}"]
    CollectionSelect --> SpaceFilter["Enforce Mandatory SpaceId Filter<br>(Authorized Spaces in Store)"]
    SpaceFilter --> SearchExec["Qdrant Vector Search<br>(Cosine, Top K=5, Threshold=0.65)"]

1. One Vector Collection per User: Every user receives a distinct Qdrant collection named after an opaque hash of their ownCloud ID (user_<hash>). Isolation is strictly enforced at the vector engine boundary.
2. Granular space_id Filtering: Within their dedicated collection, search and index operations are strictly scoped to spaces the user is legitimately authorized to access (personal space, shared drives, and project spaces).

Deduplication & Idempotent Ingestion

To prevent orphan chunk accumulation when editing large documents, the pipeline applies an idempotent lifecycle:

1. Cryptographic Hash Verification (SHA-256): Before invoking text extraction or embedding models, the file’s SHA-256 hash is computed. If it matches the hash stored in Qdrant metadata, processing halts immediately (zero compute, zero wasted tokens).
2. Atomic Pre-Deletion: If the file has changed, all existing vector points matching the target file_path or node_id are purged from Qdrant before new chunks are inserted.
3. Event-Driven Pruning: When ItemTrashed or ItemPurged events fire, a payload filter query (filter: { must: [{ key: "path", match: { value: event.path } }] }) instantly deletes the vectors from the affected user’s collection.

4. Multi-User Event Routing with NATS

In a multi-user deployment, receiving a NATS event prompts a fundamental architectural question: which user collection should this document be indexed into?

The sidecar implements intelligent routing driven by the NATS envelope’s SpaceId:

sequenceDiagram
    autonumber
    participant OCIS as ownCloud (NATS)
    participant Sidecar as ocis-qdrant-indexer
    participant Store as SQLite (AES-256-GCM)
    participant Qdrant as Qdrant DB

    OCIS->>Sidecar: NATS Event (UploadReady, SpaceId: "sp-123", Path: "doc.pdf")
    Sidecar->>Store: Lookup authorized users for "sp-123"
    Store-->>Sidecar: Authorized user list + decrypted PATs
    loop For each authorized user
        Sidecar->>OCIS: WebDAV download using user's PAT
        Sidecar->>Sidecar: Text extraction & BGE-M3 embeddings
        Sidecar->>Qdrant: Upsert into dedicated user_{hash} collection
    end

• Targeted Routing: The sidecar queries its encrypted local store to identify registered users whose authorized spaces include the event’s SpaceId.
• User-Authenticated Retrieval: Content is downloaded via /remote.php/dav/spaces/<spaceId>/<path> using the respective user’s PAT.
• Fallback Reconciliation: If an event lacks a usable SpaceId, the system executes safe per-user reconciliation without ever inferring document ownership from filename paths alone.

5. Efficient Vectorization via Cloudflare Workers AI & Multimodal Ingestion

Running heavyweight embedding models (like Sentence-Transformers or Ollama) directly on the host VPS CPU generates aggressive load spikes. Instead, vectorization is delegated to Cloudflare Workers AI.

The pipeline utilizes the multilingual @cf/baai/bge-m3 model:

• Dimensionality: 1,024 dimensions with cosine similarity.
• Context Window: Up to 8,192 tokens.
• Local Tokenizer Cache: A persistent cache mounted at /tokenizer-cache allows the Markdown chunker to calculate exact model token counts during hierarchical chunking, preventing unexpected truncations.
• Multilingual Support: High accuracy across French and English technical corpora.

Cost Analysis (Cloudflare Free Tier)

The economics of this approach are compelling:

• Cloudflare includes 10,000 free Neurons per day on its Workers AI platform.
• BGE-M3 consumes roughly 1,075 Neurons per million input tokens.
• This equates to a daily allowance of approximately 9.3 million tokens—indexing the equivalent of over 18,000 pages of text daily at zero cost, with zero CPU or memory footprint on the host VPS.

Multimodal Extraction & OCR Fallback

To handle diverse business and personal document formats:

• Structured textual documents (Markdown, plain text, native text-layer PDFs, Office documents) are parsed via Apache Tika.
• For scanned documents and images lacking a text layer, the sidecar falls back to a dedicated PaddleOCR CPU container (ghcr.io/frflo/ocis-paddleocr:latest) on the internal resources network, ensuring high-quality optical character recognition without third-party cloud data leaks.

6. Serving AI Agents: Streamable HTTP MCP Server & Background Sync

Once document chunks are indexed in per-user vector collections, autonomous agents (such as Hermes Agent, Pi, or Claude) interact with the knowledge base via the Model Context Protocol (MCP) exposed over Streamable HTTP on POST /mcp.

Decoupled Non-Blocking Synchronization (sync_owncloud_docs)

In early versions, triggering a manual re-sync blocked the MCP tool call for the entire duration of WebDAV tree walking and downloading. In strict agent execution environments (such as Executor’s QuickJS runtime), this caused client-side timeouts (Internal tool error [5225f429]).

The MCP server now cleanly decouples coordination from execution:

1. Non-Blocking Asynchronous Execution: The sync_owncloud_docs tool dispatches indexing to a background worker and returns immediately with { status: "started" }. If an operation is already underway, it safely returns { status: "already_running" }.
2. Automatic Initial Synchronization: Upon a user’s first successful MCP authentication, an initial background sync is scheduled automatically to index their existing library without blocking the session.
3. Contextual Semantic Search: The search_owncloud_docs tool performs cosine similarity search on the caller’s isolated collection with space filtering, returning cited excerpts, WebDAV paths, similarity scores, and page metadata.

Streamable HTTP Transport Session Management

To maintain robust multi-turn conversations with @modelcontextprotocol/sdk, the server tracks sessions explicitly:

const sessions = new Map<
  string,
  { server: McpServer; transport: WebStandardStreamableHTTPServerTransport }
>();

export function handleMcpRequest(req: Request): Promise<Response> {
  const sessionId = req.headers.get("mcp-session-id") ?? crypto.randomUUID();
  let session = sessions.get(sessionId);

  if (!session) {
    const transport = new WebStandardStreamableHTTPServerTransport({
      sessionIdGenerator: () => sessionId,
    });
    const server = createMcpServerInstance();
    server.connect(transport);
    session = { server, transport };
    sessions.set(sessionId, session);

    transport.onclose = () => {
      sessions.delete(sessionId);
    };
  }

  return session.transport.handlePostMessage(req);
}

Conclusion

By coupling ownCloud Infinite Scale’s native NATS event stream with a rigorous multi-user security model (isolated Qdrant collections, AES-256-GCM encrypted PAT storage, and native HTTP Basic authentication), this pipeline solves the challenge of real-time personal and collaborative RAG.

The infrastructure provides strict tenant privacy, immediate reactivity to file updates, and minimal resource overhead thanks to Cloudflare Workers AI embeddings and non-blocking background MCP synchronization.