Session-Based Context Architecture

The Problem: Stateless Tax

Every API request the enterprise assistant receives today pays a full context reconstruction cost — regardless of whether it's the first message or the tenth in a conversation. We measured this directly against production:

Compare to OpenClaw (the reference implementation this assistant is modeled on):

Why Our Architecture Makes This Easy to Fix

Most multi-user applications would face complex session management challenges here. We don't — because of a key architectural decision already made: one Container App per user.

The container IS the session. A user's Container App is dedicated exclusively to them. There's no multi-tenancy to manage. We can load their workspace context once at container startup and hold it in memory forever.

This is exactly how OpenClaw works: context is loaded once when the process starts, held in memory, and every conversation just appends incremental turns.

Two Approaches to Compare

Implementation Plan — Branch: feature/session-based-context

1. AppContext Singleton (new: packages/agent_core/app_context.py)

class AppContext:
    """Loaded once at startup. Held in memory for the container's lifetime."""

    workspace_content: str        # AGENT.md + USER.md merged
    skills_index: list[dict]      # [{name, description, content}]
    base_system_prompt: str       # rendered from workspace_content
    loaded_at: datetime
    user_email: str

    @classmethod
    async def initialize(cls, workspace_reader, user_email) -> "AppContext":
        # Called once from FastAPI lifespan startup event
        ...

    def refresh_if_stale(self, max_age_seconds=300):
        # Check if files changed, reload if needed
        ...

2. FastAPI Lifespan Startup (change: apps/gateway/main.py)

from contextlib import asynccontextmanager

@asynccontextmanager
async def lifespan(app: FastAPI):
    # STARTUP: load workspace once
    app.state.ctx = await AppContext.initialize(workspace_reader, user_email)
    app.state.conversations = {}   # conversation_id → message list (in-memory)
    yield
    # SHUTDOWN: flush any pending Postgres writes

app = FastAPI(lifespan=lifespan)

3. In-Memory Conversation Store (change: packages/context/memory.py)

class ConversationMemory:
    def __init__(self, db: Database):
        self.db = db
        self._cache: dict[str, list] = {}   # conversation_id → messages

    def get_messages(self, conversation_id: str) -> list:
        if conversation_id in self._cache:
            return self._cache[conversation_id]          # ← fast path, no DB
        msgs = self.db.fetch_all("SELECT ...", (conversation_id,))  # cold load
        self._cache[conversation_id] = msgs
        return msgs

    def add_message(self, conversation_id: str, ...):
        self._cache.setdefault(conversation_id, []).append(msg)
        asyncio.create_task(self._flush_to_postgres(conversation_id, msg))  # async

4. Agent Loop (change: packages/agent_core/loop.py)

# Before: workspace loaded per request
system = build_system_prompt(workspace.load_agent_md(), workspace.load_user_md(), ...)

# After: use pre-loaded app context
system = app_ctx.base_system_prompt   # already rendered, no I/O

5. Min-Replicas Fix (infra change)

az containerapp update \
  --name ca-entasst-gateway \
  --resource-group rg-enterprise \
  --min-replicas 1

Costs ~$30/month (one always-warm container) but eliminates 34s cold starts entirely.

Testing & Eval Framework

To compare approaches objectively, we're building a unified eval system with three components:

Eval Metrics Per Run

Dashboard Preview

The eval dashboard will live at /evals on the gateway and show:

• Overview card: current branch vs baseline — avg latency, avg tokens, quality score, pass rate, side by side
• Latency chart: histogram + p50/p95/p99 per branch
• Token efficiency chart: input/output tokens per task category
• Task drill-down table: every task, both branches, click to see full response + judge reasoning
• Run history: trend over time — shows improvement (or regression) across deploys

Decision Criteria

After running evals against both branches, the session-based approach ships if:

Analysis based on live production testing against ca-entasst-gateway.calmhill-d9f0bfc3.eastus.azurecontainerapps.io · March 12, 2026

Branch: feature/session-based-context · Report prepared by Chief · Internal use only