The Cognitive Bandwidth Problem
Consider a single patient with locally advanced rectal cancer. By the time they reach their 24-month follow-up, their record has accumulated:
- A colonoscopy video and biopsy histology slides
- Staging CT and rectal MRI with dozens of measurements
- An operative video, an operative note, and an anesthesia record
- A pathology report with margin status, lymph node yield, and molecular markers
- Neoadjuvant and adjuvant therapy records
- Serial CEA values, periodic imaging, and free-text follow-up notes
None of these are optional. Each carries information that changes a clinical decision. And they are heterogeneous, asynchronous, and semantically non-uniform — exactly the kind of data that exceeds any individual clinician's ability to integrate in real time.
In my first post, I argued that the solution begins with the data layer: a research-ready cohort must satisfy semantic consistency, temporal completeness, and computability. Today I want to go one step further and address the layer that sits on top of that foundation: how do we actually reason over this data at the point of care?
My answer is not a bigger model. It is a different architecture.
Why a Single Model Falls Short
The temptation, in 2025, is to throw a large language model at the problem and call it "AI for colorectal cancer." I have tested that temptation, and it does not survive contact with clinical reality. A single model — however capable — faces three structural fractures in the CRC setting:
1. The Modality Fracture
A colorectal cancer record is not text. It is text and images and structured tables and genomic vectors.
A single model trained on one modality cannot reason across the others. An LLM reads the pathology report but cannot see the MRI. A vision model reads the MRI but cannot weigh it against the patient's CEA trajectory. Forcing one model to do all of it produces shallow reasoning at best and confident errors at worst.
2. The Temporal Fracture
Clinical decisions are not made from a snapshot. They are made from a trajectory.
Whether a recurrence is "early" or "late," whether a complication is "expected" or "alarming" — these judgments depend on when events occur relative to each other, not on their isolated values. A single model invoked per-encounter loses the longitudinal thread.
3. The Task Fracture
Assisting a surgical decision, automating follow-up, and maintaining a research cohort are three different jobs with three different failure modes.
Conflating them into one model means optimizing for none. A model tuned to sound authoritative in a consult will be dangerously wrong when used to auto-populate a research variable.
The DACCA-MAS Architecture
Rather than one model that does everything poorly, we are building a Multi-Agent System (MAS) — a set of specialized agents, each responsible for one class of task, coordinated by an orchestrator that manages context and resolves conflict.
|
Agent |
Responsibility |
Concrete example |
|
Data Governance Agent |
Cleans, aligns, and standardizes raw multi-modal data into computable objects |
Reads a radiology report, extracts the TNM stage, and writes it back to the structured DACCA field — using the pathology report as authoritative when the two disagree |
|
Clinical Decision Agent |
Generates guideline-anchored, explainable recommendations from individual data |
For a low rectal cancer patient, proposes a neoadjuvant regimen and cites the specific staging features and guideline clause behind it |
|
Follow-up Agent |
Triggers follow-up, detects risk signals, personalizes patient interaction |
Flags a rising CEA at 6 months + equivocal imaging, escalates to clinician, and prepares a patient-specific visit plan |
|
Research Archive Agent |
Extracts variables, maintains research cohorts automatically |
Encodes a newly enrolled case into the DACCA dictionary without manual abstraction |
|
Orchestrator |
Schedules agents, manages shared context, resolves conflicts |
When the Decision Agent and Follow-up Agent disagree on risk, routes the case to human review instead of auto-acting |
The key design principle: each agent is narrow, auditable, and grounded in the data layer. None of them is asked to be "intelligent" in the general sense. They are intelligent within a bounded role, which is precisely what makes them trustworthy in a clinical setting.
A Concrete Scenario: Follow-up With a Memory
Let me make this tangible with the follow-up use case, because it is where the difference is most visible to patients.
Traditional postoperative follow-up is a temporary, episodic interaction. A patient returns at month 3. The surgeon opens the chart, reconstructs the history from memory and notes, and decides what to check. Under time pressure, the interaction is generic: "blood test, imaging, see you in 3 months." The patient's individual risk profile — a close circumferential margin, a T4 lesion, a rising CEA trend — may not surface in the conversation.
Now place a MAS behind that same visit. The Follow-up Agent, drawing on the DACCA data layer, already knows this patient's longitudinal risk map:
- Low anterior resection with a 1 mm circumferential margin
- Preoperative CRM < 2 mm (high-risk feature)
- CEA trend: 2.1 → 2.8 → 3.5 ng/mL across the last two visits
When the patient asks, "Can I resume heavy exercise? Can I travel?", the agent does not return a generic guideline sentence. It responds with evidence anchored to this specific patient's trajectory: the recovery milestones reached, the warning signs to watch for given the close margin, and the follow-up tests still pending. The interaction is temporary in form but informed in content — because the agent carries the memory the human visit cannot.
That is the point: MAS turns a transient clinical encounter into a continuous, personalized one — without asking the surgeon to hold the entire dataset in their head.
The Engineering Challenges We Cannot Hand-Wave
I want to be honest about what is hard, because the community deserves fewer demos and more candor:
- Modality alignment: When the MRI says "likely EMVI-positive" and the pathology says "no venous invasion," which wins, and how is that encoded? We resolve this with explicit source hierarchies defined by clinicians, not by the model.
- Knowledge drift: Guidelines change. When a new edition is published, the Decision Agent's knowledge must be versioned and the change auditable — not silently absorbed.
- Hallucination containment: The single most dangerous failure is a confident citation that does not exist. Our agents are constrained to cite structured fields or specific document spans; when evidence is insufficient, the correct output is "I cannot determine this from the available data."
- Explainability by default: In surgery, a recommendation without a traceable rationale is not a recommendation — it is a liability. Every agent output must be reconstructable to its inputs.
The Ceiling Is Set Below the AI Layer
Here is the thread that connects this post back to the first one.
A MAS is only as reliable as the data layer beneath it. If the "anastomotic leak" variable is coded inconsistently — as we found in our pre-standardization audit, with up to 30% source disagreement — then the Follow-up Agent will escalate the wrong cases, and the Research Archive Agent will build cohorts on sand.
Data maturity is the ceiling of AI performance. Not model size. Not parameter count. The disciplined, computable, temporally complete cohort I described in Post 1 is what makes the agents in Post 2 trustworthy.
This is why I resist the narrative that hospitals should "buy an AI" before they have "built a database." The order matters. Get the foundation right, then architect the intelligence.
A Surgeon's Pragmatic Stance
I will close with a position I hold firmly: these agents are collaborators, not replacements.
The follow-up scenario above does not remove the surgeon. It removes the drudgery of reconstruction so the surgeon can spend the visit on judgment, empathy, and the nuances no agent can capture. Clinical responsibility remains, by definition and by ethics, human.
A multi-agent system that tries to erase the clinician is a system that has misunderstood its job. Ours is built to amplify clinical judgment — by making the data behind every decision legible, continuous, and patient-specific.
What's Next
In the next post, I will go deeper into the hardest engineering layer of all: multi-modal heterogeneous data governance. How do we bring imaging, pathology, structured records, and free text into a single governed framework — and what does "governance" actually require when the sources disagree?
Join the Conversation
I'd like to hear from the community:
- If you are building clinical AI: did you start with the model or the data layer?
- For those using agents in care: how do you handle the moment when two agents disagree?
- What guardrails have you found essential for keeping clinical AI explainable?
Comment below or reach out directly. The architecture decisions we make now will shape whether AI in colorectal cancer becomes a genuine clinical tool or an expensive demo.
The views expressed in this post are the author's own and do not represent the position of any affiliated institution.