Motivation

How can AI improve hospitals? Although a seemingly simple question, it is quite a complex topic to unpack and is something our team has dedicated the past few years to exploring. Specifically, our team focused on what we believe is one of the most promising routes for improving patient care: helping clinicians better understand how patient activity and behavior change over the course of a hospital stay. A patient’s behavior, mobility, and interactions offer clues about their condition and clinical trajectory. However, maintaining this complete picture currently requires piecing together observations that are often incomplete, temporally dispersed, and difficult for any individual clinician to track. To address this challenge, we introduce TRACER (Trajectory Reasoning Across Clinical Event Records)1, a training-free multimodal system that enables LLMs and VLMs to understand and reason about bedside video and electronic health records (EHRs) to track patient behavior across hospital stays. In this blog post, we will share the years-long journey of building TRACER, how it enables reasoning about hospital stays, and how it revealed behavioral patterns associated with delirium.

Collecting the data

Many years ago, our group at the Partnership in AI-Assisted Care saw an opportunity for assistive AI technology to improve care and reduce costs associated with hospital care. Our group focuses on ambient intelligence (AI for physical spaces), and our early work consisted of detection systems for hand hygiene and ICU mobility23. In 2021, we laid the foundation for TRACER by installing depth, infrared, and video sensors inside Stanford Hospital ICU units. The approval process for installation was long and difficult, spanning over 5 years. It required navigating the complexities of respecting patient data privacy while simultaneously trying to build technology that would ultimately be beneficial for both patients and healthcare workers. This ambitious project paid off in the form of a dataset spanning years of continuous recordings in real ICU environments.

Finding a clinical problem to solve

In order to find a clinical problem to solve, we spent many hours shadowing clinicians and interviewing nurses and doctors to identify several important clinical challenges.

Delirium was the first important clinical challenge we identified. Delirium is a sudden state of confusion that is associated with long-term cognitive impairment even months after the hospital stay4, and about 1 in 3 patients in ICUs develop delirium5. Clinicians primarily manage delirium by reducing relevant medications (e.g., sedatives, opioids, etc.) and by engaging in relevant clinical bundles – groups of actions that, when taken together, give better patient outcomes6. However, delirium status fluctuates, and manual bedside assessments can only give intermittent snapshots.

The second important clinical challenge that we sought to address is the information loss between changes of shift. Naturally, human workers in the hospital can only spend so long per shift before someone takes their place. For nurses in the ICUs, this is 12 hours. From our conversations with nurses, we found that information loss across many consecutive shifts was both a prevalent issue and also difficult to address with existing systems.

When taken together with our findings on delirium, it became apparent to us that clinicians are lacking an intuitive interface for understanding how a given patient’s activity and behavior evolves over the course of the hospital stay and how patient activity changes in response to specific medications and interventions.

Building our solution

An automated 24/7 observer. The overarching goal of the project became apparent – to develop a 24/7 observer of patient behavior that can be used to better inform clinicians and nurses how changes in treatment affect patient activity and behavior across long periods of time. This goal led to some natural constraints for how we designed our system. One of these was tracking a consistent set of actions in order to detect trends and changes over time. Specifically, we spent several weeks co-designing a set of “atomic” actions with nurses and clinicians. We tried to optimize for both (1) what is clinically meaningful and (2) what VLMs can detect zero-shot from our sensor placement. Below are the diagrams outlining our final taxonomy of 15 atomic actions (left) and our atomic action detection performance (right).

Illustration of a patient lying in a hospital bed with a clinician at the bedside, with colored outlines around the head, torso, arms and legs. Labels list the 15 atomic actions in five groups. Interaction: clinician physical contact, talking, object exchange, clinician restraining. Head: looking at others, head turning. Torso: changing bed position, torso movements. Arms: grabbing medical equipment, holding or using objects, gesticulating, adjusting pillows or bed sheets, other motions such as fidgeting. Legs: leg movements in bed, leg movements out of bed. Radar chart of minute-level F1 scores for each atomic action, comparing Gemini 3.5 Flash, Gemini 2.5 Pro, Gemini 3.1 Flash Lite and Qwen 3.5 (9B). The three Gemini models score similarly on most actions, and Qwen 3.5 scores lower on most of them.

Constructing a temporal hierarchy No predefined action set can account for all important actions that could occur during a hospital stay. So for every minute of video, in addition to labeling the 15 atomic actions, we also prompt the VLM to write a short caption describing what is happening in the room. We then combine the atomic action labels and scene captions to form minute-level records of the hospital stay. These are useful as-is, but the median duration of ICU stays in our patient cohort is more than 3 days (thousands of minutes). This motivated the construction of hourly summary reports which both capture events that occur over longer periods of time and enable more efficient parsing of the hospital stay. The hourly reports include changes in patient medication alongside aggregated information from the minute-level reports. The hourly reports and minute-level records are built continuously as the hospital stay unfolds.

“No more alarms, please!” Something that we heard echoed repeatedly in conversations with hospital workers is that they already suffer from alarm fatigue. Nurses in ICUs are often bombarded with alerts, most of which are false alarms7. How can we build an interface that allows nurses/clinicians to quickly understand patient state without adding to the problem? We landed on a video timeline viewer + chat interface setup (see diagram below). The timeline allows clinicians and nurses to quickly see changes in activity over time and connect those changes to medication administration and times of the day. The chat interface supports retrospective querying of the data sources (e.g., “Which hours of the stay was the patient most active?”)

The TRACER viewer on synthetic data. Top: a timeline from 08:00 to 18:00 of minute-level atomic actions grouped into head, torso, arms, legs and interaction, above medication rows for hydromorphone, dexmedetomidine and albuterol, with a cursor at 14:14. Bottom left: a synthetic ICU video frame at 14:14:20 showing a clinician beside a patient in bed, with the atomic actions detected that minute and a one-sentence caption below it. Bottom right: a chat panel where the user asks when the albuterol nebulizer was given and what the patient was doing; the assistant lists four lookups and answers with a clickable 14:14 timestamp.

Evaluating our system. We knew it was important to choose objective evaluation criteria to guide future development. Therefore, we manually curated two evaluation benchmarks for understanding our system’s capabilities. The first benchmark evaluates perception via atomic action classification accuracy on 397 video clips. The second benchmark tests long-form reasoning via 144 MCQs that span up to the entire hospital stay. We rigorously constructed these evaluation benchmarks, spending approximately 750 hours of human effort.

Testing models on our benchmarks led to interesting discoveries. For example, on our reasoning benchmark, open-source LLM agents performed close to, and sometimes exceeded, proprietary models. However, on our perception benchmark, open-source VLMs detected atomic actions much less accurately than proprietary models (we tested only the Gemini series because of our data usage agreement).

Table comparing perception (F1 for detecting actions in one-minute clips) and reasoning (percent correct on questions about whole ICU stays). Gemini 3.5 Flash: 68.6 and 70.7. Gemini 3.1 Flash Lite: 62.2 and 61.0. Best open-weight model: 48.4 for Qwen3.5 9B and 67.9 for Gemma-4 31B. Gap to best proprietary: minus 20.2 for perception and minus 2.8 for reasoning.

Conclusion and future work

We built a system that can present trends to clinicians in an intuitive visual manner, answer questions about hospital stays, and provide video evidence to support its answers. Early feedback from nurses and clinicians who were shown the interface was encouraging, with respondents noting its potential to support their day-to-day work. We believe that systems like ours can augment the workflows of doctors and nurses to improve clinical decision-making and understanding of patient trajectories.

Our future work investigates a meaningful pattern we discovered while building TRACER. Upon deploying our system across a cohort of 280 patients, we noticed the frequency of “purposeful” atomic actions output by TRACER was significantly lower in patients with delirium. This finding highlights the potential for ambient intelligence to be used to construct digital biomarkers of human activity for health.

Paper and code

Paper (arXiv) · Code (GitHub)

This post is based on our paper TRACER: A Multimodal Harness for Patient Care in Hospitals. If you find our work useful, please cite it:

@article{durante2026tracer,
    title   = {TRACER: A Multimodal Harness for Patient Care in Hospitals},
    author  = {Durante, Zane and Tang, Paul and Chandrasegaran, Keshigeyan and
               Kaushal, Amit and Lakshmikanth, Shrinidhi Kowshika and Kolawole, Feolu and
               Zheng, Yu Jun and Geng, Yifan and Agarwal, Shobhit and
               Schulman, Kevin and Niebles, Juan Carlos and Adeli, Ehsan and
               Milstein, Arnold and Fei-Fei, Li},
    journal = {arXiv preprint arXiv:2610.21266},
    year    = {2026}
}

  1. Durante Z, Tang P, Chandrasegaran K, et al. TRACER: A Multimodal Harness for Patient Care in Hospitals. arXiv preprint arXiv:2610.21266, 2026. https://arxiv.org/abs/2610.21266 ↩

  2. Haque A, et al. Towards Vision-Based Smart Hospitals: A System for Tracking and Monitoring Hand Hygiene Compliance. Proceedings of Machine Learning for Healthcare (MLHC), PMLR 68, 2017. https://proceedings.mlr.press/v68/haque17a.html ↩

  3. Yeung S, Rinaldo F, Jopling J, et al. A computer vision system for deep learning-based detection of patient mobilization activities in the ICU. npj Digital Medicine. 2019;2:11. https://pmc.ncbi.nlm.nih.gov/articles/PMC6550251/ ↩

  4. Pandharipande PP, Girard TD, Jackson JC, et al. Long-term cognitive impairment after critical illness. New England Journal of Medicine. 2013;369(14):1306–1316. https://doi.org/10.1056/NEJMoa1301372 ↩

  5. Salluh JIF, Wang H, Schneider EB, et al. Outcome of delirium in critically ill patients: systematic review and meta-analysis. BMJ. 2015;350:h2538. https://doi.org/10.1136/bmj.h2538 ↩

  6. Pun BT, Balas MC, Barnes-Daly MA, et al. Caring for critically ill patients with the ABCDEF bundle. Critical Care Medicine. 2019;47(1):3–14. https://doi.org/10.1097/CCM.0000000000003482 ↩

  7. Drew BJ, Harris P, Zègre-Hemsey JK, et al. Insights into the problem of alarm fatigue with physiologic monitor devices: a comprehensive observational study of consecutive intensive care unit patients. PLoS One. 2014;9(10):e110274. https://doi.org/10.1371/journal.pone.0110274 ↩