Zane Durante, Paul Tang, Keshigeyan Chandrasegaran, Amit Kaushal, Shrinidhi Kowshika Lakshmikanth, Feolu Kolawole, June (Yu Jun) Zheng, Yifan Geng, Shobhit Agarwal, Kevin Schulman, Juan Carlos Niebles, Ehsan Adeli, Arnold Milstein, Li Fei-Fei
TRACER is a training-free multimodal system that lets LLMs and VLMs reason over days of ICU bedside video and electronic health records, so clinicians can see how a patient's activity changes over a hospital stay and ask questions about it.
CausalSmith is an agentic pipeline that proposes new theorems in causal inference, verifies them in Lean 4 on top of its own causal inference library, and presents each result as a paper that readers can rate, comment on, and cite.
R&B-EnCoRe is a self-improvement pretraining cycle for embodied reasoning VLAs. Instead of relying on a fixed reasoning template, the model generates its own candidate reasoning, refines it by measuring how much each piece helps predict the correct action, and bootstraps a stronger policy.
Atindra Jha, Naomi Sagan, Keisuke Kamahori, Xikai(Noah) Meng, Luke Zettlemoyer, Olivia Hsu, Jure Leskovec, Baris Kasikci, Stephanie Wang
Today's models no longer fit the mold of autoregressive token generation, but the systems supporting LLM inference have not kept up. M* is designed to maximize flexibility and performance for current and future composite models.
We introduce VAGEN, a reinforcement learning framework that trains vision-language model (VLM) agents to build internal world models through explicit visual state reasoning.
We introduce Theory of Space, a benchmark testing whether foundation models can construct, revise, and exploit spatial beliefs through active exploration. Across six state-of-the-art models, we reveal a critical exploration bottleneck, a persistent text-vision modality gap, and severe belief iner...
Sang T. Truong, Yuheng Tu, Michael Hardy, Anka Reuel, Zeyu Tang, Jirayu Burapacheep, Jonathan Jude Perera, Chibuike Uwakwe, Benjamin W. Domingue, Nick Haber, Sanmi Koyejo
We introduce a scalable framework to flag invalid benchmark questions. We analyze statistical signals and use them to guide expert review, achieving up to 84% precision across nine popular benchmarks.
Feedback Descent scales text-based optimization into a simple, domain-agnostic procedure that outperforms specialized RL methods across molecular design, prompt optimization, and visual editing.
We introduce SLPHelm, the first-ever benchmark for AI in speech-language pathology. Testing 15 models on 5 diagnostic tasks revealed that today's AI isn't ready for clinical use—but targeted fine-tuning shows promising improvements. Our findings highlight critical biases and unexpected failures t...
We propose a more efficient way to locate the "needles" in video haystacks. T* reframes long-video understanding as temporal search, enabling targeted reasoning with just a few frames and zero training.
We introduce key optimizations to the 20 year old K-SVD algorithm, show that it can match sparse autoencoder performance for interpreting LLM embeddings, and provide theoretical insights into the feasability of the dictionary learning formulation.
Data-driven methods make it easy to explore unconventional robot design to unlock novel capabilities. Meet RoboPanoptes: The All-Seeing Robot with Whole-body Dexterity, and Dynamics-Guided Diffusion Model for Sensor-less Robot Manipulator Design!
Andy K. Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, et al.
We introduce BountyBench, a cybersecurity benchmark featuring 25 systems with complex, real-world codebases, and 40 bug bounties that cover 9 of the OWASP Top 10 Web Application Security Risks.
How do LLMs memorize long sequences of texts verbatim? In this work, we show that verbatim memorization is intertwined with the LM’s general capabilities.
We developed a new expert design and annotated clinical decision-making dataset that also allows for nuanced accuracy and fairness evaluations with expert preferences, uncertainty, and soft labels.
Learning methods for robotics have a data problem. Autonomous robotic systems can deliver orders of magnitude more real-world embodied data than we have.
We discuss disparate impact, a type of discrimination in which a policy that appears neutral disproportionately impacts a protected group without any business reason. We outline the steps for proving disparate impact in a legal setting and its analogy to machine learning. We concluded with real c...
SOTA methods for Imitation Learning are heavily data-inefficient or rely on unstable Adversarial training. We present a novel framework that introduces simple, stable, and data-efficient learning from few experts that scales well to complex environments.
We provide a Bayesian inference framework for in-context learning in large language models like GPT-3 and show empirical evidence for our framework, including connections to how in-context learning can still work well despite randomizing the labels in few-shot examples.
LinkBERT is a new language model pretrained to capture document link knowledge such as hyperlinks of the web. It greatly helps knowledge-intensive applications such as question answering.
Machine learning models that achieve high overall accuracy often make systematic errors on coherent slices of validation data. Can we develop methods to automatically identify these systematic errors?
Where do the rewards for robotic reinforcement learning come from? In this blog post we study how using crowdsourced language annotations and videos of humans, we can learn reward functions in a scalable way and enable them to generalize more broadly.
We present an almost-linear time algorithm for the k-medoids problem that matches prior SOTA in clustering quality. Our solution has almost the same complexity as k-means and several advantages.
We show that selective classification, where models are allowed to abstain when they are uncertain, can fail to improve and even hurt accuracy over certain subpopulations of the data.
By tapping into knowledge stored explicitly in text corpora, retrieval helps tackle the inefficiency, opaqueness, and static nature of large language models.
How can we use machine learning to fix source code errors (e.g. in C, Python) for us? We introduce Break-It-Fix-It, a new unsupervised method to train code repair models.
A retrospective narrative from the Hazy research lab on our work in data-centric AI, and current efforts on engaging the broader machine learning community.
A novel computational tool for policymakers to assess the impacts of thousands of different mobility measures on predicted COVID-19 infections, helping them to navigate difficult tradeoffs between the economy and public health.
We conducted an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline h...
A meta-learning system for automatic feedback to student code education with applications to university level courses and a large scale online learning platform.
In this post, we examine in-context learning in GPT-3 through a series of experiments dissecting GPT-3's ability to extrapolate to less natural inputs.
Last year, Stanford won 2nd place in the Alexa Prize Socialbot Grand Challenge for social chatbots. In this post, we look into building a chatbot that combines the flexibility and naturalness of neural dialog generation with the reliability and practicality of scripted dialogue.
By exploiting architectural symmetry, our work demonstrates that we can analytically describe the learning dynamics of various parameter combinations at finite learning rates and batch sizes for state of the art architectures trained on any dataset.
Humans have a pretty good sense of scale, like the weight or price of different objects, but do pre-trained language representations? We measure how well pre-trained text representations capture scale and also come up with a improved version of BERT that captures scale better.
Machine learning models rely on some spurious features that we humans know to avoid. A natural remedy is to remove such features from the data. However, in this work we show that removal of spurious features can decrease accuracy due to the inductive biases of overparameterized models.
Discriminatory behavior towards certain groups by machine learning (ML) models is concerning in critical applications such as hiring. We show that when ML models use noisy features, they're incentivized to devalue a good score when the candidate is from a lower-performing group.
A major long-term goal of computational neuroscience is to identify the brain's learning algorithms. Can we use artificial neural networks to guide this discovery?
We introduce a framework for multi-agent interaction that represents the low-level policies of non-stationary agents with high-level latent strategies.
Identifying rare people, places, and things (entities) in text is a critical problem in AI, especially as the majority of entities are rare. We address this problem in our self-supervised system Bootleg. Bootleg learns to reason over entity types and relations for improved rare entity performance.
When estimating bias in NLP, there is usually uncertainty. We propose capturing this uncertainty using confidence intervals, finding that the datasets currently used to measure bias are too small to conclusively identify it – except in the most egregious cases.
We study how to use machine learning to repair programs from error messages, and introduce a promising approach that leverages program-feedback graphs and self-supervised learning.
Introducing the Stanford MLSys Seminar Series on the frontier of machine learning systems. How does ML change the modern programming stack, and what does it mean for how people will build and deploy applications in the future?
We developed Generalization Through Imitation (GTI) - an algorithm for learning visuomotor control from human demonstrations and generalizing to new long-horizon tasks by leveraging latent compositional structures.
We present a simple way to extend the capabilities of language models to fill in the blanks. Our framework constructs training examples by masking random spans to generate pairs of inputs and targets and fine-tunes language models on the concatenation of each pair.
Validation of safety-critical autonomous systems is necessary before deployment and is made possible by black-box validation strategies. In this blog post, we provide a model for black-box safety validation and survey the literature to uncover the most effective validation techniques.
Adapting to new environments and tasks (e.g., cooking in a new kitchen) requires first gathering information via exploration (e.g., finding the ingredients). We identify a chicken-and-egg coupling problem that prevents existing approaches from effectively exploring, and we solve this with DREAM.
Recent advances in deep generative modeling have brought forth a paradigm shift in trajectory forecasting. In this blog post, we provide an overview of existing work, highlight new opportunities, and present our latest work on developing methods that are cognizant of their downstream use cases.
Large neural language models such as BERT have seen surprising success with multilingual pre-training on a large number of languages. We demonstrate that Multilingual BERT learns cross-lingual syntax, visualizing its inherent structure.
Contact-rich manipulation tasks in unstructured environments often require both tactile and visual feedback. In this blog post, we introduce how to use self-supervision to learn a compact and multimodal representation of vision and touch.
Data augmentation is a de facto technique used in nearly every state-of-the-art machine learning model. In this blog post, we provide an overview of recent work on the practice, theory and new direction of data augmentation research.
We propose a hierarchical planning algorithm in learned latent spaces. Our method uses deep generative models to prioritize promising actions for sampling-based planning.
Replacing static vectors with contextualized word representations has led to significant improvements on virtually every NLP task. In this blog post we study the geometric properties of contextualized word representations and find surprising conclusions.
To create human-like robots, we need to understand how humans behave. We present a modeling approach enables robots to anticipate that humans will make suboptimal choices when risk and uncertainty are involved.
Does adding a theorem to any paper increase its chance of acceptance? Which writing style in an email would get the fastest response, regardless of its subject? We introduce algorithms and a python package for measuring the importance of ngram features on an outcome, while controlling for confounds.
We introduce RoboNet, an open database for sharing robotic experience, and study how this data can be used to learn generalizable models for vision-based robotic manipulation. We find that pre-training on RoboNet enables faster learning in new environments compared to learning from scratch.
We want to make it easier for humans to teleoperate dexterous robots. We present a learning approach that embeds high-dimensional robot actions into an intuitive, human-controllable, and low-dimensional latent space.
We built a system that enables collecting large-scale robot manipulation datasets with human supervision and used it to collect the largest robot dataset ever collected via teleoperation.
When groups robots work together, their actions communicate valuable information. We introduce a collaborative learning and control strategy that enables robots to harness the information contained within their partner's actions.
The NLP community has made great progress on open-domain QA, but our systems still struggle to answer complex open-domain questions in an large collection of text. We present an efficient and explainable method for enabling multi-step reasoning in these systems.
Presenting AC-Teach, a unifying approach to leverage advice from an ensemble of sub-optimal teachers in order to accelerate the learning process of actor-critic reinforcement learning agents.
Introducing a new method that achieves minimax-optimal probably approximately correct (and regret) bounds which match the statistical worst-case lower bounds in the dominating terms for reinforcement learning.
Topology is a combinatorial property that is tricky to utilize in gradient based methods, but it is also a useful and underexploited feature of data. We present an easy-to-use TopologyLayer that allows for backpropagation through a loss based on Persistent Homology.
Looking into multiple attributes of generated text and human-evaluate multiple aspects of conversational quality, in order to investigate how effectively we can control these attributes and how these attributes affect conversational quality and chatbot performance.
We introduce the problem of real-time routing for an autonomous vehicle that can use multiple modes of transportation through other vehicles in the area. We also propose a scalable and performant planning algorithm for solving such problems.
QuizBot is an AI-powered chatbot to help college students review questions through natural-language conversations. Our experimental results suggest that educational chatbot systems may have beneficial use, particularly for learning outside of traditional settings.
When learning from humans, we typically use data from only one form of human feedback. In this work, we investigate whether we can leverage data from multiple modes of feedback to learn more effectively from humans.
Machine learning practitioners are spending less time on model architectures and hardware optimizations and, instead, focusing on training data. We describe three powerful abstractions that practitioners can use to programmatically build and manage their training data.
On learning well-controlled fair representations of data, which make it much more difficult for any downstream machine learning models to discriminate.
Human–object interactions are multi-stepped and governed by physics as well human goals, customs, and biomechanics -- how can we teach machines to capture, understand, and replicate these interactions?
An overview of research at SAIL related to new techniques that allow us to look inside the black box of neural networks, to how it is possible to find and remove bias, and to how safety in autonomous systems can be assured.
Summarizing work on two recent NLP datasets — CoQA and HotpotQA — and how they enable more conversational, explainable, and capable question answering systems.
What are the ethical responsibilities of AI researchers? Or to put it in more pragmatic terms, what are best practices AI researchers can follow to avoid unintended consequences of their research?
Is China winning the global AI race? How will developments in AI impact the job market, wealth inequality and the global economy? Dr. Kai-Fu Lee and Professors Susan Athey and Erik Brynjolfsson discuss this and more.
Efficient reward learning is hard. With a focus on preference-based learning methods, we show how sample-efficiency can be achieved along with computational efficiency by using batch-active methods.
