Isn't it wasteful to represent videos with full pixel-level detail? VideoFlexTok represents videos with a flexible-length coarse-to-fine sequence of tokens, where the first tokens capture abstract information (e.g., semantics and motion) and later tokens add finer details. See how the concept of a car and its motion are captured down to 1-4 tokens (first row).

Do we need Internet-scale datasets to train vision models useful in a specific deployment environment? With Test-Space Training, we show that using only unlabeled multimodal data from the test environment, we can train specialist vision models that outperform generalist models trained on large-scale Internet data.

Different from language-focused evals, we evaluate popular multimodal foundation models (GPT-4o, Gemini, Claude, etc.) on standard computer vision tasks using established datasets (e.g., COCO, ImageNet). While these models demonstrate respectable generalist capabilities, we find that they lack behind specialist models, especially in geometric tasks.

We introduce a joint inference framework, which makes predictions for multiple inputs simultaneously, leveraging inter-sample dependencies, unlike standard zero-shot inference that makes independent predictions.

How far can a 1-pixel camera go? This project explores solving different vision tasks using a simple photoreceptor (think 1-pixel camera) instead of a high-resolution camera. We find that 1) such simple sensors can solve different vision tasks and 2) their design (e.g., placement) is crucial for their effectiveness.

We propose a method to generate synthetic training data specifically useful for a given supervised model and image domain. We introduce two feedback mechanisms to guide the generation: 1) model-based and 2) target distribution-based.

Co-organized a tutorial on computational design.

We study the key components of diversification methods that were recently shown to achieve state-of-the-art OOD generalization. We show: 1) they are sensitive to the distribution of unlabeled data, 2) diversification alone is insufficient for OOD generalization and the right learning algorithm is needed and 3) the choice of the learning algorithm and the unlabeled data distribution are co-dependent.

What are the tasks a neural network can generalize on? How do they look, and what do they indicate? We introduce a task discovery framework to approach these questions empirically. These tasks reflect the inductive biases of NNs learned with SGD and can shed more light on how they generalize. Or do not! We show how one can use these tasks to reveal the failure modes of NNs via creating adversarial train-test data partitions.

A new pre-training method that captures and leverages cross-modal interactions.

3DCC is a set of more realistic 3D corruptions that can be used as a benchmark or data augmentations.

We propose a flexible prior distribution over convolutional kernels of Bayesian neural networks.

We apply conditional normalizing flows to semi-supervised learning. We utilize multiscale architecture for computational efficiency.

We propose a probabilistic view on Batch Normalization and an efficient test-time averaging technique for uncertainty estimation in batch-normalized DNNs.