Few-Shot Design Optimization
by Exploiting Auxiliary Information

Arjun Mani Carl Vondrick Richard Zemel

Department of Computer Science, Columbia University

arXiv Paper Benchmark Dataset Code (coming soon)
Overview of few-shot design optimization with auxiliary information. Part (a) shows two design optimization problems: robotic gripper design with tactile feedback and neural network hyperparameter tuning with learning curves. Part (b) shows our method's optimization loop.
Few-shot design optimization with auxiliary information. We introduce a design optimization setting where evaluating a design returns high-dimensional auxiliary information h(x) along with the reward f(x). Part (a) shows examples of two design problems in this setting, robot hardware design and neural network hyperparameter tuning. In the first domain, the aim is to design a customized robotic gripper that grasps an object as stably as possible, using tactile feedback of the object during each grasp attempt. In the second domain, evaluating a hyperparameter setting returns per-epoch learning curves, which can provide useful information beyond the reward alone (e.g. indicating overfitting here). Part (b) shows how our model performs optimization of a new design task. Our model Pθ accepts a few-shot context of observations including auxiliary information h(x), predicts f(x) for unobserved designs, selects a promising new design for evaluation, and iterates with an updated context, finally returning the best observed design at termination. Pθ is trained on a history of design tasks to acquire this few-shot prediction ability.

Abstract

Many real-world design problems involve optimizing an expensive black-box function f(x), such as hardware design or drug discovery. Bayesian Optimization has emerged as a sample-efficient framework for this problem. However, the basic setting considered by these methods is simplified compared to real-world experimental setups, where experiments often generate a wealth of useful information. We introduce a new setting where an experiment generates high-dimensional auxiliary information h(x) along with the performance measure f(x); moreover, a history of previously solved tasks from the same task family is available for accelerating optimization. A key challenge of our setting is learning how to represent and utilize h(x) for efficiently solving new optimization tasks beyond the task history. We develop a novel approach for this setting based on a neural model which predicts f(x) for unseen designs given a few-shot context containing observations of h(x). We evaluate our method on two challenging domains, robotic hardware design and neural network hyperparameter tuning, and introduce a novel design problem and large-scale benchmark for the former. On both domains, our method utilizes auxiliary feedback effectively to achieve more accurate few-shot prediction and faster optimization of design tasks, significantly outperforming several methods for multi-task optimization.

Gripper Design Benchmark

We introduce a new design problem, based on designing robotic grippers to grasp objects as stably as possible, using tactile feedback of the object during each grasp attempt. Our benchmark dataset contains 4.3 million gripper design evaluations across 1000 diverse objects (i.e. 1000 design tasks). The design space is parametrized as a cubic Bézier surface, and the initial height of the gripper is also optimized. Below shows an example of a design task and our simulation evaluating the grasp stability.

Two gripper designs for grasping a chair object. The top row shows a lower-quality flat gripper that fails under mild disturbance. The bottom row shows a high-quality gripper discovered by our method that wraps around the chair legs for maximum stability.
A tale of two grippers. Two gripper designs for grasping a chair object. Our simulation for evaluating each design closes the gripper, lifts the object, and applies disturbance forces to the object in a given direction (here, downward). The force is incremented while the object stays in the grasp, and the reward is the maximum force before the object falls, measuring the grasp's stability. The top row shows a lower-quality gripper with a flat surface, which can only resists a mild disturbance before the object falls. The bottom row shows a high-quality gripper discovered by our method, which wraps around both chair legs right under the seat while clasping the front leg above the seat, resisting disturbances in any direction and achieving a high reward. See the close-up gripper view for further clarity on this strategy.

Below shows several examples of gripper design tasks in the test set. The optimal design in the dataset is shown along with the design our model discovers over optimization. Our model discovers sophisticated, creative grasping strategies that maximize grasp stability, such as clasping the seat of a chair to resist any disturbances (first row), clamping thin structures in the object like piano legs (second row), and protruding into gaps in the object (third row). Zoom into each row for best viewing.

Qualitative examples showing a t-SNE visualization of gripper designs and optimal grippers discovered by our method for chairs, pianos, tables, and cars.
Qualitative examples of design tasks in the test set. Top shows a 2D t-SNE visualization of data-points in the test set (10K points out of 650K points total). Each point corresponds to a (20-D) gripper design evaluated for a particular object, with green indicating high-reward designs and red low-reward designs. Right shows the optimal gripper in the dataset and the gripper discovered by our method for several objects. Our method discovers sophisticated grasping strategies that resist disturbance forces in any direction and achieve high reward. Note that optimization is performed over the large finite set of designs evaluated for each task in the dataset.

Citation

@article{mani2026designopt,
  title={Few-Shot Design Optimization by Exploiting Auxiliary Information},
  author={Mani, Arjun and Vondrick, Carl and Zemel, Richard},
  journal={arXiv preprint arXiv:2602.12112},
  year={2026}
}