Behavioral cloning mystery

I've recently started working in robotics, where I've had the chance to train many behavioral cloning (BC) and reinforcement learning (RL) policies on real robot data. My impression so far is: real-world demonstration data is very different from the data in existing simulated RL benchmarks (e.g., D4RL, OGBench, etc.). In particular, I've noticed that there are many mysterious phenomena that appear with real robot data, which are not easily observed in many standard BC and RL benchmarks.

These mysteries really confused me, so I decided to properly study them. Unfortunately, it's really difficult to investigate this in the real world, because nothing is fully reproducible in robotics. In the real world, results depend on everything, including lighting conditions, background, reset distribution, robot temperature, and so on. This means that even when you suspect that something weird is happening, you can't be 100% sure whether it's actually a thing or just noise.

To do proper science, I believe it's important to reproduce them in a controlled benchmark. So I scripted policies for simulated manipulation tasks to generate data that mimic the properties as human demonstrations. After lots of iterations, I was able to successfully reproduce many of the mysteries that I wanted to study in this benchmark.

In this blog post, I'll discuss these mysteries, especially the BC-related ones, based on concrete evidence that I reproduced in the simulated benchmark. I'll also discuss the potential causes and implications of these phenomena, and share my perspectives on them.

The setup

Here are some of the tasks I made for this study. These are standard robotic manipulation tasks, such as block stacking, the Tower of Hanoi, and bowling.

I then scripted policies to generate human-like data. Of course, perfectly mimicking human demonstrations is impossible (it's equivalent to solving robotics...). However, what is possible is to mimic the properties of human demonstrations. It turned out that this was enough to reproduce many of the peculiar phenomena that happen with real robot data.

The key property of human demonstrations is that they are very narrowly distributed, highly temporally correlated (non-Markovian), and "smoothly" random.

To illustrate how the new benchmark is different from our previous one (OGBench), I made a video comparison below.

As you can see, the new data is more natural, diverse, and similar to real human demonstrations in terms of both speed and motion (although not perfect).

Moreover, the new benchmark supports MJWarp, which is basically GPU-accelerated MuJoCo. This means that we can do infinite-data(!) training, by continuously generating fresh demo data on the fly, training the agent using the data exactly once, re-generating new data, and so on. This enables us to study the "true limit" (or "scalability") of different BC and RL algorithms under infinite data.

For behavioral cloning, I used standard flow matching to train a length-25 action chunking policy: \(\pi(a_{t:t+24} \mid s_t)\). Since everything is state-based, the policy is simply modeled by an MLP (but it's a huge MLP; see below!). A bit surprisingly, I found that many of the mysteries can be reproduced even in this state-based setting, without having to go to VLAs. This means that these phenomena are due to the properties of the data rather than the model architecture.

The mysteries

Now I'll discuss four behavioral cloning mysteries. To reiterate, the results below are my "reproductions" of real-world phenomena in sim, meaning that they're not limited to these specific sim environments.

Disclaimer: I don't claim that any of these mysteries are novel! Many of them are probably already known to practitioners in robotics. The goal of this blog post is to show that these "folkloric" phenomena are real and reproducible, and to discuss their implications.

Mystery 1: Overfitting is... not bad?

The first mystery is that overfitting is often beneficial in behavioral cloning. This is what I found most counterintuitive. One might think that overfitting is bad and should be avoided. It turns out that it's often the opposite: in many cases, overfitting is not necessarily harmful; it's sometimes even helpful.

Here's the result. The plots above are from flow BC on a block pick-and-place task with 10K episodes. You can see that performance simply gets better (and then stabilizes) as we train more and more, even when the validation flow loss keeps growing.

Even more confusingly, I found that in some cases, using a larger dataset can hurt(!) performance. The plots above show that 10K datasets are better than 50K datasets on the same task. These datasets are sampled from the same distribution, and the results are averaged over 4 random seeds with 4 independently sampled datasets each, so it's not just a random fluke! This indicates that overfitting is sometimes even desirable, for some reason.

Mystery 2: Open-loop is better than closed-loop

The second mystery is that open-loop control is essential in behavioral cloning. For those who are not familiar with the terminology, here's the definition:

• Open-loop policy \(\pi(a_{t:t+24} \mid s_t)\) is a policy that generates a chunk of actions, and blindly executes the full chunk without replanning while executing it.
• Closed-loop policy \(\pi(a_t \mid s_t)\) is a policy that simply generates and executes a single action at each timestep.

Of course, if we ignore practical issues like latency, closed-loop policies should be strictly better than open-loop policies because they're more reactive. Right?

Well, it turns out that a pure closed-loop policy doesn't work at all! Importantly, this is under the infinite data setting: the policies are trained on a stream of freshly generated data. So the closed-loop policy is not failing due to data scarcity or overfitting issues.

Here are the videos. Honestly I was (and still am) surprised by how bad the closed-loop policy is, even on this very simple task with infinite data. It doesn't even touch the cube! And yes, I double-checked the code.

Mystery 3: Policies should be very large

Here's a question: how large do you think a policy should be to learn the pick-and-place behavior in the videos above? Recall that everything is state-based. Initially, I thought that a [512, 512, 512]-sized MLP would be more than enough. After all, the task is simple and fixed (it's not goal-conditioned), and the state space has only 37 dimensions.

I was completely wrong.

It turns out that we need at least [4096]*8-sized(!) residual MLPs to learn this task well. And 8192-dimensional MLPs further improve the performance! This is almost a billion-scale (0.5B) model. Personally, I've never seen such large MLPs being used in state-based single-task environments before. For example, previous benchmarks like D4RL and OGBench typically require much smaller MLPs ([1024]*4 at most).

Mystery 4: Features matter even with infinite data

Given infinite data and compute, one might think that we don't need to worry about manual feature engineering, as the model will autonomously learn the right features from data. This is part of the Bitter Lesson.

It turns out that this is not necessarily the case in behavioral cloning.

The plot above compares two input features, which only differ by scaling:

• Handcrafted means that I manually choose the scale of each state dimension.
• Standardized means that each state dimension is standardized to the same range.

They both contain exactly the same information. Indeed, their flow losses and MSEs are almost identical, as expected. However, their actual performance is clearly different. Recall that this is again under the "infinite" data setting with a sufficiently large model, so it's not due to overfitting or underfitting.

Closing thoughts

In this post, I discussed four real-world behavioral cloning "mysteries" based on their reproductions in sim environments. I'd like to conclude by sharing a few of my perspectives on these phenomena, and what they might mean for VLAs and large-scale robotic control.

1. As we've seen throughout this post, the fundamental cause behind these mysteries is test-time distribution shift. Interestingly, autoregressive LLMs don't seem to suffer from this issue as much, even though they should have the same problem in principle. Why?

Here's my hot take: I think this test-time distribution shift is actually an impossible problem. That is, the only solution is to avoid this problem, by scaling data to the degree that everything becomes "in-distribution" at test time. This explains why LLMs don't suffer as much from this issue: since they're trained on internet-scale data, they build a very robust internal representation, so that any text is virtually in-distribution in the representation space at test time. This also explains why we can simply use the training loss as a good proxy for performance in LLMs. So, as we scale up data and models in VLAs to the level of LLMs, many of these mysteries might eventually disappear!

2. Another major cause behind these phenomena is the expressivity mismatch between non-Markovian data distributions and (chunked) Markovian policies. I think this is a huge blocker for scalable robotic control, as it imposes a strong and incorrect inductive bias on the policy.

There seem to be two potential solutions to this issue. One is to use fully history-conditioned, autoregressive policies. Another is to use hierarchical policies that first output a detailed plan that kills off multimodality, so that the low-level policy can be more Markovian. Of course, these solutions are non-trivial and are research topics in themselves, but I think we'll eventually need to close this expressivity gap to further scale up robotics.

Acknowledgments

I'd like to thank Kyle Stachowicz, Sergey Levine, Srinath Mahankali, and Kevin Black for helpful discussions and feedback on this post.