Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in
  Robotics

Dongyoung Kim; Sumin Park; Huiwon Jang; Jinwoo Shin; Jaehyung Kim; Younggyo Seo

返回文献探索

Paper · arXiv 2506.00070

Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics

Dongyoung Kim, Sumin Park, Huiwon Jang, Jinwoo Shin, Jaehyung Kim, Younggyo Seo

29 upvotesMay 29, 2025arXiv 预印本

AI 摘要

Robot-R1, a reinforcement learning framework, enhances embodied reasoning for robotics by predicting keypoint states, outperforming supervised fine-tuning methods and even surpassing GPT-4o in low-level action control tasks.

Large Vision-Language Models (LVLMs)embodied reasoningrobot controlSupervised Fine-Tuning (SFT)catastrophic forgettinggeneralization performancereinforcement learningkeypoint statescene imageenvironment metadataexpert demonstrationsDeepSeek-R1reasoning-based responsesparameter-efficient fine-tuning

Abstract

Large Vision-Language Models (LVLMs) have recently shown great promise in advancing robotics by combining embodied reasoning with robot control. A common approach involves training on embodied reasoning tasks related to robot control using Supervised Fine-Tuning (SFT). However, SFT datasets are often heuristically constructed and not explicitly optimized for improving robot control. Furthermore, SFT often leads to issues such as catastrophic forgetting and reduced generalization performance. To address these limitations, we introduce Robot-R1, a novel framework that leverages reinforcement learning to enhance embodied reasoning specifically for robot control. Robot-R1 learns to predict the next keypoint state required for task completion, conditioned on the current scene image and environment metadata derived from expert demonstrations. Inspired by the DeepSeek-R1 learning approach, Robot-R1 samples reasoning-based responses and reinforces those that lead to more accurate predictions. Our experiments show that models trained with Robot-R1 outperform SFT methods on embodied reasoning tasks. Despite having only 7B parameters, Robot-R1 even surpasses GPT-4o on reasoning tasks related to low-level action control, such as spatial and primitive movement reasoning.