Laumy的技术栈
首页
支持与合作

lerobot之smolvla体验

🕒 2025-08-18 📁 lerobot 👤 laumy 🔥 29 热度

环境安装

pip install -e ".[smolvla]"

在原来lerobot的环境基础上。

启动训练

本文主要是记录复现lerobot smolvla策略的效果，为了快速看到效果，这里不进行采集数据了，直接用此前ACT采集的数据，将数据打包放到autodl云服务器上进行训练。

python src/lerobot/scripts/train.py \
    --dataset.root=/root/autodl-tmp/lerobot/data/record-07271539 \
    --dataset.repo_id=laumy/record-07271539 \
    --policy.push_to_hub=false   \
    --policy.path=lerobot/smolvla_base \
    --policy.device=cuda   \
    --output_dir=outputs/train/smolvla_test   \
    --job_name=smolvla_test   --batch_size=64 \
     --steps=20000   --wandb.enable=false


或者
    python -m lerobot.scripts.train \
      --policy.type=smolvla \
      --policy.vlm_model_name=HuggingFaceTB/SmolVLM2-500M-Video-Instruct \
      --policy.load_vlm_weights=true \
      --policy.num_vlm_layers=16 \
      --policy.num_expert_layers=8 \
      --dataset.repo_id=laumy/record-07271539 \
      --output_dir=outputs/train/smolvla_test2 \
      --job_name=smolvla_test \
      --batch_size=64 --steps=20000 --wandb.enable=false

关于数据集的获取取决于两个参数，一个是repo_id另外一个是dataset.root。

  • repo_id: 必填字段，是在 Hugging Face Hub 上的数据集标识（datasets 仓库名）。
  • dataset.root ：选填字段，是本地数据集所在目录。

训练首先从 dataset.root 读取本地数据；如果本地缺失需要的文件，才会用 repo_id 到 Hub 拉取缺的内容到这个 root 目录里。

下面有几个场景。

如果同时给定了dataset.root和dataset.repo_id

  • 如果 root 目录已经是规范的 LeRobot v2 数据集结构（有 meta/info.json、data/.parquet、可选 videos/.mp4），会直接用本地文件，不会下载。
  • 如果本地缺少 meta（或部分 data 文件），代码会用 repo_id 从 Hub 把缺的部分同步到你指定的 root 目录后再加载。

如果只传dataset.repo_id
会把本地根目录设为默认缓存：~/.cache/huggingface/lerobot/{repo_id}（若设置了环境变量 LEROBOT_HOME，则用 $LEROBOT_HOME/{repo_id}），如果本地缓存里已经有完整数据，则直接用本地文件，不再下载。如果本地没有缓存，远端也没有数据，就会报错。

推理验证

python -m lerobot.record    \
    --robot.type=so101_follower   \
    --robot.disable_torque_on_disconnect=true   \
    --robot.port=/dev/ttyACM0 --robot.cameras="{ handeye: {type: opencv, index_or_path: 4, width: 640, height: 480, fps: 30}, fixed: {type: opencv, index_or_path: 6, width: 640, height: 480, fps: 30}}"   \
    --robot.id=R12252801   \
    --display_data=false   \
    --dataset.single_task="Grab the cube"   \
    --policy.path=outputs/smolvla_weigh_08181710/pretrained_model  \
    --dataset.episode_time_s=240    \
    --dataset.repo_id=laumy/eval_smolvla_08181710

常见问题

训练报错如下：

Traceback (most recent call last):
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/connection.py", line 753, in connect
    self.sock = sock = self._new_conn()
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/connection.py", line 207, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x7fe651566380>, 'Connection to huggingface.co timed out. (connect timeout=None)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/HuggingFaceTB/SmolVLM2-500M-Video-Instruct/tree/main/additional_chat_templates?recursive=False&expand=False (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fe651566380>, 'Connection to huggingface.co timed out. (connect timeout=None)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/autodl-tmp/lerobot/src/lerobot/scripts/train.py", line 291, in <module>
    train()
  File "/root/autodl-tmp/lerobot/src/lerobot/configs/parser.py", line 226, in wrapper_inner
    response = fn(cfg, *args, **kwargs)
  File "/root/autodl-tmp/lerobot/src/lerobot/scripts/train.py", line 139, in train
    policy = make_policy(
  File "/root/autodl-tmp/lerobot/src/lerobot/policies/factory.py", line 168, in make_policy
    policy = policy_cls.from_pretrained(**kwargs)
  File "/root/autodl-tmp/lerobot/src/lerobot/policies/pretrained.py", line 101, in from_pretrained
    instance = cls(config, **kwargs)
  File "/root/autodl-tmp/lerobot/src/lerobot/policies/smolvla/modeling_smolvla.py", line 356, in __init__
    self.language_tokenizer = AutoProcessor.from_pretrained(self.config.vlm_model_name).tokenizer
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/transformers/models/auto/processing_auto.py", line 288, in from_pretrained
    config_dict, _ = ProcessorMixin.get_processor_dict(pretrained_model_name_or_path, **kwargs)
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/transformers/processing_utils.py", line 873, in get_processor_dict
    for template in list_repo_templates(
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/transformers/utils/hub.py", line 161, in list_repo_templates
    return [
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/transformers/utils/hub.py", line 161, in <listcomp>
    return [
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 3168, in list_repo_tree
    for path_info in paginate(path=tree_url, headers=headers, params={"recursive": recursive, "expand": expand}):
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/huggingface_hub/utils/_pagination.py", line 36, in paginate
    r = session.get(path, params=params, headers=headers)
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 96, in send
    return super().send(request, *args, **kwargs)
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/requests/adapters.py", line 688, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/HuggingFaceTB/SmolVLM2-500M-Video-Instruct/tree/main/additional_chat_templates?recursive=False&expand=False (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fe651566380>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: 7f4d5747-ec95-47cc-a55f-cb3e230c52e2)')

原因是训练在初始化 SmolVLA 的 VLM 时需要从 Hugging Face Hub 拉取资源（AutoProcessor.from_pretrained 默认用 vlm_model_name=HuggingFaceTB/SmolVLM2-500M-Video-Instruct）。你的机器连到 huggingface.co 超时，导致下载失败并报 ConnectTimeout。

解决办法：export HF_ENDPOINT=https://hf-mirror.com

把原本指向 https://huggingface.co 的所有 Hub 请求（模型/数据集下载、API 调用）改走 https://hf-mirror.com。作用范围仅当前这个终端会话。关闭终端或开新终端就失效。

训练过程过程中警告

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

TOKENIZERS_PARALLELISM是分词器在一次调用会开多线程并行处理文本。分词器库是hugging Face的分词器库，负责把文本指令变成模型可用的token id序列，也能把id还原会文本，跟我们此前在一步步实现transformer

的词表类型。出现这样的警告是tokenizers它开了多线程并发，而 DataLoader 再 fork 出子进程并发（本身DataLoader是可以并发），这样容易有死锁风险，为安全起见，库检测到这种顺序就自动把自己的多线程并行关掉，并给出提示。如果要关掉tokenizers的多线程并发，export TOKENIZERS_PARALLELISM=false。

发表你的看法

\t↑