Thermal-aware robotic manipulation

Thermo-VLA

Thermal-Aware Vision-Language-Action Models with Gated Residual Fusion for Robotic Manipulation

Jingxuan Chen, Junshi Luo, Chencheng Dong, Xiaojian Nie, Tongge Huang, Qiang Huang

Recent Vision–Language–Action (VLA) models have begun incorporating thermal sensing, but existing approaches typically treat thermal images as ordinary visual inputs, providing limited support for temperature-aware relational reasoning in manipulation. To address this limitation, we propose Thermo-VLA with a ThermoRes Encoder, which decomposes thermal images into cold and hot attribute representations and injects them into the RGB token through language-conditioned residual modulation, enabling temperature-driven target selection and continuous manipulation. We further construct six real-world robotic tasks on a dual-arm Unitree G1 platform, spanning basic grounding to relational reasoning. Experiments show that Thermo-VLA achieves an average success rate of 0.86 across four thermal-dependent tasks, outperforming a π0.5 baseline that directly incorporates RGB–thermal observations by 18 percentage points. Under unseen-object and scene variations, Thermo-VLA maintains an average success rate of 0.78, achieving the best generalisation performance among the compared methods. Beyond thermal sensing, we transfer the same residual modulation mechanism to colour attributes and observe consistent improvements in both simulation and real-robot experiments, indicating its potential to generalise beyond thermal attributes.

Paper (Coming Soon)
Arxiv
Appendix
Data (Coming Soon)
Code (Coming Soon)

Overview

Thermo-VLA overview figure

Real-world Benchmark

Comprising four temperature-related tasks and two RGB-based tasks for evaluating representative VLA methods under RGB and thermal settings.

ThermoRes Encoder

Combining temperature-attribute decomposition and text-conditioned residual injection for temperature-driven manipulation.

Best Performance

Achieving the best overall performance on the proposed benchmark and showing strong generalisation to unseen objects and scene variations.

Method

Thermo-VLA builds on a VLA policy and augments it with thermal attribute residuals. RGB observations are encoded by the RGB visual encoder, while language instructions and robot states are mapped to textual embeddings. ThermoRes Encoder decomposes thermal observations into cold and hot attributes and injects them into RGB tokens through context-gated residuals. The resulting multimodal prefix is used for flow-matching policy training and continuous action generation.

Thermo-VLA method figure

Experiment

The experiments are conducted on a Unitree G1 EDU humanoid robot with Unitree Dex3-1 hands, a head-mounted Intel RealSense D435i RGB camera, two wrist-mounted RealSense D405 cameras, and a Tiny1-C thermal camera. Training is performed on an NVIDIA A800 80GB GPU, and policy inference runs on an NVIDIA GeForce RTX 5090 48GB GPU.

Unitree G1 data collection setup

Tasks

Task Difficulty Episodes
I. Position-guided bottle selection Spatial grounding 156
II. Temperature-guided bottle selection Thermal perception 303
III. Identity-guided object selection Appearance grounding 236
IV. Temperature-guided object selection Thermal grounding 340
V. Temperature-aware coffee service Thermal ranking 173
VI. Abnormal-animal cage removal Thermal exclusion 99

Results

Comparison Experiments

In the comparison experiments, Thermo-VLA achieves the best overall performance on the proposed benchmark while preserving strong RGB manipulation ability. Its advantage is most evident on temperature-dependent tasks that require relative thermal reasoning or exclusion of thermally abnormal targets.

Task Avg. ACT π0.5 π0.5 (RGBT) ThermoAct Thermo-VLA
I. Position-guided bottle 0.75 0.93 0.85 0.45 0.88
II. Temperature-guided bottle - - 0.80 0.45 1.00
III. Identity-guided object 0.69 0.84 0.71 0.49 0.76
IV. Temperature-guided object - - 0.85 0.55 0.86
V. Temperature-aware coffee service - - 0.50 0.00 0.58
VI. Abnormal-animal cage removal - - 0.55 0.50 1.00

Generalisation

In the generalisation experiments, Thermo-VLA also maintains the strongest average performance under temperature reassignment, unseen objects, illumination changes, and unseen tablecloths, showing better robustness to both target and scene variations.

Setting Avg. ThermoAct π0.5 (RGBT) Thermo-VLA
Seen-object temperature reassignment 0.50 0.10 0.60
Unseen objects 0.38 0.64 0.79
Illumination change 0.35 0.68 0.76
Unseen tablecloth 0.34 0.72 0.79

Demonstrations

Task
Case
Prompt:

π0.5 with RGBRes Encoder

We further investigate whether these directions are thermal-specific or attribute-dependent by transferring the same residual mechanism to pure-RGB colour-conditioned tasks. we extend π0.5 with an additional RGBRes Encoder to introduce colour-guided residual modulation into the RGB visual representation.

RGBRes Encoder pipeline

Additional Experiments

To examine whether this structure extends to other attributes, we apply RGBRes to four pure-RGB colour-conditioned tests in simulation and the real world.

Method Task VII Task VII (Tablecloth) Task VIII Task VIII (Tablecloth) Avg.
π0.5 0.707 0.307 0.600 0.675 0.572
π0.5 + RGBRes Encoder 0.873 0.453 0.900 0.875 0.775

We further analyse the learned residual features with PCA across thermal and RGB tasks to verify whether the attribute corrections form a compact and consistent representation. Here, ηk denotes the cumulative centred PCA energy retained by the top-k principal components, and higher values indicate that the residual variation is concentrated in a lower-dimensional subspace.

Task Avg. Cosine η4 η8 η16 η32
Task II 0.942 54.2 66.1 74.5 83.1
Task IV 0.947 58.3 69.3 76.5 84.4
Task VI 0.956 69.6 79.6 84.5 89.6
Task VII 0.999 88.1 95.7 98.4 99.5
Task VII(Tablecloth) 0.999 84.8 94.0 97.6 99.1

Demonstrations

Conclusion

We propose Thermo-VLA, which introduces a ThermoRes Encoder to inject temperature attributes from thermal images into RGB visual representations as text-conditioned residuals, enabling temperature-driven target selection and continuous manipulation while preserving fundamental vision–action capabilities. We establish a real-world benchmark on the Unitree G1 platform, covering RGB-only grounding, temperature-dependent manipulation, and thermal reasoning. Experiments show that Thermo-VLA achieves superior performance across the benchmark and diverse generalisation settings. Beyond thermal sensing, our results support low-dimensional attribute-conditioned residual modulation as a broader representation principle for augmenting generic vision-action representations with additional physical attributes.

Statement

This work is currently under review. © 2026 The Authors. All rights reserved.