RL Control Example

This example demonstrates how to run reinforcement learning (RL) policies through the SDK’s built-in state machine to control the Lite3 quadruped robot. Both simulation and real robot modes are supported.

Code repositories: RL policy training · ROS 2 policy deployment

Official video tutorials cover training, UDP deployment and ROS 2 migration. Choose the tutorial for your SDK workflow.

Simulation

Training

# Rough terrain training
python scripts/reinforcement_learning/rsl_rl/train.py \
    --task Rough-Deeprobotics-Lite3-v0 \
    --headless

# Flat terrain training
python scripts/reinforcement_learning/rsl_rl/train.py \
    --task Flat-Deeprobotics-Lite3-v0 \
    --headless

Inference / Playback

python scripts/reinforcement_learning/rsl_rl/play.py \
    --task Rough-Deeprobotics-Lite3-v0 \
    --num_envs 50

Keyboard-controlled playback:

python scripts/reinforcement_learning/rsl_rl/play.py \
    --task Rough-Deeprobotics-Lite3-v0 \
    --num_envs 1 \
    --keyboard

Environment Registration

Environment ID	Terrain	Config Entry
`Flat-Deeprobotics-Lite3-v0`	Flat	`DeeproboticsLite3FlatEnvCfg`
`Rough-Deeprobotics-Lite3-v0`	Rough	`DeeproboticsLite3RoughEnvCfg`

Observation Space

Observation	Dim	Scale	Noise	Note
base_lin_vel	—	—	—	Disabled (set to None)
base_ang_vel	3	0.25	±0.2	Angular velocity
projected_gravity	3	1.0	±0.05	Gravity projection
velocity_commands	3	1.0	—	Velocity commands (vx, vy, ωz)
joint_pos	12	1.0	±0.01	Joint relative position
joint_vel	12	0.05	±1.5	Joint relative velocity
last_action	12	1.0	—	Previous action
height_scan	—	—	—	Disabled (set to None)

Total observation dimension: 48 (45 effective observation dimensions + 3 zero-padding)

Action Space

Parameter	Value
Action dimension	12
Action type	Joint target position (JointPositionAction)
HipX action scale	0.125
HipY / Knee action scale	0.25
Action clip range	(-100.0, 100.0)

Velocity Command Range

Command	Range
Linear velocity X (forward/backward)	-1.5 ~ 1.5 m/s
Linear velocity Y (left/right)	-0.8 ~ 0.8 m/s
Angular velocity Z (rotation)	-0.8 ~ 0.8 rad/s

Reward Function

Tracking Rewards (positive)

Reward	Weight	Description
track_lin_vel_xy_exp	4.0	Linear velocity XY tracking
track_ang_vel_z_exp	1.5	Angular velocity Z tracking
feet_air_time_lin_xy	5.0	Air time under linear velocity command (threshold 0.5s)
feet_air_time_ang_z	5.0	Air time under angular velocity command (threshold 0.5s)
feet_gait	0.5	Gait symmetry reward (diagonal leg sync: FL-HR, FR-HL)
phase_foot_trajectory_exp	2.0	Phase foot-end trajectory tracking
feet_contact_without_cmd	0.1	Foot contact reward when no command

Penalties (negative)

Reward	Weight	Description
action_rate_l2	-0.1	Action rate of change
base_height_l2	-50.0	Base height deviation (target 0.55m)
lin_vel_z_l2	-20.0	Z-direction linear velocity
flat_orientation_l2	-20.0	Body tilt
ang_vel_xy_l2	-0.25	XY plane angular velocity
foot_impact_velocity	-2.0	Foot-end impact velocity at contact
joint_torques_l2	-2.5e-4	Joint torques
joint_acc_l2	-1e-8	Joint acceleration
joint_power	-8e-4	Joint power
joint_pos_limits	-5.0	Joint limit violation
contact_forces	-0.1	Foot-end contact forces
feet_slide	-0.05	Foot-end sliding
stand_still	-0.5	Joint deviation when no command
undesired_contacts	-0.5	Non-foot-end collisions
hipx_joint_pos_penalty	-0.4	HipX joint deviation penalty
knee_joint_pos_penalty	-2.0	Knee joint deviation penalty
joint_mirror	-0.05	Diagonal leg joint mirror symmetry

Domain Randomization

Randomization	Mode	Parameters
Rigid body material	startup	Static friction 0.3~~1.0, dynamic friction 0.3~~0.8
Rigid body mass	startup	Scale 0.85~1.15 (all links)
Rigid body inertia	startup	Scale 0.85~1.15
CoM offset	startup	x/y ±0.03m, z ±0.02m
Joint reset	reset	Position scale 1.0~1.0
Actuator gains	reset	Stiffness/damping scale 0.85~1.15
Base reset	reset	Position x/y ±1.0m, roll/pitch ±0.3rad, yaw ±π; velocity ±0.2m/s

Terrain Configuration (Rough)

Terrain Type	Proportion
random_rough	0.4
hf_pyramid_slope	0.3
hf_pyramid_slope_inv	0.3
boxes	0.0 (disabled)
pyramid_stairs	0.0 (disabled)
pyramid_stairs_inv	0.0 (disabled)

Rough terrain noise range: 0.01 ~ 0.06m, step 0.01m.

Height scan resolution: 0.07m, grid size [1.6, 1.0].

Policy Export

Auto Export via play.py

Running play.py automatically exports JIT and ONNX models to exported/:

python scripts/reinforcement_learning/rsl_rl/play.py \
    --task Rough-Deeprobotics-Lite3-v0

Exported files:

exported/policy.pt (JIT)
exported/policy.onnx (ONNX)

Quick Export (No Isaac Sim Required)

python scripts/tools/export_onnx_fast.py \
    --checkpoint_path logs/rsl_rl/deeprobotics_lite3_rough/2025-01-01_12-00-00/model_5000.pt \
    --robot lite3 \
    --output_path exported/lite3_policy.onnx

Exported ONNX file includes robot metadata:

Joint names, link names, base link name
Joint stiffness / damping
Default joint positions
Action scale

Policy Deployment

Interface Example

PolicyRunnerBase defines input observation, output action, and inference lifecycle:

virtual RobotAction getRobotAction(const RobotBasicState&, const UserCommand&) = 0;
virtual void OnEnter() = 0;
virtual void SetDecimation(int d);

Lite3PolicyRunner

Lite3 ONNX policy implementation.

session_ = Ort::Session(env_, policy_path_.c_str(), session_options_);

Observation and action dimensions:

const int observation_dim = 48;
const int action_dim = 12;

Inference:

VecXf current_action_eigen = Onnx_infer(current_observation_);

Observation assembly:

// Observation dimension: 48 (45 effective + 3 zero padding)
Eigen::VectorXf obs_b = Eigen::VectorXf::Zero(48);
obs_b.block(0, 0, 3, 1) = basic_state.imu.angular_velocity.cast<float>();         // base_ang_vel (3)
obs_b.block(3, 0, 3, 1) = basic_state.imu.projected_gravity.cast<float>();         // projected_gravity (3)
obs_b.block(6, 0, 3, 1) = basic_state.command.getCommand();                        // velocity_commands (3)
obs_b.block(9, 0, 12, 1) = (basic_state.motor_state.position - dof_default_eigen_robot).cast<float>(); // joint_pos (12)
obs_b.block(21, 0, 12, 1) = basic_state.motor_state.velocity.cast<float>();        // joint_vel (12)
obs_b.block(33, 0, 12, 1) = last_action_output_.cast<float>();                     // last_action (12)
// obs_b[45..47] zero padding

Action output:

robot_action.goal_joint_pos = default_joint_pos + action_scale * current_action_eigen;

Policy model file: src/Lite3_sdk_deploy/policy/policy.onnx

Default posture:

dof_default_eigen_robot <<
    0.0, -0.8,  1.6,   // Left front leg
    0.0, -0.8,  1.6,   // Right front leg
    0.0, -0.8,  1.6,   // Left rear leg
    0.0, -0.8,  1.6;   // Right rear leg

Action scaling:

action_scale <<
    0.25, 0.25, 0.25,   // Left front: HipX, HipY, Knee
    0.25, 0.25, 0.25,   // Right front
    0.25, 0.25, 0.25,   // Left rear
    0.25, 0.25, 0.25;   // Right rear

Real Robot Deployment

Additional steps: network connection, file sync, and ARM compilation.

File Sync

ssh user@10.21.31.103 'mkdir -p ~/Lite3_sdk_deploy/src'
scp -r ~/sdk_deploy/src/Lite3_sdk_deploy user@10.21.31.103:~/Lite3_sdk_deploy/src

Compilation

cd Lite3_sdk_deploy
source /opt/ros/foxy/setup.bash
colcon build --packages-select lite3_sdk_deploy --cmake-args -DBUILD_PLATFORM=arm

Launch

source install/setup.bash
ros2 run lite3_sdk_deploy rl_deploy

Control Methods

main.cpp selects input method via RemoteCommandType enum.

Keyboard Control

std::shared_ptr<StateMachineBase> fsm = std::make_shared<qw::QwStateMachine>(
    RobotName::Lite3, RemoteCommandType::kKeyBoard);

Key	Function
z	Stand up (Default Position)
c	RL control
x	Lie down
w a s d	Forward / Left / Backward / Right
q e	Clockwise / Counterclockwise rotation

Gamepad Control

std::shared_ptr<StateMachineBase> fsm = std::make_shared<qw::QwStateMachine>(
    RobotName::Lite3, RemoteCommandType::kGamepad);

Button	Function
L1	Standing Up
L2	RL control
R1	Lie Down
R2	Joint Damping
Left joystick	Forward / Left / Backward / Right
Right joystick	Clockwise / Counterclockwise rotation