Skip to content

People Tracking Example

Introduction

Section titled “Introduction”

This example uses DeepStream, YOLOv8, and TensorRT to recognize and track target personnel in the scene, then calculates target position and sends it to the motion host, enabling the robot to follow the target. Based on DeepStream hardware decoding for h264-encoded 720p RTSP video stream, using TensorRT-accelerated YOLOv8 person detection model, recognizing people in open scenes at approximately 20fps, with tracking at approximately 10fps.

This example consists of two parts:

  1. Recognition Algorithm: Deep learning neural network recognition on visual input to locate human bodies in images. When multiple people appear, all detected bodies are identified first, then deep learning features are extracted to determine the same person’s trajectory across frames.
  2. Tracking Algorithm: Users can select a target person to follow. The robot achieves real-time locking and continuous tracking. The recognition algorithm determines the person’s direction and distance relative to the robot, making the robot respond accordingly (translation or rotation), and adjusts speed based on distance:
    • Too close: Robot stops to prevent collision.
    • Near: Robot dynamically decelerates to approach the target.
    • Far: Robot moves at maximum speed.

The yolov8 and sdk_hub source code come from ultralytics and hub-sdk respectively.

Usage

Section titled “Usage”

Note: The program requires approximately 40s to initialize video decoding and deep learning inference environments. If it takes too long to start, connect a controller to check if the video stream is working.

  1. Open a terminal and run:
Terminal window
cd /home/ysc/lite_cog/track/src    # for ROS1
cd /home/ysc/lite_cog_ros2/track/src    # for ROS2
python3 run_tracker.py

  1. Use the App to stand up the robot and enable autonomous mode.

  2. When a person appears, the system numbers all recognized persons and displays them on screen. Enter the target person number using the keyboard and press Enter.

Note: Ensure the video window is on top when entering the target number.

  1. The robot will track and follow the target.

  2. During tracking, press Enter to reset the target, or reset when “Miss Object” is displayed.

  3. Press Ctrl+C to end the program.

Development Notes

Section titled “Development Notes”

This example provides two packages in ~/lite_cog/track and ~/lite_cog_ros2/track workspaces for different ROS versions. The difference is only in kUseRos1Transfer = True in RobotController.py under /src/RobotController, specifying ROS1 or ROS2 for message sending.

people_tracking package structure:

People_tracking
├── model
│   ├── export_engine.sh
│   ├── yolov8n_amd.engine
│   ├── yolov8n_arm.engine
│   ├── yolov8n.onnx
│   └── yolov8n.pt
└── src
    ├── GStreamerWrapper
    │   └── GStreamerWrapper.py
    ├── hub_sdk
    ├── RobotController
    │   ├── FpsCounter
    │   │   ── FpsCounter.py
    │   ├── RobotController.py
    │   ├── ROSTransfer
    │   │   ├── ROS1Transfer.py
    │   │   ├── ROS2Transfer.py
    │   │   └── TransferConstants.py
    │   └── YoloWrapper
    │        ├── CocoTypeId.py
    │        └── YoloWrapper.py
    ├── run_tracker.py
    ├── test
    │   ├── pull.py
    │   ├── pull.sh
    │   └── yolov8.py
    └── ultralytics

File descriptions:

  1. run_tracker.py: Main program.
  2. GStreamerWrapper: DeepStream-based GStreamer hardware decoder for RTSP video stream.
  3. RobotController.py: Main logic in Run() function for recognizing human bodies and sending motion commands.
  4. ultralytics: Open-source YOLOv8 package for inference and tracking. The ultralytics/cfg folder contains configuration files with detailed comments.
  5. sdk_hub: Open-source sdk_hub package, a runtime dependency for YOLOv8.