10. Visual 3D Grasping#

A step-by-step method for picking and placing objects on a table with an AIRBOT arm and vision system.

Prerequisites:

• Click here to learn how to install the AIRBOT Play driver and Python SDK

• Click here to learn how to connect the AIRBOT Play hardware

• Click here to learn how to power up the AIRBOT Play arm

• Click here to learn how to perform automated camera-intrinsic and -extrinsic calibration.

10.1 Environment Setup#

10.1.1 Python Environment#

The vision hardware used in this article is the Intel RealSense camera family.

10.1.2 RealSense Camera Driver#

10.1.3 Physical Setup#

Choose a desktop with a white background whenever possible. This demo uses color-space segmentation to detect small wooden blocks; blue, green, or red backgrounds may cause recognition to fail.

10.1.4 Environment Installation Steps#

Install RealSense camera driver. On Ubuntu you can skip manually downloading the SDK—just run:

Activate the conda virtual environment

conda activate [airbot]

Install pyrealsense2:

pip install pyrealsense2 -i https://pypi.tuna.tsinghua.edu.cn/simple

10.2 Launch the Robot Arm Control Service#

Open a terminal and start the arm control service:

airbot_fsm -i <INTERFACE> -p <PORT>

airbot_server -i <INTERFACE> -p <PORT>

• <INTERFACE> interface name shown by ip link ; default device names are canX (X starts at 0), e.g. can0, can1.
• <PORT> – gRPC server listen port. Use a unique port for each device, e.g. 50000, 50001, 50010. Key points: Example:

airbot_server -i can0 -p 50051

10.3 Run stacking_bricks.py – the arm-vision vertical-grasp demo.#

10.3.1 Download and extract Stacking_Bricks.zip:#

#Must match the `<PORT>` used in the previous step (`airbot_fsm -i can0 -p 50051`).
python3 -m stacking_bricks -p 50051

Package contents explained

- stacking_bricks.py – implements vision-based vertical grasping

• brick_detect_realsense.py – Performs brick detection with the RealSense camera.

• arm_kdl.py – AIRBOT_Play kinematics

• airbot_realsense.py – RealSense configuration

• configs – camera parameters & calibration data

10.3.2 Intrinsic & extrinsic parameter configuration#

For details on obtaining the intrinsic and extrinsic parameters, please refer to Automated Camera Intrinsic & Extrinsic Calibration

Save the .txt file obtained from Automated Camera Intrinsic & Extrinsic Calibration in this directory; the program will automatically read the camera's intrinsic and extrinsic parameters from it.

10.3.3 Program Execution & Demonstration#

Open a terminal in the package directory.

conda activate airbot
python3 -m stacking_bricks

Once the program is running, the screen will show:

• Green rectangles mark the detected small bricks, with parameters for brick color and its coordinates in the camera frame
• This demo can detect blue, green, and red bricks

Press q to close the window and start stacking; a darker background is recommended for higher recognition accuracy.