Introduction to the ESP32-S31 with Vision: Assignment 3

Detect hands in live camera frames

#

In this assignment, you will run an ESPDet Pico hand-detection model on live camera frames. The application draws a rectangle and confidence score around each detected hand and displays the annotated image on the ESP32-S31-Korvo-1 LCD.

This application performs detection, not tracking. Each frame is processed independently, so detections do not receive persistent IDs.

Hand-detection pipeline

#

graph TD
    Camera[Camera frame] --> Model[ESPDet Pico Hand]
    Model --> Results[Boxes and confidence scores]
    Results --> Overlay[Draw detection overlays]
    Overlay --> LCD[LCD]

espdl.ESPDet performs image preprocessing, inference, and model-specific output decoding. It returns one tuple for each detection that passes the score and non-maximum suppression thresholds:

(x, y, width, height, score, category)

The first four values define the bounding box. score describes the model’s confidence, and category identifies the detected class.

Step 1: Prepare the hand-detection model

#

Download the ESP32-S31 build of the ESPDet Pico Hand 224 × 224 model from the model collection on the ESP-Vision website. Copy it to the root directory of the ESP-Vision flash drive exposed over USB mass storage with this filename:

espdet_pico_224_224_hand.espdl

The model path on the device will be:

/espdet_pico_224_224_hand.espdl

After copying the model to the root directory of the board’s internal flash disk, use the REPL to verify that it is available:

import os
print(os.listdir("/"))

Step 2: Configure the application

#

Create hand_detection.py and import the camera, display, inference, and timing modules:

import display
import espdl
import sensor
import time

Define the model path and detection thresholds:

MODEL = "/espdet_pico_224_224_hand.espdl"
SCORE_THRESHOLD = 0.5
NMS_THRESHOLD = 0.7
BOX_COLOR = (255, 0, 0)

• SCORE_THRESHOLD discards detections with a confidence score below 0.5.
• NMS_THRESHOLD controls when overlapping candidate boxes are considered duplicates.
• BOX_COLOR draws red boxes and labels in RGB order.

These values are starting points. Lowering the score threshold can recover difficult detections but may increase false positives. See Lecture 1 for additional details.

Step 3: Initialize the camera, LCD, and model

#

Initialize the LCD and configure the camera for 320 × 240 RGB frames:

lcd = display.Display()

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.skip_frames(time=1000)

Create the detector once, before the frame loop:

detector = espdl.ESPDet(
    MODEL,
    score=SCORE_THRESHOLD,
    nms=NMS_THRESHOLD,
)

Constructing the model outside the loop avoids repeatedly loading it and allocating inference resources.

Step 4: Detect and display hands

#

Capture a frame and pass it to the detector:

image = sensor.snapshot()
detections = detector.detect(image)

Draw each returned bounding box and confidence score:

for x, y, width, height, score, _category in detections:
    image.draw_rectangle(
        x,
        y,
        width,
        height,
        color=BOX_COLOR,
        thickness=2,
    )
    image.draw_string(
        x,
        max(0, y - 12),
        "hand %.2f" % score,
        color=BOX_COLOR,
    )

The label is placed above the box when space is available. max(0, y - 12) prevents the text coordinate from moving above the image boundary.

Write the annotated frame to the LCD and briefly yield before capturing the next frame:

lcd.write(image)
time.sleep_ms(20)

Step 5: Run and evaluate

#

1. Connect the ESP-Vision Web IDE to the board.
2. Create hand_detection.py.
3. Paste the complete code and save it to the device.
4. Select Run.
5. Observe the LCD and REPL output.

Test these conditions:

If detections are unreliable:

• Improve front lighting and reduce motion blur.
• Keep the full hand visible and within the camera frame.
• Move the hand closer if it occupies very few pixels.
• Lower SCORE_THRESHOLD gradually, for example from 0.5 to 0.4, while checking for false positives.
• Confirm that the model file is the ESP32-S31 hand-detection model.

Complete working code

#

# SPDX-FileCopyrightText: 2026 Espressif Systems (Shanghai) CO LTD
# SPDX-License-Identifier: Apache-2.0

import display
import espdl
import sensor
import time

MODEL = "/espdet_pico_224_224_hand.espdl"
SCORE_THRESHOLD = 0.5
NMS_THRESHOLD = 0.7
BOX_COLOR = (255, 0, 0)

lcd = display.Display()

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.skip_frames(time=1000)

detector = espdl.ESPDet(
    MODEL,
    score=SCORE_THRESHOLD,
    nms=NMS_THRESHOLD,
)

try:
    while True:
        image = sensor.snapshot()

        detections = detector.detect(image)

        for x, y, width, height, score, _category in detections:
            image.draw_rectangle(
                x,
                y,
                width,
                height,
                color=BOX_COLOR,
                thickness=2,
            )
            image.draw_string(
                x,
                max(0, y - 12),
                "hand %.2f" % score,
                color=BOX_COLOR,
            )

        lcd.write(image)
        time.sleep_ms(20)
finally:
    detector.deinit()
    sensor.shutdown()
    lcd.deinit()

What you will see

#

• A live 320 × 240 camera image on the ESP32-S31-Korvo-1 LCD.
• A red rectangle around each detected hand.
• A hand label and confidence score above each box.
• Boxes that update independently on every frame.

What you learned

#

In this assignment, you:

• Loaded an ESP32-S31-compatible ESPDet Pico hand model from internal flash
• Configured score and NMS thresholds
• Interpreted detector boxes, scores, and categories
• Drew detection results on live camera frames
• Displayed annotated frames on the board LCD
• Released model, camera, and display resources when the script stops

Next step

#

Assignment 4: Add a safety zone

Return to the workshop main page