Introduction to the ESP32-S31 with Vision: Lecture 1

Choose the right model for the task

#

ESP-Vision uses the ESP-DL .espdl format for optimized neural networks. A model file contains a quantized graph, parameters, and metadata that ESP-DL can load on the device.

Models do not all solve the same problem:

• Classification answers “what is in this image?”
• Object detection answers “what is present, and where is it?”
• Pose estimation returns semantic keypoints such as shoulders, elbows, and knees
• Segmentation labels pixels or object masks

The final workshop application needs the location of each hand, so it uses object detection.

Object detection models

#

ESP-Vision supports dedicated wrappers for two detector families:

Common ESPDet Pico models in the ESP-Vision model collection include:

YOLO11n COCO detects 80 everyday object categories, including person, bottle, chair, laptop, and cell phone. The linked COCO dataset definition lists every category. It is useful when the application needs broad coverage, but using it only to locate hands would waste compute and does not provide a hand class.

Classification, pose, and other models

#

ESP-Vision also provides examples for:

• Image classification: the current public model catalog includes a TensorFlow Lite Micro person-presence classifier
• YOLO11n Pose: detects people and returns 17 COCO body keypoints
• TensorFlow Lite Micro: runs compatible .tflite models through the tflite module
• Generic ESP-DL graphs: exposes raw tensor inputs and outputs when a model-specific Python decoder is needed

The presence of an API does not guarantee that every model fits every board. Check the model’s target, memory use, input size, and operator support before choosing it.

ESP-Vision model APIs

#

The task-specific wrappers perform preprocessing and decode model output. For example:

detector = espdl.ESPDet(
    "/espdet_pico_224_224_hand.espdl",
    score=0.5,
    nms=0.7,
)

results = detector.detect(frame)

Each ESPDet result is:

(x, y, width, height, score, category)

Score and NMS thresholds

#

The score and nms arguments control which candidate detections are returned:

The confidence score indicates how strongly a detection matches a learned category. It is useful for ranking and filtering results, but it should not be interpreted as a precisely calibrated probability. With score=0.5, detections below 0.5 are discarded.

A detector often proposes several boxes around the same object. Non-maximum suppression (NMS) keeps the highest-scoring box and compares it with competing boxes. IoU measures their overlap:

IoU = intersection area / union area

With nms=0.7, a lower-scoring box is suppressed when its IoU with the selected box exceeds 0.7. Lowering the NMS threshold removes more overlapping boxes, while raising it can preserve nearby objects but may leave duplicate detections.

The values score=0.5 and nms=0.7 are starting points, not universal settings. Test them with the actual camera position, lighting, backgrounds, hand sizes, and number of simultaneous hands:

• Lower score if real hands are frequently missed.
• Raise score if background objects are frequently reported as hands.
• Lower nms if one hand receives several overlapping boxes.
• Raise nms if boxes for two nearby hands suppress each other.

espdl.Model is lower-level. Its predict() method returns named raw tensors, so the application must implement model-specific box decoding and non-maximum suppression.

Choosing a model

#

Consider these factors:

For the safety-zone application:

• The task is hand localization, not gesture classification.
• One class is sufficient.
• A 224 × 224 ESPDet Pico model has a better size/speed trade-off than a general detector.
• A confidence threshold of 0.5 is a reasonable starting point, but it must be evaluated in the real scene.

Model storage and loading

#

ESP-Vision loads the .espdl model at runtime, so the model file must be uploaded to storage that the device can access before running the application. You can store it in either location:

This workshop uses internal flash. Download espdet_pico_224_224_hand.espdl from the model collection on the ESP-Vision website.

The remaining assignments therefore expect:

/espdet_pico_224_224_hand.espdl

If you use a microSD card instead, copy the model to the card and update MODEL_PATH to the /sdcard/... path shown above.

Create the detector once, outside the frame loop:

detector = espdl.ESPDet(MODEL_PATH, score=0.5, nms=0.7)

Release it when the application stops:

try:
    # Capture and inference loop
    pass
finally:
    detector.deinit()

Repeatedly constructing a detector inside the loop increases latency and can fragment memory.

What you learned

#

In this lecture, you learned:

• The differences between classification, detection, pose, and segmentation
• When to choose ESPDet Pico instead of a general YOLO model
• Which ESP-Vision classes load common ESP-DL models
• How model input, latency, memory, storage, and target affect selection
• Why the hand-detection model is appropriate for the final application

Next step

#

Assignment 3: Hand detection with ESPDet

Return to the workshop main page