Edge-AI with ESP32-S3 Workshop: Assignment 7

Assignment 7: Going further

#

In this final assignment there are no build or flash steps. Instead you will explore two resources that extend what you have learned throughout the workshop: the full ESP-DL Model Zoo and how custom models are trained and quantized for ESP devices, and ESP-Vision, a Python-based platform that lets you prototype vision AI applications with just a few lines of code.

The ESP-DL Model Zoo

#

All models that Espressif provides for ESP devices are published as open-source components in the ESP-DL repository and on the ESP Component Registry. You have already used several of them during this workshop. The complete set is listed below.

Every model in the table above is an ESP-IDF component. Adding one to your project takes a single line in idf_component.yml and a few lines of C++ — the same pattern you used throughout this workshop.

Training and deploying a custom model

#

The models in the zoo cover many common tasks, but real products often need a model trained on a specific domain — your own objects, environments, or gesture vocabulary. ESP-DL supports deploying custom models through a quantization pipeline based on ESP-PPQ.

ESP-PPQ is Espressif’s quantization toolkit, built as an extension of the open-source PPQ framework. It takes a full-precision ONNX model and converts it into the 8-bit .espdl format that ESP-DL can load and execute on chip. ESP-PPQ handles graph optimization, operator fusion, calibration, and weight quantization — all in a single Python API call. It can be installed with:

pip install esp-ppq

The workflow

#

graph LR
    A[Train model\nPyTorch / TF / Paddle] --> B[Export\nto ONNX]
    B --> C[Quantize with ESP-PPQ\nespdl_quantize_onnx]
    C --> D[.espdl\nmodel file]
    D --> E[Deploy with\nESP-DL\non chip]

1. Train your model

Train a standard neural network in any framework. ESP-PPQ has native support for PyTorch and ONNX. Models from TensorFlow, PaddlePaddle, and other frameworks must be converted to ONNX first using tools such as tf2onnx or paddle2onnx.

2. Export to ONNX

# PyTorch example
import torch
torch.onnx.export(model, dummy_input, "model.onnx",
                  opset_version=11, input_names=["input"])

3. Quantize with ESP-PPQ

The default method is Post Training Quantization (PTQ), which requires no retraining — only a small unlabeled calibration dataset (32–100 images) representative of the real input distribution.

from espdl import espdl_quantize_onnx

quant_graph = espdl_quantize_onnx(
    onnx_import_file="model.onnx",
    espdl_export_file="model_s3.espdl",
    calib_dataloader=calib_loader,   # DataLoader wrapping your calibration images
    calib_steps=32,                  # number of calibration batches
    input_shape=[1, 3, 224, 224],    # must match the shape used during ONNX export
    target="esp32s3",                # esp32 | esp32s3 | esp32p4 | esp32s31
    num_of_bits=8,                   # INT8 quantization
    export_test_values=True,         # embed test vectors for on-chip accuracy verification
)

Key parameters to understand:

Three files are produced after quantization:

4. Load and run on device

Once you have the .espdl file, deploy it with ESP-DL using the generic dl::Model class:

#include "dl_model_base.hpp"
#include "dl_image_define.hpp"

dl::Model *model = new dl::Model("path/to/model.espdl",
                                  fbs::MODEL_LOCATION_IN_FLASH_PARTITION);
// Build input tensor, run forward pass
model->run(inputs, outputs);

PTQ vs QAT

#

PTQ is the default starting point. If the quantized model shows significant accuracy degradation compared to the full-precision version, switch to QAT by incorporating the quantization error into the training loss function.

Quantization differences per SoC

#

The quantization strategy varies by target. Set the target parameter in ESP-PPQ accordingly:

For detailed tutorials and example scripts, see the ESP-DL quantization documentation and the ESP-PPQ repository.

ESP-Vision

#

ESP-Vision is a Python-based platform that runs on top of ESP32 hardware and lets you build real-time vision AI applications with just a few lines of code — no C++, no toolchain, and no build system required. It is the fastest way to prototype an idea or demonstrate a concept on actual hardware.

How it works

#

ESP-Vision provides a MicroPython-compatible runtime with a set of high-level modules:

A complete object detection application that runs a YOLO11n detection loop on the ESP32-P4 looks like this:

import espdl
import sensor
import time

sensor.reset()
sensor.set_pixformat(sensor.RGB565)
sensor.set_framesize(sensor.QVGA)
sensor.skip_frames(time=1000)

det = espdl.ESPDet("/sdcard/hand_det.espdl", score=0.5, nms=0.7)
while True:
    img = sensor.snapshot()
    for x, y, w, h, score, category in det.detect(img):
        img.draw_rectangle(x, y, w, h, color=(255, 0, 0), thickness=2)
        img.draw_string(x, max(0, y - 12), "%.2f:%d" % (score, category))
    img.flush()

Supported boards

#

ESP-Vision runs on the following boards:

The ESP32-S3-EYE you have been using throughout this workshop is fully supported.

Getting started

#

1. Go to vision.espressif.com.
2. Click Flash It Now to install the ESP-Vision firmware on your board directly from the browser using Web Serial — no installation needed.
3. Open the Web IDE to write and run Python scripts in your browser.
4. Browse the model zoo on the site to download ready-to-use .espdl model files, copy them to an SD card, and load them with espdl.ESPDet() or espdl.YOLO11().

Model zoo

#

ESP-Vision ships with a curated set of ready-to-use models:

All models use the same .espdl format you have been working with in this workshop, so any custom model you export with ESP-PPQ can be loaded by ESP-Vision without modification.

AI-assisted development with MCP

#

ESP-Vision also exposes an MCP server that connects to AI coding assistants. You can add it to Cursor, VS Code, Claude Code, or any MCP-compatible client to get context-aware assistance when writing ESP-Vision scripts:

{
  "mcpServers": {
    "esp-vision-mcp": {
      "url": "https://mcp.vision.espressif.com"
    }
  }
}

Thank you

#

You have reached the end of the workshop. Over the course of these assignments you have:

• Set up the development environment for ESP-WHO and ESP-DL
• Explored the OV2640 camera sensor and the ESP-Video V4L2 pipeline
• Run face detection and face recognition using ESP-WHO
• Used ESP-DL directly to run hand gesture classification and YOLO11 object detection on static images
• Learned how to quantize and deploy a custom model with ESP-PPQ

We hope this gives you a solid foundation for building your own Edge AI applications with Espressif hardware. If you have questions or feedback, feel free to open a discussion on the developer portal repository.

espressif/esp-dl

Espressif deep-learning library for AIoT applications

Assembly

1126

222

espressif/esp-who

Face detection and recognition framework

C++

2136

546

Next step

#

Return to the workshop main page