Edge-AI with ESP32-S3 Workshop: Introduction

Introduction to Edge-AI Vision

#

In many real-world applications, reacting to what a camera sees is a fundamental capability: a doorbell that recognizes a face, a robot that avoids obstacles, a factory line that spots defects, or a security system that detects motion. Traditionally, achieving this required sending raw image data to a cloud server for processing, adding latency, bandwidth costs, privacy exposure, and a hard dependency on network connectivity.

Edge AI moves the inference workload directly onto the device, processing sensor data locally without relying on the cloud. This approach unlocks a different class of applications:

• Low latency: decisions happen in milliseconds, not round-trip seconds
• Privacy by design: raw images never leave the device
• Offline operation: works without internet connectivity
• Lower cost: no cloud compute or data transfer fees at scale

The challenge: constrained devices

#

Running a neural network model on a microcontroller is fundamentally different from running one on a server or even a smartphone. Embedded devices have strict constraints:

This means that models must be carefully quantized, pruned, and optimized to fit within the available memory and run at a useful frame rate. Model weights are typically quantized from 32-bit floats to 8-bit integers (INT8), which reduces memory usage by 4x and allows faster arithmetic with minimal accuracy loss.

Espressif SoCs for edge AI

#

Espressif offers several SoCs purpose-built for edge AI workloads. The table below compares the three most relevant ones for vision applications:

ESP32-S3

#

The ESP32-S3 is the SoC used in this workshop, via the ESP32-S3-EYE development board. It was designed with AI workloads in mind, adding 128-bit SIMD vector instructions to the Xtensa LX7 core (see the ESP32-S3 TRM > Chapter Processor Instruction Extensions (PIE)). These instructions accelerate operations common in neural network inference, such as multiply-accumulate (MAC), dot products, and convolution. They are the hardware foundation that makes frameworks like ESP-DL practical on a microcontroller.

Notable advantages for edge-AI vision:

• Dual-core Xtensa LX7 at 240 MHz: one core handles inference while the other manages peripherals and communication
• 128-bit SIMD vector instructions: hardware-accelerated INT8 neural network operations with ~18x speedup over unoptimized code
• 8 MB Octal PSRAM: enough to buffer camera frames and model activations simultaneously
• DVP camera interface: direct connection to image sensors like the OV2640
• Wi-Fi + Bluetooth: results can be reported wirelessly without a separate communication chip

ESP32-P4

#

The ESP32-P4 is Espressif’s current high-performance SoC for edge AI and rich HMI. Its dual-core RISC-V at 400 MHz, combined with PIE (Processor Instruction Extensions) that fuse multiply-accumulate-shift into a single cycle on INT8 vectors, delivers roughly 2.7-3x faster inference than the ESP32-S3 for typical vision models.

Notable advantages for edge-AI vision:

• MIPI-CSI camera input: supports higher-resolution and higher-framerate image sensors
• Up to 32 MB PSRAM: headroom for larger models and higher-resolution frame buffers
• Hardware JPEG codec and 2D PPA: offloads image pre/post-processing from the CPU
• 768 KB on-chip SRAM: reduces reliance on external memory for small intermediate buffers

ESP32-S31

#

The ESP32-S31 is Espressif’s newest connectivity-focused SoC, announced in 2026. It targets applications that combine edge AI with comprehensive wireless protocols, including Wi-Fi 6, Bluetooth 5.4 (LE + Classic), Thread, and Zigbee, all in a single SoC.

Notable advantages for edge-AI vision:

• 128-bit SIMD on one RISC-V core: accelerates INT8 inference with wide parallel data paths
• Wi-Fi 6 + BT 5.4 + Thread + Zigbee: ideal for smart home and industrial automation requiring multi-protocol connectivity
• Hardware JPEG codec and 2D PPA: same image processing acceleration as the ESP32-P4
• Supported from ESP-IDF v6.1+

Introduction to ESP-WHO

#

To help developers with vision applications such as face detection and recognition, pedestrian detection, and QR code recognition, Espressif has developed a framework for image processing that runs on ESP SoCs. ESP-WHO is built on top of ESP-DL, Espressif’s neural network inference engine, which handles all model loading and execution. ESP-WHO adds the camera pipeline, display integration, and FreeRTOS task scaffolding on top of it.

An introduction article called ESP-WHO: Get started was published recently and it is an excellent source for information about ESP-WHO.

In this workshop, we will go further than the article and deep-dive into vision for other applications.

Supported hardware

#

ESP-WHO targets ESP SoCs with hardware AI acceleration and uses the BSP abstraction layer so that the same application code runs across supported boards without modification. The table below lists the supported development boards from Espressif:

Features

#

ESP-WHO ships with ready-to-run examples that cover the most common vision AI use cases:

Beyond the individual examples, ESP-WHO provides several framework-level capabilities:

• ESP-DL powered inference: all model execution in ESP-WHO goes through ESP-DL. ESP-DL handles model loading, memory placement, and hardware-accelerated INT8 inference — ESP-WHO builds its detection and recognition pipelines directly on top of it. The same models can also be used directly via the ESP-DL C++ API without going through ESP-WHO
• Asynchronous pipeline: the camera capture and model inference run on separate cores concurrently, maximizing frame throughput
• LVGL integration: results are rendered directly on the LCD display using the LVGL graphics library, with no extra glue code needed
• BSP-based portability: hardware differences between supported boards are fully abstracted. The same application code runs on the ESP32-S3-EYE, ESP32-P4-Function-EV-Board, and ESP32-S3-Korvo-2 by switching the BSP configuration
• Pre-quantized model zoo: face detection, face recognition, gesture recognition, and object detection models are provided by ESP-DL, pre-quantized and ready to load — either through ESP-WHO pipelines or directly via the ESP-DL API

espressif/esp-who

Face detection and recognition framework

C++

2136

546

Architecture

#

ESP-WHO follows a layered architecture. Your application sits at the top, composed from ESP-WHO building-block components. These components depend on ESP-DL for inference and ESP-BSP for hardware abstraction. The BSP brings in the individual peripheral drivers, all sitting on top of ESP-IDF.

graph TD
    A([Your Application])
    A --> B[ESP-WHO]
    B --> C[ESP-DL]
    B --> D[ESP-BSP]
    C --> E[ESP-IDF]
    D --> E
    D --> F[esp_video]
    D --> G[esp_lvgl_port]
    F --> E
    G --> E

Each node in the diagram represents a distinct layer of the stack:

ESP-WHO internal components:

ESP-WHO is structured as a set of composable C++ components, each responsible for a single stage of the vision pipeline. They are designed to run as FreeRTOS tasks and communicate through queues, so individual stages can be combined or replaced without rewriting the whole application.

External dependencies:

ESP-WHO relies on a set of Espressif-maintained components that are declared in the project’s idf_component.yml and fetched automatically from the ESP Component Registry at build time.

ESP-DL

#

ESP-DL is Espressif’s lightweight neural network inference framework designed specifically for ESP SoCs. It provides the low-level engine that loads, optimizes, and runs AI models on the device, and is the foundation that ESP-WHO builds on for vision tasks.

Key capabilities:

• .espdl model format: a FlatBuffers-based format (similar to ONNX) optimized for embedded targets, with support for zero-copy deserialization to reduce startup time and RAM usage
• Optimized operators: common AI operators (Conv, DepthwiseConv, Gemm, Add, Mul, etc.) are implemented using SoC-specific SIMD/PIE instructions for maximum throughput
• Static memory planner: automatically places model layers into the optimal memory region (internal SRAM vs PSRAM) based on user-specified constraints
• Dual-core scheduling: computationally heavy operators (Conv2D, DepthwiseConv2D) are automatically split across both cores
• 8-bit LUT activations: all activation functions except ReLU/PReLU are computed via an 8-bit look-up table, keeping inference latency flat regardless of activation complexity

The typical ESP-DL workflow is:

1. Train a model in PyTorch or TensorFlow
2. Export to ONNX
3. Quantize from FP32 to INT8 using ESP-PPQ (pip install esp-ppq), producing a .espdl file
4. Load and run the .espdl model on-device using the ESP-DL C++ API

ESP-DL ships with a model zoo of pre-trained and pre-quantized models ready to deploy, including face detection, face recognition, hand gesture recognition, and YOLO11-based object detection. All of these are used in this workshop.

espressif/esp-dl

Espressif deep-learning library for AIoT applications

Assembly

1126

222

ESP-BSP

#

ESP-BSP is Espressif’s Board Support Package framework. It provides a unified hardware abstraction API that covers peripherals such as display, camera, microphone, buttons, SD card, and LEDs. With the BSP, you can write portable application code without managing low-level pin assignments and driver initialization manually.

For this workshop, the BSP component for the ESP32-S3-EYE is espressif/esp32_s3_eye.

Capabilities

The table below lists the hardware capabilities exposed by the ESP32-S3-EYE BSP and the underlying components used:

Basic usage

The BSP handles peripheral initialization through a clean API. Below are some common patterns:

Display (LVGL):

#include "bsp/esp32_s3_eye.h"

bsp_display_start();
bsp_display_backlight_on();

Camera:

bsp_camera_init(&camera_config);

Buttons:

bsp_iot_button_create(buttons, NULL, BSP_BUTTON_NUM);

SD card:

bsp_sdcard_mount();
// ... use SD card ...
bsp_sdcard_unmount();

Compatible examples

ESP Video Components

#

esp-video-components is Espressif’s collection of video-related components for ESP-IDF, covering camera capture, image signal processing, and video encoding. The most relevant component for this workshop is esp_video, which provides the camera driver used by the ESP32-S3-EYE BSP. Each component in the repository can be used independently — the BSP pulls in esp_video as its camera dependency, not the repository as a whole.

The key feature of esp-video-components is its Linux V4L2-compatible API, using the same open(), ioctl(), and buffer queue model used in Linux camera stacks. This makes it consistent across all supported interfaces (DVP, MIPI-CSI, SPI, USB) and all supported SoCs.

Key capabilities

#

• V4L2-compatible API: unified interface regardless of sensor or physical bus
• ISP pipeline: built-in image signal processing support (ESP32-P4)
• H.264 hardware encoding: high-speed video encoding (ESP32-P4)
• Multi-camera support: manage multiple sensors simultaneously
• Broad SoC support: ESP32-S3 (DVP), ESP32-P4 (DVP + MIPI-CSI), ESP32-S31 (DVP + MIPI-CSI), ESP32-C series (SPI)

For the ESP32-S3-EYE, esp_video operates over the DVP interface with the OV2640 sensor. Since the BSP handles initialization, you interact with the camera through BSP calls rather than the esp_video API directly in most ESP-WHO examples.

espressif/esp-video-components

Collections of components to be used for camera related functionalities.

C

87

39

AI capabilities of ESP32-S3-EYE

#

The ESP32-S3-EYE is a small-sized AI development board produced by Espressif. It is based on the ESP32-S3 SoC, featuring a 2-megapixel camera, a 1.3" LCD display, and a digital microphone for image recognition and audio processing.

Key features

#

Block diagram

The block diagram below shows the main components of the ESP32-S3-EYE-MB main board v2.2 (left) and the ESP32-S3-EYE-SUB sub board (right).

Main board components (ESP32-S3-EYE-MB)

Sub board components (ESP32-S3-EYE-SUB)

Resources

#

• Getting Started Guide
• ESP32-S3 Datasheet (PDF)
• ESP32-S3-WROOM-1 Datasheet (PDF)
• Main Board Schematic v2.2 (PDF)
• Sub Board Schematic (PDF)

Next step

#

After this introduction, it is time to get started and install the development environment.

Assignment 1: Install ESP-IDF and ESP-WHO

Return to the workshop main page