↓ Skip to main content

ESP-TFLite-Micro workshop - Assignment 1: Collect and prepare gesture data

·9 mins·
Table of Contents
WS004EN - This article is part of a series.
Part 6: This Article

Assignment steps
#

In this assignment, you will define a gesture, record repeated BMI270 gyroscope samples, and save them as a CSV dataset. Each captured sample contains 200 readings from the x, y, and z axes, for a total of 600 values.

Get the training scripts
#

The workshop gesture recognition project already includes the training workspace. To refresh it with the latest capture, validation, training, conversion, and embedding scripts, download the training script bundle and extract it from the gesture_recognition project directory:

1curl -fL \
2https://dl.espressif.com/AE/esp-iot-solution/gesture_recognition/training.zip \
3-o training.zip
4unzip -o training.zip
5rm training.zip
1Invoke-WebRequest `
2-Uri "https://dl.espressif.com/AE/esp-iot-solution/gesture_recognition/training.zip" `
3-OutFile "training.zip"
4Expand-Archive -Path "training.zip" -DestinationPath "." -Force
5Remove-Item "training.zip"

Graphical alternative: Download the ZIP archive in a browser and extract its training folder into the gesture_recognition project directory. Confirm replacement if your archive manager asks whether to overwrite scripts already supplied by the code repository.

Step 1: Create the dataset folder
#

From the gesture_recognition project directory, create the folder that will contain the original datasets and your newly collected gesture:

  1. Open the gesture_recognition project in VS Code.
  2. In the Explorer, create a folder named training.
  3. Create shuttle_dataset inside training.
mkdir -p training/shuttle_dataset
mkdir -p training/shuttle_dataset

PowerShell:

New-Item -ItemType Directory -Force training\shuttle_dataset

The resulting path is:

gesture_recognition/training/shuttle_dataset/

You will save the CSV collected in this assignment in this folder. Assignment 2 will add the three original gesture datasets alongside it.

Step 2: Enter data collection mode
#

Keep the serial monitor open after flashing the gesture recognition example. The application starts in automatic inference mode.

Try a V and a counterclockwise circle before changing mode. Each detected movement should print a class name, such as V or Counterclockwise circle, and a confidence value. Reference output:

I (...) Model Pipeline: Motion detected: V (confidence: 99.7%)
I (...) Model Pipeline: Motion detected: Counterclockwise circle (confidence: 87.6%)

Timestamps and confidence values vary between runs. Other movements are often classified as Unknown.

Press and hold the Boot button for at least two seconds, then release it. The monitor should confirm the mode change:

I (...) APP_BTN: Mode switched to COLLECT

Long-press and release Boot again whenever you want to return to automatic inference mode.

Step 3: Define a new gesture
#

Choose a movement that you can repeat consistently. Give it a short, descriptive name, such as clockwise_circle, horizontal_line, or letter_s.

Keep the new gesture distinct from the counterclockwise circle (o) and the V (v) already in the example. Avoid a similar shape, and avoid a path that contains either of them. With the same grip, an M includes the motion of a V, so those samples also contain the data that identifies V.

Before collecting data, decide:

  • How you will hold the board
  • The starting and ending positions
  • The direction and size of the movement
  • Which hand you will use
  • How quickly you will perform it

Keep the board orientation consistent. The BMI270 measures motion along axes fixed to the board. Holding the board from another side, rotating it, or turning it over changes which axes receive the motion and may reverse their signs. The same hand gesture can therefore produce substantially different data.

Choose one grip and orientation for the full dataset, and use the same setup when testing the trained model. If the application must work with several orientations, collect and label enough examples in every supported orientation.

Try to complete the gesture within approximately one second. The application records 200 samples at 200 Hz, so each row represents one second of motion.

Step 4: Capture one sample
#

  1. Briefly press and release the Boot button.
  2. Immediately perform the gesture.
  3. Wait for the movement to exceed the configured gyroscope threshold.
  4. Keep the board still after finishing until the values appear in the serial monitor.

When the application detects enough motion, it prints:

1I (...) Model Pipeline: Trigger met (...), collecting 200 samples (include trigger sample)
2
3=== Data Collection Results ===
412.34567,-8.90123,4.56789,...,1.23456,-2.34567,3.45678
5=== End of Data Collection ===

The line between the two markers contains exactly 600 comma-separated floating-point values, ordered as:

x0,y0,z0,x1,y1,z1,...,x199,y199,z199

If the trigger is not reached, the monitor prints Trigger not met. Press Boot and try the movement again with a clearer or faster motion.

Step 5: Repeat the gesture
#

Capture the same gesture at least 25 times. Twenty-five samples are not enough for a production-quality model, but they are sufficient to complete this workshop and observe the full training workflow.

Use a similar starting position and overall shape each time, but allow small natural variations in speed, angle, and amplitude. If the gesture is complex or the repetitions vary significantly, collect more samples.

Discard a capture and repeat it when:

  • You started too late or stopped before completing the gesture
  • The board hit an object or slipped in your hand
  • The output contains an incomplete row
  • You accidentally performed a different movement

Copying 25 lines of 600 numbers from the serial monitor is easy to get wrong. You can optionally let a script copy each valid line into training/shuttle_dataset/new_gesture.csv. You still perform the gesture 25 times. Step 6 explains how to paste the lines yourself if you do not use the script.

The code repository and training bundle provide training/capture_gesture.py. Its implementation is shown below for reference:

  1import argparse
  2import re
  3import sys
  4from pathlib import Path
  5
  6import serial
  7from serial.tools import list_ports
  8
  9START = "=== Data Collection Results ==="
 10END = "=== End of Data Collection ==="
 11ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
 12ESPRESSIF_VID = 0x303A
 13
 14
 15def board_ports():
 16    skipped = ("bluetooth", "debug-console")
 17    candidates = []
 18    for info in list_ports.comports():
 19        device = info.device or ""
 20        label = f"{device} {info.description or ''}".lower()
 21        if any(word in label for word in skipped):
 22            continue
 23        candidates.append(info)
 24
 25    espressif = [info.device for info in candidates if info.vid == ESPRESSIF_VID]
 26    if espressif:
 27        return espressif
 28
 29    usb_marks = ("usbmodem", "usbserial", "ttyACM", "ttyUSB", "COM", "wchusb", "SLAB")
 30    return [info.device for info in candidates if any(mark in info.device for mark in usb_marks)]
 31
 32
 33def open_board():
 34    names = board_ports()
 35    if not names:
 36        print("No board serial port found.", file=sys.stderr)
 37        return None
 38    for name in names:
 39        try:
 40            port = serial.Serial(name, 115200, timeout=0.2)
 41        except serial.SerialException as exc:
 42            print(f"Skipping {name}: {exc}")
 43            continue
 44        print(f"Using {name}")
 45        return port
 46    print("Could not open a board serial port.", file=sys.stderr)
 47    return None
 48
 49
 50def main():
 51    parser = argparse.ArgumentParser()
 52    parser.add_argument("-n", "--count", type=int, default=25)
 53    parser.add_argument(
 54        "-o",
 55        "--output",
 56        type=Path,
 57        default=Path("shuttle_dataset/new_gesture.csv"),
 58    )
 59    args = parser.parse_args()
 60
 61    args.output.parent.mkdir(parents=True, exist_ok=True)
 62    rows = []
 63    capturing = False
 64    pending = ""
 65
 66    print("Long-press Boot for about 2 seconds, then release.")
 67    print("Short-press Boot and perform the gesture once for each sample.")
 68    print(f"Need {args.count} rows. Press Ctrl+C to stop early.")
 69
 70    port = open_board()
 71    if port is None:
 72        return 1
 73
 74    try:
 75        while len(rows) < args.count:
 76            chunk = port.read(4096)
 77            if not chunk:
 78                continue
 79            pending += ANSI.sub("", chunk.decode("utf-8", "replace"))
 80            while "\n" in pending:
 81                line, pending = pending.split("\n", 1)
 82                line = line.strip()
 83                if "Mode switched to" in line:
 84                    print(line)
 85                if START in line:
 86                    capturing = True
 87                    continue
 88                if END in line:
 89                    capturing = False
 90                    continue
 91                if not capturing or "," not in line:
 92                    continue
 93                values = [item for item in line.split(",") if item]
 94                if len(values) != 600:
 95                    print(f"Skipped a row with {len(values)} numbers.")
 96                    continue
 97                rows.append(",".join(values))
 98                print(f"Saved {len(rows)}/{args.count}")
 99    except KeyboardInterrupt:
100        print("\nStopped.")
101    finally:
102        port.close()
103
104    text = "\n".join(rows)
105    if text:
106        text += "\n"
107    args.output.write_text(text)
108    print(f"Wrote {len(rows)} rows to {args.output}")
109    return 0 if len(rows) >= args.count else 2
110
111
112if __name__ == "__main__":
113    raise SystemExit(main())

Exit the serial monitor with Ctrl+] before you start the script. Only one program can use the board’s serial port. Install pyserial, then run the script from the training directory. It scans the serial ports on the computer and prints the one it uses. You do not pass a port name.

  1. Install the Microsoft Python extension if it is not already installed.
  2. Open training/capture_gesture.py.
  3. Open a new terminal with Terminal → New Terminal and run:
cd training
python -m pip install pyserial
python capture_gesture.py
cd training
python3 -m pip install pyserial
python3 capture_gesture.py
cd training
python3 -m pip install pyserial
python3 capture_gesture.py

PowerShell, from the gesture_recognition directory:

Set-Location training
py -3 -m pip install pyserial
py -3 capture_gesture.py

While the script is running:

  1. Press and hold Boot for about two seconds, then release it. The terminal prints Mode switched to COLLECT when the board changes mode.
  2. Briefly press Boot and perform the gesture immediately.
  3. Wait until the terminal prints Saved 1/25. Repeat until it prints Saved 25/25.

The script writes only lines that contain 600 values. It skips log lines, marker lines, and incomplete rows. Press Ctrl+C to stop early and keep the rows already saved. When it finishes, the CSV file already exists, so skip the manual copy in Step 6 and continue with Step 7.

Step 6: Create the CSV file
#

Inside training/shuttle_dataset, create a file named new_gesture.csv. This is the filename used by the training script in Assignment 2:

gesture_recognition/training/shuttle_dataset/new_gesture.csv

Skip this step if capture_gesture.py already created the file.

For each valid capture:

  1. Copy only the line between === Data Collection Results === and === End of Data Collection ===.
  2. Paste it into the CSV file as a new row.
  3. Do not copy timestamps, log messages, marker lines, or a column header.

After 25 captures, the file should contain at least 25 rows. Every row represents one gesture and must contain exactly 600 values.

Step 7 (optional): Validate and save the dataset
#

This validation is optional but recommended. The code repository and training bundle provide training/shuttle_dataset/validate_dataset.py; its implementation is shown below for reference:

 1import pandas as pd
 2
 3data = pd.read_csv("new_gesture.csv", header=None)
 4
 5print("Samples:", data.shape[0])
 6print("Values per sample:", data.shape[1])
 7
 8assert data.shape[0] >= 25
 9assert data.shape[1] == 600
10assert not data.isnull().any().any()

Run the script:

  1. Install the Microsoft Python extension.
  2. Open validate_dataset.py.
  3. Select a Python interpreter that has pandas installed.
  4. Click Run Python File (the triangular play button) in the editor toolbar.

The output appears in the VS Code Terminal panel.

python validate_dataset.py

Spreadsheet alternative: Open the CSV in LibreOffice Calc, Microsoft Excel, or another spreadsheet application. Confirm that it contains at least 25 rows, exactly 600 populated columns, and no empty cells. Do not save it in a format that adds a header or changes the comma-separated numeric values.

Expected result:

Samples: 25
Values per sample: 600

Save the validated CSV file. Repeat this collection process for every gesture class used by your model. A three-class classifier also needs an unknown class containing varied movements that do not match either defined gesture. Keep the class sizes reasonably balanced.

Acceptance criteria
#

  • The serial monitor confirms that the application is in COLLECT mode.
  • The new gesture has a clear name and can be repeated consistently.
  • training/shuttle_dataset/new_gesture.csv contains at least 25 valid samples.
  • Each CSV row contains exactly 600 floating-point values with no header.
  • The dataset contains no empty or incomplete values.
  • Separate, reasonably balanced CSV files exist for every model class.

Next step
#

Assignment 2: Train and evaluate the model

Back to workshop home

WS004EN - This article is part of a series.
Part 6: This Article

Related