Assignment steps#
In this assignment, you will define a gesture, record repeated BMI270 gyroscope samples, and save them as a CSV dataset. Each captured sample contains 200 readings from the x, y, and z axes, for a total of 600 values.
Get the training scripts#
The workshop gesture recognition project already includes the training workspace. To refresh it with the latest capture, validation, training, conversion, and embedding scripts, download the training script bundle and extract it from the gesture_recognition project directory:
1curl -fL \
2https://dl.espressif.com/AE/esp-iot-solution/gesture_recognition/training.zip \
3-o training.zip
4unzip -o training.zip
5rm training.zip1Invoke-WebRequest `
2-Uri "https://dl.espressif.com/AE/esp-iot-solution/gesture_recognition/training.zip" `
3-OutFile "training.zip"
4Expand-Archive -Path "training.zip" -DestinationPath "." -Force
5Remove-Item "training.zip"Graphical alternative: Download the ZIP archive in a browser and extract its training folder into the gesture_recognition project directory. Confirm replacement if your archive manager asks whether to overwrite scripts already supplied by the code repository.
Step 1: Create the dataset folder#
From the gesture_recognition project directory, create the folder that will contain the original datasets and your newly collected gesture:
- Open the
gesture_recognitionproject in VS Code. - In the Explorer, create a folder named
training. - Create
shuttle_datasetinsidetraining.
mkdir -p training/shuttle_datasetmkdir -p training/shuttle_datasetPowerShell:
New-Item -ItemType Directory -Force training\shuttle_datasetThe resulting path is:
gesture_recognition/training/shuttle_dataset/You will save the CSV collected in this assignment in this folder. Assignment 2 will add the three original gesture datasets alongside it.
Step 2: Enter data collection mode#
Keep the serial monitor open after flashing the gesture recognition example. The application starts in automatic inference mode.
Try a V and a counterclockwise circle before changing mode. Each detected movement should print a class name, such as V or Counterclockwise circle, and a confidence value. Reference output:
I (...) Model Pipeline: Motion detected: V (confidence: 99.7%)
I (...) Model Pipeline: Motion detected: Counterclockwise circle (confidence: 87.6%)Timestamps and confidence values vary between runs. Other movements are often classified as Unknown.
Press and hold the Boot button for at least two seconds, then release it. The monitor should confirm the mode change:
I (...) APP_BTN: Mode switched to COLLECTLong-press and release Boot again whenever you want to return to automatic inference mode.
Step 3: Define a new gesture#
Choose a movement that you can repeat consistently. Give it a short, descriptive name, such as clockwise_circle, horizontal_line, or letter_s.
Keep the new gesture distinct from the counterclockwise circle (o) and the V (v) already in the example. Avoid a similar shape, and avoid a path that contains either of them. With the same grip, an M includes the motion of a V, so those samples also contain the data that identifies V.
Before collecting data, decide:
- How you will hold the board
- The starting and ending positions
- The direction and size of the movement
- Which hand you will use
- How quickly you will perform it
Keep the board orientation consistent. The BMI270 measures motion along axes fixed to the board. Holding the board from another side, rotating it, or turning it over changes which axes receive the motion and may reverse their signs. The same hand gesture can therefore produce substantially different data.
Choose one grip and orientation for the full dataset, and use the same setup when testing the trained model. If the application must work with several orientations, collect and label enough examples in every supported orientation.
Try to complete the gesture within approximately one second. The application records 200 samples at 200 Hz, so each row represents one second of motion.
Step 4: Capture one sample#
- Briefly press and release the Boot button.
- Immediately perform the gesture.
- Wait for the movement to exceed the configured gyroscope threshold.
- Keep the board still after finishing until the values appear in the serial monitor.
When the application detects enough motion, it prints:
1I (...) Model Pipeline: Trigger met (...), collecting 200 samples (include trigger sample)
2
3=== Data Collection Results ===
412.34567,-8.90123,4.56789,...,1.23456,-2.34567,3.45678
5=== End of Data Collection ===The line between the two markers contains exactly 600 comma-separated floating-point values, ordered as:
x0,y0,z0,x1,y1,z1,...,x199,y199,z199If the trigger is not reached, the monitor prints Trigger not met. Press Boot and try the movement again with a clearer or faster motion.
Step 5: Repeat the gesture#
Capture the same gesture at least 25 times. Twenty-five samples are not enough for a production-quality model, but they are sufficient to complete this workshop and observe the full training workflow.
Use a similar starting position and overall shape each time, but allow small natural variations in speed, angle, and amplitude. If the gesture is complex or the repetitions vary significantly, collect more samples.
Discard a capture and repeat it when:
- You started too late or stopped before completing the gesture
- The board hit an object or slipped in your hand
- The output contains an incomplete row
- You accidentally performed a different movement
Copying 25 lines of 600 numbers from the serial monitor is easy to get wrong. You can optionally let a script copy each valid line into training/shuttle_dataset/new_gesture.csv. You still perform the gesture 25 times. Step 6 explains how to paste the lines yourself if you do not use the script.
The code repository and training bundle provide training/capture_gesture.py. Its implementation is shown below for reference:
1import argparse
2import re
3import sys
4from pathlib import Path
5
6import serial
7from serial.tools import list_ports
8
9START = "=== Data Collection Results ==="
10END = "=== End of Data Collection ==="
11ANSI = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
12ESPRESSIF_VID = 0x303A
13
14
15def board_ports():
16 skipped = ("bluetooth", "debug-console")
17 candidates = []
18 for info in list_ports.comports():
19 device = info.device or ""
20 label = f"{device} {info.description or ''}".lower()
21 if any(word in label for word in skipped):
22 continue
23 candidates.append(info)
24
25 espressif = [info.device for info in candidates if info.vid == ESPRESSIF_VID]
26 if espressif:
27 return espressif
28
29 usb_marks = ("usbmodem", "usbserial", "ttyACM", "ttyUSB", "COM", "wchusb", "SLAB")
30 return [info.device for info in candidates if any(mark in info.device for mark in usb_marks)]
31
32
33def open_board():
34 names = board_ports()
35 if not names:
36 print("No board serial port found.", file=sys.stderr)
37 return None
38 for name in names:
39 try:
40 port = serial.Serial(name, 115200, timeout=0.2)
41 except serial.SerialException as exc:
42 print(f"Skipping {name}: {exc}")
43 continue
44 print(f"Using {name}")
45 return port
46 print("Could not open a board serial port.", file=sys.stderr)
47 return None
48
49
50def main():
51 parser = argparse.ArgumentParser()
52 parser.add_argument("-n", "--count", type=int, default=25)
53 parser.add_argument(
54 "-o",
55 "--output",
56 type=Path,
57 default=Path("shuttle_dataset/new_gesture.csv"),
58 )
59 args = parser.parse_args()
60
61 args.output.parent.mkdir(parents=True, exist_ok=True)
62 rows = []
63 capturing = False
64 pending = ""
65
66 print("Long-press Boot for about 2 seconds, then release.")
67 print("Short-press Boot and perform the gesture once for each sample.")
68 print(f"Need {args.count} rows. Press Ctrl+C to stop early.")
69
70 port = open_board()
71 if port is None:
72 return 1
73
74 try:
75 while len(rows) < args.count:
76 chunk = port.read(4096)
77 if not chunk:
78 continue
79 pending += ANSI.sub("", chunk.decode("utf-8", "replace"))
80 while "\n" in pending:
81 line, pending = pending.split("\n", 1)
82 line = line.strip()
83 if "Mode switched to" in line:
84 print(line)
85 if START in line:
86 capturing = True
87 continue
88 if END in line:
89 capturing = False
90 continue
91 if not capturing or "," not in line:
92 continue
93 values = [item for item in line.split(",") if item]
94 if len(values) != 600:
95 print(f"Skipped a row with {len(values)} numbers.")
96 continue
97 rows.append(",".join(values))
98 print(f"Saved {len(rows)}/{args.count}")
99 except KeyboardInterrupt:
100 print("\nStopped.")
101 finally:
102 port.close()
103
104 text = "\n".join(rows)
105 if text:
106 text += "\n"
107 args.output.write_text(text)
108 print(f"Wrote {len(rows)} rows to {args.output}")
109 return 0 if len(rows) >= args.count else 2
110
111
112if __name__ == "__main__":
113 raise SystemExit(main())Exit the serial monitor with Ctrl+] before you start the script. Only one program can use the board’s serial port. Install pyserial, then run the script from the training directory. It scans the serial ports on the computer and prints the one it uses. You do not pass a port name.
- Install the Microsoft Python extension if it is not already installed.
- Open
training/capture_gesture.py. - Open a new terminal with Terminal → New Terminal and run:
cd training
python -m pip install pyserial
python capture_gesture.pycd training
python3 -m pip install pyserial
python3 capture_gesture.pycd training
python3 -m pip install pyserial
python3 capture_gesture.pyPowerShell, from the gesture_recognition directory:
Set-Location training
py -3 -m pip install pyserial
py -3 capture_gesture.pyWhile the script is running:
- Press and hold Boot for about two seconds, then release it. The terminal prints
Mode switched to COLLECTwhen the board changes mode. - Briefly press Boot and perform the gesture immediately.
- Wait until the terminal prints
Saved 1/25. Repeat until it printsSaved 25/25.
The script writes only lines that contain 600 values. It skips log lines, marker lines, and incomplete rows. Press Ctrl+C to stop early and keep the rows already saved. When it finishes, the CSV file already exists, so skip the manual copy in Step 6 and continue with Step 7.
Step 6: Create the CSV file#
Inside training/shuttle_dataset, create a file named new_gesture.csv. This is the filename used by the training script in Assignment 2:
gesture_recognition/training/shuttle_dataset/new_gesture.csvSkip this step if capture_gesture.py already created the file.
For each valid capture:
- Copy only the line between
=== Data Collection Results ===and=== End of Data Collection ===. - Paste it into the CSV file as a new row.
- Do not copy timestamps, log messages, marker lines, or a column header.
After 25 captures, the file should contain at least 25 rows. Every row represents one gesture and must contain exactly 600 values.
Step 7 (optional): Validate and save the dataset#
This validation is optional but recommended. The code repository and training bundle provide training/shuttle_dataset/validate_dataset.py; its implementation is shown below for reference:
1import pandas as pd
2
3data = pd.read_csv("new_gesture.csv", header=None)
4
5print("Samples:", data.shape[0])
6print("Values per sample:", data.shape[1])
7
8assert data.shape[0] >= 25
9assert data.shape[1] == 600
10assert not data.isnull().any().any()Run the script:
- Install the Microsoft Python extension.
- Open
validate_dataset.py. - Select a Python interpreter that has
pandasinstalled. - Click Run Python File (the triangular play button) in the editor toolbar.
The output appears in the VS Code Terminal panel.
python validate_dataset.pySpreadsheet alternative: Open the CSV in LibreOffice Calc, Microsoft Excel, or another spreadsheet application. Confirm that it contains at least 25 rows, exactly 600 populated columns, and no empty cells. Do not save it in a format that adds a header or changes the comma-separated numeric values.
Expected result:
Samples: 25
Values per sample: 600Save the validated CSV file. Repeat this collection process for every gesture class used by your model. A three-class classifier also needs an unknown class containing varied movements that do not match either defined gesture. Keep the class sizes reasonably balanced.
Acceptance criteria#
- The serial monitor confirms that the application is in
COLLECTmode. - The new gesture has a clear name and can be repeated consistently.
training/shuttle_dataset/new_gesture.csvcontains at least 25 valid samples.- Each CSV row contains exactly 600 floating-point values with no header.
- The dataset contains no empty or incomplete values.
- Separate, reasonably balanced CSV files exist for every model class.
