Hand Recognition Jetbot

SchoolworkFinal year project

GitHub: https://github.com/DnG-00/Hand-Recognition-JetBot.git

Project

This was my final-year project at Nanyang Polytechnic. I built a hand-sign recognition system that lets you control a car simulator using simple hand gestures. I finished the base features early, so I also ported the same system to an NVIDIA JetBot, for those who are unfamiliar, it’s like a robot car that has AI functionalities.

The longer-term idea was to pair the JetBot with a small robotic arm so it could help people with limited mobility pick up or deliver small items.

Libraries used

  • OpenCV (cv2) — capture frames from the camera and handle image I/O.
  • MediaPipe — detect hand landmarks (the x/y/z coordinates for key points on the hand).
  • TensorFlow / Keras — train and run the classifier that maps landmark data to gestures.

How it works

  1. OpenCV grabs frames from the camera.
  2. MediaPipe finds the hand and returns landmark coordinates for key points (thumb, fingers, wrist, etc.).
  3. Those coordinates are turned into a feature vector and saved to a dataset for the gesture you’re recording.
  4. The dataset is used to train a TensorFlow/Keras classifier.
  5. During live use, the model predicts the current gesture and that prediction is translated into simulator or JetBot commands.

Hand Gesture Recognition

Creating the dataset

  1. Run app.py to start the capture tool.
  2. Press K to begin a capture session.
  3. Put your hand in the neutral starting position for the gesture you want to record.
  4. Perform the gesture a few times while the session is active so you get multiple samples.
  5. Press the number key for the label you want to save samples under:
    • 0 = Left
    • 1 = Right
    • 2 = Throttle
    • 3 = Brake
    • 4 = Toggle Reverse
    • 5 = Full Stop (not used on the JetBot)

If you want extra labels, add them to /Client/model hand_history_classifier/hand_history_label.csv.

Training (hand_history_classification.ipynb)

There’s a Jupyter notebook that loads the recorded dataset, does preprocessing, trains the Keras model, and shows evaluation metrics. It’s a good place to try different model architectures, tweak preprocessing, or inspect which gestures are getting confused.

Collision avoidance

On the JetBot I have also added a simple collision-avoidance layer. If the robot’s sensors spot an obstacle, the avoidance logic takes temporary control, steers away, and then hands control back to the user once the path is clear.

Creating the dataset (data_collection-Modified.ipynb)

This Juypter Notebook is used to create the dataset that will be used to train the collision avoidance model.

Training (train_model_resnet18.ipynb)

This Juypter Notebook then re-trains the resnet18 model for collision avoidance using the dataset created.

Jetbot Builds

Hand Gesture Recog.ipynb

This Jetbot build only contains the hand gesture recognition program, which can be used to control it through the use of hand gestures after the hand_history_classification.ipynb have been ran to create the trained AI model.

Hand Gest + Colli Avoid.ipynb

This JetBot build includes an additional collision-avoidance subsystem. When the platform detects a nearby obstacle using its onboard sensors, the avoidance logic temporarily overrides user commands to steer the robot away from the obstacle. Control is returned to the user once the path is clear.

Video Demos