Interactive Sketchpad

This project was completed as the final project of Northwestern University’s MSAI 495: Computer Vision. The goal of the project was to implement various computer vision techniques and concepts in an application that allows a user to draw on a virtual canvas using their hand.

Demo

Tools Used

Pipeline

When the application is running, webcam frames are captured using cv2.VideoCapture() and fed into the MediaPipe Hands model. If a hand is detected, then the 21 landmark positions (shown in the image below) are updated.

One important note is that only the first hand detected on-screen is recognized as the “drawing hand” and is used for landmark updates and interacting with the sketchpad. With this being said, if two hands are detected and the “drawing hand” moves off-screen, then the other hand that is still on-screen is recognized as the “drawing hand”. Based on landmark position updates, different user actions can be discerned such as selecting colors, adjusting a slider, or drawing on the screen and the sketchpad can respond accordingly.

Features

Usage

When the applcation is running, a user can move their hand on-screen where it can be detected. Once detected, the user can extend their thumb to turn the cursor on (shown below) which also causes the words “Cursor On” to be displayed in green text in the bottom-right corner of the screen.

If the cursor is turned on, any finger on the “drawing hand” (besides the thumb) can be used for adding annotations if the finger is extended as shown in the example below.

Colors can be selected, buttons can be pressed, and sliders can be adjusted by moving the “drawing hand” index fingertip within the associated region.

Potential Improvements and Source Code

GitHub Repo