SPELL

Keywords: Unity, Machine Learning, TinyML, magic, Arduino, Bluetooth...

<Spell> 2022 Designed and documented by Xiangrui Zeng and Meijie

Spell consists of a spell-sensing wand and virtual game space. Players can hold the wand and say three kinds of spells (“Expelliarmus”, "Avada Kedavra", "Protego") to attack the monster or add a protective shield to themselves.

We recorded our friends speaking those spells at different speeds and accents. Each recording is about 1 to 3 seconds, for a total of 150 per spell. We also collected recording data for silence and unknown so that the wand will not activate when the player isn't speaking or saying the wrong spell. Then we uploaded all the data to Edge Impulse and train the machine learning model. The images below show our model.

We deployed the trained model on Arduino Nano 33 BLE Sense, and attach the microcontroller board to a wand. There is a microphone on the Nano 33 BLE Sense board, so when the player holds the wand and presses a finger on the pressure sensor, the signal will trigger the microphone to start collecting sound. Then it will determine which spell the player has spoken and send out a signal to the virtual environment made with Unity wirelessly through Bluetooth. In the game, different magic will be shot out based on the signal received from Nano 33.