1.1. Implementation¶

The implementation is divided into three stages:

1.1.1. User Interface and Data Processing¶

The first stage involves the user interface, implemented using Marcelle and a web framework that handles data collection and processing.

This stage combines:

Frontend: Implemented in JavaScript, it includes the Marcelle interface (GUI) where users interact with the system to collect training data.
Backend: Developed in Python, it processes the collected data and prepares it for model training.

The second stage is the AML training server, responsible for creating trained AML models from the user-provided training examples.

Key components include:

Server Implementation: Built using Python and Flask, leveraging the AML Engine’s Python bindings.
Model Serialization: The model structure is serialized to JSON and sent to the web application.
JavaScript Integration: A JavaScript counterpart of the model is created for real-time predictions. For comparison algorithms, TensorFlow.js is used, allowing easy integration into Marcelle and enabling the user to compare AML results with various models. In our use case, we added a neural network that can be trained in-browser without additional components.

Warning

This feature is only available when the dataset is Sensors.

The third stage involves mapping user-defined gestures to sound patterns using real-time predictions based on the selected model.

This stage includes:

Real-time Predictions: Utilizing the JavaScript model created from the AML server, the system maps gestures to sound patterns in real-time.
Sound Patterns: For demonstration purposes, we used four pre-determined musical instruments to showcase the system’s potential.