Artificial Intelligence in Music

Our App changes the way music is taught and trained and is based on state-of-the-art multimodal (audio, image, video, and motion) technologies.

What is it?
It is an intelligent music learning App based on sound and motion real-time analysis and artificial intelligence technology. The App monitors sound quality, intonation, rhythm, posture and gives expression feedback via the use of artificial intelligence.

It contains:

  • Catalog of exercises and pieces from beginner to professional levels
  • Multimodal Sound quality (tone) feedback
  • Music score-based real-time intonation and rhythm feedback
  • Dynamics (loudness level) real-time feedback
  • Motion capture for real-time feedback of gestures
  • For strings instruments, bow speed, tilt, angle, contact point (distance to the bridge), inclination, and trajectory feedback.

Why use it?
It empowers students with tools to detect and correct errorsbuild awareness of their technique and allows them to practise more efficiently, whether alone or in a class setting.

Who is it for?
It is designed with a wide range of student levels in mind, from beginners to professional players.

Sound quality
In instruments such as the violin, clarinet and trumpet, good sound production is one of the first skills to acquire. Our App allows you to improve it.

The AI is able to automatically learn to differentiate sound qualities/colors by listening to samples provided by the users. Once the AI has learned different sound colours (e.g. gentle, sweet, tight, warm, excited), it can provide real-time feedback about the sound colour being played both in isolated notes and in musical pieces.

Posture/Technique
Mastering an instrument technique is of paramount importance. Our App provides students with real-time feedback on their gestures using motion capture.

Intonation/pitch
In instruments such as the violin, in addition to playing the correct note (i.e. pitch), playing it with the correct intonation (i.e. tuning) is very important. Our App provides intonation visual feedback to music students.

Pitch/timing
In music performance playing the correct note (i.e. pitch) at the correct time (i.e. rhythm) is of paramount importance. The App provides pitch and timing accuracy visual feedback to music students.

The video illustrates AI assessment of the intonation and timing of a student’s performance. The AI is able to detect and provide real-time feedback about whether a note is played with the correct pitch (showing the note being played in green if it is the correct pitch, and a red note/cross to indicate mistakes), timing (placing the performed note before, on, or after the corresponding score note), and intonation (placing the played note as a green note slightly above or below the corresponding score note). The student can click on areas/bars where there are detected errors and the AI will show an explanation video or exercise relevant to the error type detected. After the student has practiced, the AI gives feedback on the improved student’s performance.

Gestures

Using an off-the-shelf 3D camera (e.g. Kinect, Realsense camera), the AI can accurately track the position of the violin and bow during music playing. Using this information, the AI is able to provide real-time feedback about bow skewness, bow angle, contact point (i.e. distance of bow contact point to the violin bridge), bow tilt (i.e. bow rotation), bow speed and trajectory. This feedback can be used to correct posture and gesture errors.

Bow stroke technique

Using audio analysis and artificial intelligence methods, the AI is able to detect different bow stroke techniques (e.g. Détaché, Martelé, Bariolage, Stacatto, Spicatto, Ricochet, Sautillé). The system detects bow technique errors and shows a relevant explanation video or suggest technique exercises to allow the student to fix the problem.  After the student has practiced, the AI gives feedback on the improved student’s performance.