Perfect ear is a playful ear training tool for musicians to train their musicality. Through AI technology, users can use gesture as input to interact with music notes.
Machine learning
Oct - Dec, 2019


Perfect Ear is a playful ear training tool for musicians to train their musicality, which is a is an important and desirable skill for all musicians. Traditional ear training methods, which can only happen in class, might be tedious and boring. Compared to them, Perfect Ear provides a chance for musicians to train their ear after class, without a piano or a teacher in an easier and fun way. It is friendly enough for music amateurs and beginners with its playful experience.


The core design feature of Perfect Ear are the uses of AI technology and solfege hand signs, which is also called the Kodaly hand signs. It uses machine learning techniques to allow computers to recognise gestures, so users can use hand signs as an input to accompany each pitch they hear, and even create music with gestures.


My experience at Berklee College of Music

Last summer I got the chance to go to the summer program of Berklee College of music. I’m a vocalist and I went there to learn vocal music, as well as some knowledge and skills that all kinds of musicians should have, such as music theory and musicianship.

I was a little uncomfortable with some experience. For example, piano was needed for most learning and practicing experience. But I’m a vocalist and it’s hard to have a piano with me every time when I was learning music theory or doing some ear training, especially after class.

So I was thinking...

What if....

  • professional training method were more accessible to music amateurs?

  • ear training were more fun?

Design Purpose

To make professional music learning training method more accessible to music amateurs and beginners and make ear training more fun with a playful experience using multimedia technology.

Design Statement 1 (previous)

I want to design a playful learning tool for music beginners to learn more and train their musicality.


Previous Prototypes


These two prototypes are inspired by the color version of Tonnetz, or “tone network”, found in Riemannian theory. It is an arrangement of notes to illustrate harmonies and their relationships.


Each triangle represent a chord which is made of three notes. Users can sort the triangles into different sequences to create their chord progression. It is a very useful visual aid for understanding music theory and composing.


The statement of my design was not very clear.

  • What kind of "music beginner" do I target to?
  • What specific kind of experience do I focus? Learning or training?

Learning and training are two different experiences. Learning is about the process of knowing and understanding existing knowledge, while training is more about practicing a skill. It is hard to create a playful experience that can approach both of learning and training aspects.

Therefor, I thought about making my statement more specific by specifying my target audience and design goals.

Design Statement 2 (Final)

Design a playful ear training tool for musicians to train their musicality.


What inspired me most is the solfege hand signs (also called the Kodaly hand signs or the Curwen hand signs), which is always used in music class. It provide a visual aid during singing exercises. The distance between them is also a good example of the size of the interval they represent.

So I thought maybe I can use the gesture as an input to do something interesting.

I used teachable machine from google to allow computers to recognise gestures.


Perfect Ear has two mode: the practice mode and the game mode. In the practice mode, users are supposed to learn and memorise the hand signs that corresponding to each pitch. By following the instruction pictures bellow, users gesture in front of the camera to trigger a tone and a visual effect at the same time. In the game mode, users are supposed to listen to a tone first. Then, recognising what pitch of it only by ears, users respond with the correct hand sign in front of the camera to light up the star. The computer will tell if they do it right or wrong.

Future Work

I would like to add higher level in the future. At this stage, it may seems that this project is only for beginners, since it onlyprovides training experience for major pentatonic scale. But it can be more professional when it comes to higher level. There are manyother scale and modes in music that people can train with. And I will also keep working on the UI, which now seems to target more on chirldren.

discover more projects