Musical Scale Game

Hi everyone.
Just to let the community know I'm busy with a game to make practicing scales fun - I'm a music teacher, and I always find practicing scales boring.

I'm using Unity to make it a 2D app for Android (at this stage).
Basic idea is that you select a scale to practice (or go through levels increasing in difficulty), then you play the scale and the game gives you a score based on how many notes were correct, tuning, speed, etc. If your score is too low, you don't pass that level. If you get a high enough score, you get a reward (still figuring out what the reward will be).

I've found other scale apps on Play Store, but none that can tell you if what you're playing is correct or not, which will be my main feature.
I've already created a basic prototype that can display the scale and also figure out what note you're playing from the audio input, check if they're the same or not, and give out points. Will be testing with instruments soon, then will upload a video to show you.

Biggest challenge by far: pitch recognition - using unity's GetSpectrumData (Fast Fourier Transform, so at least I didn't have to write my own FFT algorithm). It's really not as simple as "the strongest frequency is the note", as lots of people on forums spit out. Particularly when using a small microphone, that doesn't pick up low frequencies so well (I've been testing with the built-in laptop microphone, comparing with a studio microphone). Way more complex, and I'd love to get into it if anyone is interested. Sound techies and programmers out there?

I'm quite new to game making, and graphics are not my skill at all, so I may be looking for some help (even paid help) to make the game look awesome and marketable.

That's all for now.

Comments

  • Sounds great. I love games that are educational and fun at the same time :)
  • Oh man. I always hated practising scales.

    I've struggled with detecting pitch before too, especially with an instrument with a more textured timbre (in my case a violin). I'll be keeping an eye on this thread for solutions!
  • edited
    This is what we would call a "non-trivial" problem. You're not just dealing with the complexities of detecting pitch, you're also dealing with the complexities of hardware, latency, shitty microphones and a whole host of other low level problems you probably don't have very much control over. Note how rocksmith ships with it's own peripheral to deal with this? Also note that they are just targeting a single type of instrument (ie guitar).

    I don't want to shit all over your idea, if you can get it working that would be great! But I wonder if perhaps you're not barking up the wrong tree? For me I usually know when I've messed up a scale, unless I'm blazing so fast no microphone/program is going to be able to keep track anyway.

    So let's take it back a bit, practicing scales is boring as shit. This is a real world problem you've identified that's worth solving. But why is it boring? IMHO It's because most scales practice is just mechanically trudging through the series. I'm not really concerned with playing a scale right, because the end of the day I've only learnt how to play a scale.

    So how about an app that actually shows you how scales are awesome. That show's you the ways you can use scales as compositional aides, as solo techniques, as ways of making your playing more musical. This then becomes something I care about. It's a much tougher design problem, but engages with how people learn, and why they care about the stuff they are learning. ;)

  • Also vaguely worried about the tech impacts of pitch detection. I know that Evan Balster is working on a pitch detection thing called Imitone that is apparently all sorts of hax, but he's been deep into this for ages now. Like people licensing his tech levels of hardcore.

    Anyway, what about other ways to encourage people hitting the correct notes? Like playing tones or notes that would be dissonant with any other note played by the user? (Not a musician, but the dissonance in Guitar Hero made me try harder)
  • Thanks guys for the awesome comments and suggestions. Will keep them in mind as I go on.
    @TheFuntastic:
    I hear you and completely agree with what you're saying. Scales are very boring, and at the end of the day not as useful as everyone claims for being musical. But there are a lot of things (particularly at school level) that become a lot simpler and allow you to get into the real music of a piece if you're not having to worry about what the notes are (the initial idea for the app was to program the student's piece into the game, then have them play that; scales are just way easier to program!). I will definitely try to instil useful learning of scales, not simply rote performance, if I can. Maybe that's the PRO version! :)

    Unfortunately my testing found some more bugs with the pitch recognition. Trombone and trumpet are still finding different octaves instead of the fundamental, but it works perfectly for voice - think it's something to do with overloading the mic, but I know I'm close.

    I don't mind sharing my pitch recognition idea, and if anyone has suggestions, I'll welcome them!

    Basically, take the FFT data, sift through for the top 6 peaks (hopefully they are all harmonics of a fundamental). Take the highest peak (which has to be a harmonic), divide by 1,2,3 and 4 to get possible fundamentals, then create a harmonic series for each of them. Compare the harmonic series with the top six peaks and see which has the greatest correlation. That should be the fundamental.

    All worked perfectly until you sing a different note - the FFT smears over time, so I end up picking up harmonics of multiple fundamentals. Used a weighting system to find the fundamental with the highest peaking harmonics, and it works great for voice. Now to figure out why I'm getting some ghost peaks in there that throw the note up or down a few octaves (not always exact octaves) for a splitsecond...
    Thanked by 1Tuism
  • edited
    This seems to me to be a very interesting project. I guess it depends on the game that gets attached to it, but pitch recognition opens up a lot of awesome opportunities.

    Robin Arnot is working on a game that uses your voice (or any sound on a microphone) to control the visuals. https://www.kickstarter.com/projects/soundself/soundself (Also with Evan Balster)

    Sound Self (the game Robin is working on) is designed for meditation, rather than improving singing ability, but any game that uses voice as a controller would be quite novel. If you can detect pitch in voice that opens up a lot of possibilities.
  • Above is a link to a video I made just now with a first public testing of the app.
    @BlackShipsFilltheSky, thanks. I mostly hang out with my colleagues at school, most of whom are interested in the possibility of getting rid of some of the more tedious work of a music teacher - practising intervals with a student, for example. Aural practicing is certainly an avenue this could branch into, but I also want to make it as fun and engaging as possible, more like a game and less like an excercise. Difficult, when most people hear "scales" and want to go do something else...


  • Progress report of the scale game (really need a better name...)
    This video shows how a student selects the instrument and level (UI and graphics to be improved hugely) then tries the scale, sees where they went wrong, then tries again to improve the score.

    Features now include blocks that show tuning, better pitch detection algorithm (even works on my android tablet, but this was recorded on iMac with external microphone).
    Also basic UI and level creation is in place. Levels can be in free time (as here) or at a set tempo, major, minor scales on any note, up to three octaves, up, down or combination. Also minimum score to pass the level can be set, and some or all of the notes can be hidden from view to test memory or ability to work out a scale.

    Still to come: rewards system - thinking of rewarding with media on composers, with levels grouped by composer and practical Grade (i.e. Grade 1 scales are all about Mozart).
    Another idea that was very popular with my students was having Mozart and being rewarded with funny clothing to dress him up in...

    Any comments, or anyone like to make a donation (just kidding, sort of), feel free to let me know.
    Thanked by 1NickCuthbert
  • I really like this idea. Being a music teacher myself I feel making scales more fun is really essential to get pupils interested especially the younger ones... the only suggestion I have is possibly finding ways to dress the game up to make it seem less technical without changing anything in the way it works.

    I really don't know how feasible this is as I am have little to no programming knowledge but an example I thought of is instead of showing the music as the primary display have some sort of basic platformer game where as you play the right notes your character can jump over obstacles as you play, or defeat enemies etc etc. Then the music can maybe be displayed in the bottom right for reference and you can get more complicated levels as the tempo increase or as you make the player switch between scales...

    I've thought of something like this before except with aural training, like trying to get people to recognise intervals and I thought if different intervals could be player to perform different attacks or moves that would progress a character through a level in a similar way to mentioned above.... never got round to finding someone else who cared though :P
    Thanked by 1Gazza_N
  • Thanks for commenting, Tim.
    Funny you say that, just this week I was talking with a colleague about making it more game and less exercise. What we want to do is have an avatar (Mozart, or Bach possibly) climbing up and down stairs made by the coloured tuning blocks in the video. Don't play the note in time, and the avatar falls to his death... or the note becomes a bomb and blows him up... you know, something really kid friendly. Now the player's job is not to play a scale, but to help Mozart get across a river teeming with crocodiles.

    Definitely possibilities to expand into the aural training sector here... Like your idea of different intervals being different moves that the avatar makes.
  • Calling any Unity (4.6) wizzes who feel really kind today:
    I'm using the new UI ScrollRect thing to make my scale scrollable. It looks pretty, works with touch and mouse, just allround cool tool. Unfortunately, since the scrollrect (which is on a world space canvas) has a rectransform and my content uses transforms. The two don't play nicely together. They display OK, but for some reason, the gameObject positions are all messed up.
    When I match positions of an object with recttransform and transform, they don't actually match.
    i.e. a.transform.position=b.transform.position. a is the rectransform, and b the transform. But the rectransform appears below the transform...
    And I can't find the problem, or use localPosition because changing the Local/Global button at the top doesn't change the values...

    Very difficult to describe this clearly enough for anyone to give me a fix, but maybe someone else has used rectTransform with Transform and found similar problems that they fixed?
  • edited
    You probably need to set the anchor position of the rectTransform. Just learned a bit about this today. I think if you select the rect transform and click on the little square that show where it is anchored, you will be able to set the anchor pivot. Since your rect transform appears below a transform, try to set the anchor to top center.
  • Can you provide us with a screenshot? I don't think I understand the problem correctly, but I assume you want to drive the scale's transform by using the slider? I'd suggest that you use the value of the scrollbar to set the scale's position between a range of two transforms? But as I said, this is just a shot in the dark.
  • Thanks for the respones. Screenshot to follow when I get a chance. Sorry, I said "below" in the previous post. The object with recttransform, which should appear at the same y-axis position as the object with the transform, instead appears about half a unit lower on y-axis.

    Also discovered that my note sprites (all on transforms), which are generated at run-time and moved into position by script, get put into the correct place, but the game objects that hold them (and normally have the same position), are also slightly out of position. And all of the notes, regardless of where they actually are, have the same position readout in the inspector. Maybe it's fetching the position of the closest parent with a recttransform? Or some other crazy RectTransform anchor setting?

    To be clear, the scroll rect is working perfectly. It's just that the objects on the scroll rect don't want to position properly.
  • Here are some screenshots.
    And here is some of the code:

    Transform noteTrans = note.transform.Find ("Note Sprite");

    Vector2 newPos = new Vector2 (noteTrans.position.x - scaleScript.noteSpacing * 4, noteTrans.position.y);
    firstFloor.position = newPos;

    Before anyone suggests, I've tried using localPosition too. localPosition.y of the sprite holder is always 0. And it seems that position.y is always 1.579003. No matter where the actual note is!!

    Anyone got ideas to help?
    The third screenshot is the heirarchy. Selected items are RectTransforms, others are Transforms
    Floor 1.png
    626 x 437 - 432K
    Floor 2.png
    626 x 442 - 439K
    Heirarchy.png
    258 x 288 - 4K
  • Just been looking again. The y values for the note change in the inspector depending on where the note is, but when I access them with code it's always the same value. Why, oh why?
  • Hi
    A late thought here (if it hasn't already been mentioned) - to get accurate note recognition it might help to correct for room resonance. App would first record room frequency. Then would remove this from the ftt analysis recording of instrument. Or at least suppress it if it coincides with the resonance of the instrument.
  • So, after looking through some conversations in the Unity General Questions Discussion I came across a conversation about order of execution, and wondered if that might be my problem. Put a call to my code on a button instead of start(), and true as bob, worked perfectly. For some reason that I haven't quite fathomed, the call in start() happens before the note sprite gets moved to its correct place, so of course the calculation is always the same. Fix: new Waitforseconds (0.01f) before calling any of that code (in an IEnumerator). I'm sure that's not the best way, but it works. Anyone have a cleaner way to do this?

    Thanks to the community for the help. And thanks @Tuism for having an execution order problem in 2013!

    @Whitechina, not at all a late thought. The app currently tells the user to be quiet for a couple of seconds to calibrate silence and correct for the amount of noise (assuming the noise stays the same, like a buzz in the background), but I hadn't thought about room resonance... That would most likely require exciting the room resonance with a sweeping tone or such during calibration. Interesting... may be beyond the scope of what this app can do, since it can't control the quality of the device's speaker, but a very interesting thought.
  • edited
    Time for an update!
    1 min video showing some gameplay and game modes.
    Starting to get art and animation in. Slowly... But starting to look a bit more like a game and less like a music test.



    Critique always welcome. And of course, work in progress.
    Thanked by 2MCA Sash
  • This is looking very cool and promising! How are you controlling input? Or, what is the input controller? Are you playing the instrument into a mic? Do you/will you have MIDI support?
  • Yes, playing into a mic. On a mobile device I'm relying on the built-in mic, so iOS is looking more promising than Android since their mics are all the same and decent quality, though I'm testing on a Galaxay Tab 2 and it seems to be OK.
    I don't have MIDI yet, but it shouldn't be too hard to get that in if it was ever required. The main focus right now is single line instruments, to focus on speed and intonation. And just learning and practicing scales in various different patterns.

    The video is me playing trombone into an SM58, with an audio interface to PC.
  • edited
    Update time! Making good progress with the game design of this thing: Got levels going, managed to figure out a scoring system and a quick level creation tool for multiple instruments. Basically I make a pattern of consecutive levels, then choose the instrument and starting note, then I've got those levels, instant. Happy with that.

    But! Since upgrading to Unity 5.1.0 I've been struggling with the audio when I test on Android (in editor is still working beautifully). Any note I play only gets registered about 6 to 10 seconds after I play it. Here is the code that starts the recording and playback process :
    if (audioDevicePresent) {
    micPlay.clip = Microphone.Start (micDevice, true, 10, samplerate); //Record from default mic into 10 second clip looping, at 44,1kHz
    micPlay.loop = true;
    micPlay.mute = true; //We don't want to hear what the mic picks up.
    while (!(Microphone.GetPosition(null)>0)) {
    }
    micPlay.Play ();
    }

    I changed that one line to:
    micPlay.clip = Microphone.Start (micDevice, true, 1, samplerate);
    and now I get about 1 second of delay, so assuming it has to do with the length of the clip that I record. But my reasoning from that is that it gets recorded, then played back only the next time around the clip.

    Anyone got ideas of how I can make it as "instantaneous" as possible? I know latency is always going to be an issue on mobile, but as far as I can tell from some profiling I'm not pushing the CPU very hard with the processing it needs to do, and the bulk of the processing only happens above a threshold. And it's only picking up the threshold delayed.

    Any help much appreciated!

    EDIT: Forgot to say, testing on a Samsung Galaxy Tab 2
  • @McCoySoundD, dumb question I know, but where are you putting this code, in the Start() method?
  • Thanks for the response, Nick.
    It's in its own procedure, which gets called in start, and from other places, whenever the audio settings change, for instance.
    Thanks also for the links. Have looked at some of the answers, seems the closest response to what I'm doing has the same problem, but an interesting solution. Will go experiment, then report back.
  • OK, so experimentation complete, just raises more questions! Here's my reasoning:
    It seems like that 1 second of delay (changeable by setting the length of the clip), again, only on Android!!, is due to the read head being ahead of the write head (if you'll excuse the tape analogy). It's trying to play what hasn't been recorded yet, but because it's a loop, its playing what was recorded a second ago. Solution seems simple, add a little latency, delay the playback by a few milliseconds. I do so, and set about measuring the difference in samples between the playback position and the recording position.

    Result: the gap between the two is constantly changing! How can this be? Sometimes the playback is ahead of the recording, sometimes behind, by up to about 0.2 seconds. Digital audio boffins, please help.
  • edited
    Probably just a limitation of Android, combined with the overhead of Unity: superpowered.com/androidaudiopathlatency/
  • OK, it's been long enough since the last update. Still haven't fixed the whole latency thing, but there is news. My artist got too busy with her studies and asked to leave. So I've got a new super-artist and marketing go-getter who is making me rethink everything, not just as an artist, but as a project partner, which is great for me, since I've been working alone for a long time on this.

    I know that everyone here has been helping (which I'm very grateful for), but it's not the same as someone coming over, trying it out and saying "what about this?"

    So I'm very excited about this again, and I've been working on getting some important scene flow aspects correct (we want to have a more seamless design, going from one scale onto the next, rather than loading up a new scene that breaks the narrative flow). Will post a video again when I have something.
  • edited
    You should add some relative pitch training tools - I'm studying classical guitar and this is a BIG one! I know it's a little outside of the scope of the project, but it could be a cool addition...
  • Hi jackshiels.
    That is almost certainly the next step, probably a separate project building on similar ideas. Aural training can be helped hugely by the app knowing what note you're singing.
  • edited
    Hi MakeGamesSA community!
    Finally time for a playable build! And here it is (for Windows, even though the intended market is Android and iOS, because most people here are on Windows, I think):

    https://www.dropbox.com/s/f4b0fau2tnmpvcl/ScaleGameWinx86_1_38.exe?dl=0

    If anyone would like to try this on an Android device, let me know and I'll give you a link. That would be great in terms of microphone etc testing.

    Some things to note:
    The game requests permission to connect to the internet. I have no idea why, it's something that got added when I upgraded to Unity 5. I promise the app does not connect to the internet (I would have no clue how to write network or internet programming). It seems to still work if you don't allow it through your firewall, which is great.

    Plug in your microphone and set your default recording device before opening the app. I haven't got around to in-app changing devices yet, and for Android/iOS may never have to.

    The art is primarily programmer art, so please don't get put off by the basic look. I've just got a new artist in and we're rethinking the whole look of it from the ground up, so keeping the old stuff in there while we're still in the design phase.

    This game should be playable with the following instruments or voices:
    Woodwinds
    Strings
    Brass
    All voice types

    I've made basic levels for each instrument, so feel free to try it out on whatever you have to hand. A warning, it's not designed primarily for voice (I would need to add helpers like a note starting sound that plays). Voice is mainly in there to test microphone things when I don't want to go fetch my trombone every 10 minutes.

    There is no audio whatsoever - pretty weird coming from a sound guy like me making a music game, but I thought it would interfere too much with the pitch recognition algorithm. Are players likely to wear earphones while playing their instruments just to hear little congratulatory or sad noises when they pass or don't pass a level? Let me know what you think.

    All constructive criticism welcome, even on art.

    Thanked by 1Mexicanopiumdog
  • It gave me an error when I tried running the .exe :(
  • Darn, sorry, will try fix that. Don't suppose it told you what error? Did you have audio interface running at the time?
  • https://www.dropbox.com/sh/cgfdj3f47zx3do7/AADaptzepO0p83bjgh4F81-ga?dl=0
    New link. Sorry, my stupidity... thought I could just upload the exe, seems you need a data folder too.
    Thanked by 1Sash
Sign In or Register to comment.