Live Web week 5
Assignment: Come up with an idea for a midterm project, blog about it, prepare to discuss in class.
Reallocating my long term relationships to the digital realm has been an interesting process. Most of my friends live far from me, so the internet has become almost exclusively the site of our interactions. In acknowledging this new dynamic, I have been trying to figure out ways in which I can reinvent some of my favorite activities — specially the ones involving music.
A few months ago, I was video chatting with a friend, and he opened the sound of our call in his monitors. We then started playing around with feedback, and figuring out the possibilities of sound on that interface. We had tried to play and sing together a few times before, and since he has a lot more musical training then I do, we would find ways to measure the delay time, and he would play/sing that amount of time ahead, so that it would sound right to me.
Knowing that there is no way to instantly send the data, I started to think about the ways in which it would be possible to explore music using the limitations of the call system as a creative tool in itself. I wanted to build something that would allow us to play and record online in real time, but I was sure that there must be tools that let you do that, so I was curious to see how they would deal with the issue of the delay. In doing some research, I found a few services that offer functionalities that intersect with what I want to do, and here is their response to how they deal with latency:
The NINJAM client records and streams synchronized intervals of music between participants. Just as the interval finishes recording, it begins playing on everyone else’s client. So when you play through an interval, you’re playing along with the previous interval of everybody else, and they’re playing along with your previous interval. If this sounds pretty bizarre, it sort of is, until you get used to it, then it becomes pretty natural. In many ways, it can be more forgiving than a normal jam, because mistakes propagate differently.
How does jammr work?
jammr sends your audio to other users and receives their audio. In order to avoid lag, jammr is “live” but not “real-time”. This means other players hear you after a fixed amount of time. This fixed amount is called the interval and can be set to fit the chord progression you are jamming to. This allows you to jam synchronised with people from all over the world.
Why is jammr not real time?
At the speed of light it takes 67 milliseconds to go half-way across the world. This means jamming with someone halfway across the world would be noticably laggy. Therefore, to ensure that you can jam with people from all over the world without delay we use intervals.
How does JamKazam work?
Basically, JamKazam takes the music you play that is captured into your computer – either by a device you already use for recording or by your computer’s built-in mic and headphones – and then uses an ultra-low latency encoder to stream your music to each of the other musicians in your session – and vice versa. Your own music stream is mixed together with all the incoming streams into a master output that you hear in your headphones. This all happens so fast that it sounds like you are sitting in the same room together. You can also voice chat with each other at the same time to communicate and collaborate.
I’m skeptical. Is this really possible?
Yes, it definitely is. Major research universities have conducted various studies that indicate that as long as you can keep latency down to 25 milliseconds or less, musicians can play pretty much any music together, stay in sync, and have a great experience. Even above 25 milliseconds, many musicians can play a variety of music enjoyably – especially pieces with a slower tempo or pieces that require less precision. Many audio interfaces when combined with JamKazam will deliver 10 to 15ms of round trip audio processing latency to capture, encode, decode, and play music – basically everything but Internet latency. So with one-way Internet latency of 15ms or less, you can play pretty much anything. This equates to a ping time of 30ms between computers. A cross-country ping can be as low as about 75ms, so you can play with musicians pretty far away, as long as you don’t suffer from an irrational route path by your ISP that, for example, sends your packets from Austin to Atlanta to Dallas – like a really bad connecting flight plan. For more information, check out one of our favorite university research papers on this subject at http://bit.ly/1bry9QP.
These three gave me good insight on my options, so my idea is to build a social tool that, while still being a video chat, allows the users to record and export their sound, giving them access to options of feedback and delay. Since I don’t have a lot of knowledge on how audio works, I believe I will discover what makes more sense recording-wise by trying a few options, but for now, these are my goals:
- At least two video windows, one for the user stream and the other one for the partner;
- Allow the user to choose their partner in a list of online users;
- Give them control of the audio streams, allowing them to:
- Control their own volume (feedback);
- Control the interval in which they will hear their partner’s audio (latency);
- Control the other user’s volume;
- Record the audio from either sides;
- Export the files;
- Share the recordings in both sides;
- A text document to work together on lyrics and have it displayed at all times;
- Option to link/unlink parameters — making it so that the users can change the parameters only for what they’re listening, or to both of them;
- Optional metronome;
- Supporting more than two users;
- Adding both sides in a single audio file;
- Giving preference for the audio stream to be faster;
- Also record video;
- Add video effects;
I am already foreseeing issues with having the video, since the amount of data is way bigger, but I want it to be a social tool, focusing on building a fun experience for the users more than a musical tool, so I’ll try to work around this issue.