Audio scaling

Published: Oct 20, 2017 by nemanjan00

Hi guys, what I had to do few days ago is dynamically change speed of audio playback.

Terminology

Sample is sound amplitude.

Audio file is just encoded/compressed list of samples.

Samplerate is number of samples per second. (the one I will be using is 44100)

Buffer is Array of samples that are going to go out of your speaker. (Common sizes are 256, 512, 1024, etc. sample rates)

What did I end up using?

Technology I was using for that is Web Audio API.

AudioNode

AudioNode is basicaly Object which you pipe audio into and/or get audio out of it.

AudioNodes connect to each other just like any audio equipment would in real life.

For example you maybe have microphone, speaker, spectrum analyzer with passthrough and amplifier.

You can connect for example microphone -> amplifier -> spectrum analyzer -> speaker

Getting audio inside system

What most of the people in Web Audio API would use if they wanted to play music inside it is MediaElementAudioSourceNode (implementation of AudioNode that gets sound from HTML5 audio tag).

It simply plays audio and that is it.

Problem with this one is that it is impossible to change speed. (might be possible to slow it down, but I did not find any ways to make it play music faster)

decodeAudioData

What I ended up doing is getting mp3 using XHR as ArrayBuffer and running decodeAudioData from audio context on it.

That gave me back buffer containing whole audio:

module.exports = function(url){
	return new Promise((resolve, reject) => {
		const audioCtx = new (window.AudioContext || window.webkitAudioContext)(); // Getting audio context. Anything you do with Web Audio API requires use of audio context

		const xhr = new XMLHttpRequest();

		xhr.open('GET', url, true);
		xhr.responseType = 'arraybuffer'; // This will make sure I get response in format I need for passing it to Web Audio API

		xhr.onload = function(e) {
			var audioData = xhr.response;

			audioCtx.decodeAudioData(audioData, function(buffer) { // This will decode that audio file and return buffer using callback. 
				resolve(buffer);
			}, function(e){
				reject(e);
			});
		};

		xhr.send();
	});
}

What I can do now is take data from buffer at any rate.

Scaling audio using input audio buffer

Now that I have list of amplitudes I can play that audio back.

Web Audio API does not have any way to generate sound, just process it (at least I did not find anything like that).

What I ended up doing is using ScriptProcessor (AudioNode for transforming sound).

I did not hook up any audio sources to it, just hooked it up to sink.

transformationFunction is function that is getting called each time you need to process audio. Inside it I am taking audio from music buffer and putting it into output buffer.

Calculating sample in output buffer

What I have is huge buffer that contains song and I have very small buffer I am writing to.

I have counter i which tells me where I am inside input buffer.

What I do is iterate inside output buffer and calculate what I want to take from input buffer for that sample.

Let’s say playback speed is 1.5.

What I need to put into output buffer are samples 1.5, 3, 4.5, etc.

But, there are samples 1 and 2, and there is no sample 1.5, what do I do now?

Well, what you can do is:

sample = 0.5 * input[0] + 0.5 * input[1];
module.exports = function(audioCtx, buffer, outputNode){
	let i = 0;
	let cnt = 0;

	let speed = {
		speed: 1,
		playing: 1,
		buffer: undefined,
		transformationFunction: function(audioProcessingEvent) {
			let outputBuffer = audioProcessingEvent.outputBuffer;

			let buffer = speed.buffer;

			let oldI = i;

			let end = false;

			for(chan = 0; chan < 2; chan++){
				i = oldI;
				let position = 0;

				let input = buffer.getChannelData(chan);
				let out = outputBuffer.getChannelData(chan);

				do {
					i += speed.speed * speed.playing;

					if(i < 0 || i >= input.length){
						end = true;
						speed.playing = false;
					}

					if(!end){
						let floor = Math.floor(i);
						let diff = i - floor;

						out[position] = input[floor] * diff + input[floor + 1] * (1 - diff);
					} else {
						out[position] = 0;
					}
					
					position++;
				} while(position < out.length);
			}
		},
		init: function(audioCtx, buffer, outputNode){
			const scriptNode = audioCtx.createScriptProcessor(512, 0, 2);

			speed.buffer = buffer;

			scriptNode.onaudioprocess = speed.transformationFunction;

			scriptNode.connect(outputNode);

			return speed;
		}
	}


	return speed.init(audioCtx, buffer, outputNode);
}

Share

Latest Posts

How to print all rejections in NodeJS
How to print all rejections in NodeJS

Did you forget to print error on rejection, in a huge project and are now having issues debugging it? I wrote a snippet of code to solve that.

Why argv sucks for users
Why argv sucks for users

There are some conventions and patterns that are part of so much software we are never getting rid of them. One of those conventions defines how we pass arguments to applications.

Do not run chrome from your app!
Do not run chrome from your app!

This is example how your user can be hacked if you misuse chromium to open web links.