Streaming Audio

This topic contains 15 replies, has 4 voices, and was last updated by Diffrenzy 2 years, 7 months ago.

Viewing 15 posts - 1 through 15 (of 16 total)

1 2 →

Author

Posts

June 26, 2016 at 2:50 am #1232

Simon Armstrong

Participant

As the SDL2 audio mixer is a little limited I thought I would have a go streaming audio in monkey2.

My first issue was that SDL_OpenAudio returning an already in use error. Calling Mix_CloseAudio fixes that for us but means we are on our own and can’t share device with mojo2 mixer audio.

Second issue was a small bug in sdl2.monkey2[1025] which should read

Alias SDL_AudioCallback:Void(Void Ptr,UByte Ptr,Int)

And with that solved we have a functioning callback and a question.

Can I cast a monkey2 class to Void Ptr?

#Import "<std>"
#Import "<mojo>"
#Import "<sdl2>"

Using std..
Using mojo..
Using sdl2..

Global instance:AppInstance

Class VSynth Extends Window

	Method New(title:String)
		Super.New(title,720,560,WindowFlags.Resizable)				
		OpenAudio()
	End
	
	Field audioSpec:SDL_AudioSpec Ptr

	Method OpenAudio()
		Local spec:SDL_AudioSpec
		spec.freq=44100
		
		spec.format = AUDIO_S16
		spec.channels = 2
		spec.samples = 16384
		spec.callback = audio_callback
'		spec.userdata = this

		Mix_CloseAudio()		

		Local error:Int = SDL_OpenAudio(Varptr spec,audioSpec)
		
		If error
			Print "error="+error+" "+String.FromCString(SDL_GetError())
		Endif
		
		SDL_PauseAudio(0)
	End
	
	Function audio_callback(a:Void Ptr, b:UByte Ptr, c:Int)
		Print "callback!"
	End

End

Function Main()
	instance = New AppInstance	
	New VSynth("VSynth0.01")
	App.Run()	
End

#Import "<std>"

#Import "<mojo>"

#Import "<sdl2>"

Using std..

Using mojo..

Using sdl2..

Global instance:AppInstance

Class VSynth Extends Window

Method New(title:String)

Super.New(title,720,560,WindowFlags.Resizable)

OpenAudio()

End

Field audioSpec:SDL_AudioSpec Ptr

Method OpenAudio()

Local spec:SDL_AudioSpec

spec.freq=44100

spec.format = AUDIO_S16

spec.channels = 2

spec.samples = 16384

spec.callback = audio_callback

' spec.userdata = this

Mix_CloseAudio()

Local error:Int = SDL_OpenAudio(Varptr spec,audioSpec)

If error

Print "error="+error+" "+String.FromCString(SDL_GetError())

Endif

SDL_PauseAudio(0)

End

Function audio_callback(a:Void Ptr, b:UByte Ptr, c:Int)

Print "callback!"

End

Function Main()

instance = New AppInstance

New VSynth("VSynth0.01")

App.Run()

End

June 26, 2016 at 3:35 am #1234

Mark Sibly

Keymaster

Nice idea, but I think you’ll run into problems with the complete lack of thread sync in mx2, as the audio callback is called on a different thread (I think).

An alternative would be to have the SDL callback trigger a ‘main thread’ callback. There’s actually a kludgy behind-the-scenes way to do this in mojo.app and mojo.process that I eventually want to wrap in a nicer way. This would only work as long as the main thread was running fast enough of course.

Alias SDL_AudioCallback:Void(Void Ptr,UByte Ptr,Int)

Thanks, fixed.

Can I cast a monkey2 class to Void Ptr?

Nope, but you can wrap the object in a struct and pass a pointer to that. I use this in the pixmap loader to pass the mx2 stream as ‘context’ void ptr, something like:

Struct Context
   Field stream:Stream
End

Function Callback( context:Void Ptr )
   Local stream:=Cast<Context Ptr>( context )->stream
   ...got the mx2 stream...
End

Function LoadPixmap(...)
   ...open stream etc...
   Local context:Context
   context.stream=stream
   stb_load_image_from_user_stream( Varptr context,Callback )
End

Struct Context

Field stream:Stream

End

Function Callback( context:Void Ptr )

Local stream:=Cast<Context Ptr>( context )->stream

...got the mx2 stream...

End

Function LoadPixmap(...)

...open stream etc...

Local context:Context

context.stream=stream

stb_load_image_from_user_stream( Varptr context,Callback )

End

Casting object to void ptr may happen eventually, but it’s potentially dangerous for a number of reasons. I like having the struct wrapper as it ensures the object is wrapped in a variable and will be kept ‘alive’ while the callback is happening and not inadvertantly collected, similar to sticking something in a global to keep it alive.

June 26, 2016 at 4:43 am #1236

Simon Armstrong

Participant

Cool, am slowly getting the hang of structs.

Yup, got threading and debugger acting in most arbitrary manner. To be honest, it was nice to be back in unsafe environment:)

The actual call back needs to do a mem copy and increment a pointer so I will move that to a cpp file.

Or yes, it could block and wait for a monkey2 signal to say the copy has been performed on main thread…

Also, without a NoDebug flag like BlitzMax I think rule seems to be all monkey2 code should be app thread only no matter what.

I aim to implement a freeaudio style mixer in monkey2 code using float samples and just poll it per update to keep write pointer ahead of read pointer on some shared memory.

June 26, 2016 at 6:02 am #1237

Mark Sibly

Keymaster

> Or yes, it could block and wait for a monkey2 signal to say the copy has been performed on main thread…

Not quite sure what you mean here, but the native callback should never block. It should just copy in audio data, signal the gui/mx2 thread that it needs more data, and return ASAP.

In a polled scenario, the mx2 polling code could just check a flag, but sync signaling is nicer – see native process.cpp, esp:

int callback=g_mojo_app_AppInstance_AddAsyncCallback( finished );

…and…

postEvent( callback|INVOKE|REMOVE );

AddAsyncCallback is called on the main thread at setup time, while postEvent is called by native code running in it’s own thread.

Here, ‘finished’ is an mx2 function that gets called when the process is finished.

For an audio mixer, ‘finished’ would be something like ‘fillBuffer’ and postEvent would be called once native code has copied out the last buffer. You could in fact double buffer here (on the native thread) and call postEvent after flipping buffers but before copying audio data. This would help allow native/mx2 threads to run concurrently.

June 27, 2016 at 9:05 pm #1256

Simon Armstrong

Participant

I am running with fragment size of 32 samples on Mac in release mode and in audio heaven.

A vsynth banana featuring 5 types of oscillator, an ADSR envelope and arpeggiator from unknown universe looks to be this weeks project.

June 28, 2016 at 1:05 am #1257

Simon Armstrong

Participant

I have written following C++ class so vsynth.monkey2 can hopefully be thread safe.

It seems to be working and hooking up C++ to monkey2 felt pretty darn cool.

#pragma once
#include <deque>
#include <mutex>

typedef double Sample;

class AudioPipe{

	std::mutex mutex;
	std::deque<Sample> buffer;

public:
	int readPointer=0;
	int writePointer=0;
		
	void WriteSamples(Sample *samples,int count){	
		mutex.lock();
		for(int i=0;i<count;i++){
			buffer.push_back(samples[i]);
		}
		writePointer+=count;
		mutex.unlock();
	}
	
	void readSamples(short *dest, int sampleCount){
		mutex.lock();	
		int available=buffer.size();
		if (available>=sampleCount){
			for(int i=0;i<sampleCount;i++){
				Sample s=buffer.front();
				buffer.pop_front();			
				dest[i]=32767*std::max(-1.0, std::min(s, 1.0));
			}
			readPointer+=sampleCount;
		}
		mutex.unlock();
	}

	static void Callback(void *a, unsigned char *b, int c){
		memset(b,0,c);
		auto pipe=(AudioPipe*)a;		
		int sampleCount=c/2;
		short *dest=(short *)b;
		pipe->readSamples(dest,sampleCount);
	}
	
	void *Handle(){
		return (void *)this;
	}

	static AudioPipe *Create(){
		return new AudioPipe();
	}
};

#pragma once

#include <deque>

#include <mutex>

typedef double Sample;

class AudioPipe{

std::mutex mutex;

std::deque<Sample> buffer;

public:

int readPointer=0;

int writePointer=0;

void WriteSamples(Sample *samples,int count){

mutex.lock();

for(int i=0;i<count;i++){

buffer.push_back(samples[i]);

}

writePointer+=count;

mutex.unlock();

}

void readSamples(short *dest, int sampleCount){

mutex.lock();

int available=buffer.size();

if (available>=sampleCount){

for(int i=0;i<sampleCount;i++){

Sample s=buffer.front();

buffer.pop_front();

dest[i]=32767*std::max(-1.0, std::min(s, 1.0));

}

readPointer+=sampleCount;

}

mutex.unlock();

}

static void Callback(void *a, unsigned char *b, int c){

memset(b,0,c);

auto pipe=(AudioPipe*)a;

int sampleCount=c/2;

short *dest=(short *)b;

pipe->readSamples(dest,sampleCount);

}

void *Handle(){

return (void *)this;

}

static AudioPipe *Create(){

return new AudioPipe();

}

};

As a possible optimisation I would like to change the deque to be fragments (const sized array of doubles) but not sure if that makes sense.

June 28, 2016 at 2:58 am #1259

Mark Sibly

Keymaster

That’s the easy part – the syncing is where it gets tricky!

App code can’t just WriteSamples at will, it will need to halt to allow ReadSamples to catch up or the buffer will get ahead. On the other hand, if app code can’t WriteSamples fast enough, well yer basically hosed…

IMO, the best thing to do is start with getting the Callback moved to the monkey side, via either polling or signalling, so the audio device pulls data from SDL code running on it’s own audio thread, which pulls data from mx2 code running on the GUI thread.

Then you could perhaps look at using a fiber to provide synchronous/blocking writes to audio. This way, the fiber can block, not the callback.

And it’s probably time to add Deque I guess…

Just some thoughts!

June 28, 2016 at 4:49 am #1260

Simon Armstrong

Participant

I use a mutex to stay thread safe and it is up to monkey2 app to keep the buffer fill.

This is my current polling technique which I call every render:

	Method UpdateAudio()
		While True
			Local buffered:=audioPipe.writePointer-audioPipe.readPointer
			If buffered>=WriteAhead Exit
			Local samples:=FragmentSize
			Local buffer:=vsynth.FillAudioBuffer(samples)			
			Local pointer:=Varptr buffer[0]
			audioPipe.WriteSamples(pointer,samples*2)
		Wend
	End

Method UpdateAudio()

While True

Local buffered:=audioPipe.writePointer-audioPipe.readPointer

If buffered>=WriteAhead Exit

Local samples:=FragmentSize

Local buffer:=vsynth.FillAudioBuffer(samples)

Local pointer:=Varptr buffer[0]

audioPipe.WriteSamples(pointer,samples*2)

Wend

End

June 28, 2016 at 5:56 am #1261

Mark Sibly

Keymaster

Ok, that looks like it’s basically just moving fillBuffer from SDL->mx2 which should work.

It seems a little long-winded to me though – how about something like:

// c++ 
class AudioStream{

	float *WriteBuffer(){
		return _data[_buf^1];
	}

	private:

	float _data[2][MAX_SAMPLES];
	std::atomic_int _buf=0;

	void AsyncCallback( ubyte *stream,int len ){
		_buf^=1;   //effectively signals mx2 thread...
		for( int i=0;i<len;++i ){
			*stream++=Blah( _data[_buf][i] );   //convert data...
		}
	}
};

// mx2
Global stream:AudioStream

Function UpdateAudio()
	Global _buf:Float ptr
	local buf:=stream.WriteBuffer()
	If buf=_buf return
	_buf=buf
	vsynth.FillBuffer( buf )
End

// c++

class AudioStream{

float *WriteBuffer(){

return _data[_buf^1];

}

private:

float _data[2][MAX_SAMPLES];

std::atomic_int _buf=0;

void AsyncCallback( ubyte *stream,int len ){

_buf^=1; //effectively signals mx2 thread...

for( int i=0;i<len;++i ){

*stream++=Blah( _data[_buf][i] ); //convert data...

}

};

// mx2

Global stream:AudioStream

Function UpdateAudio()

Global _buf:Float ptr

local buf:=stream.WriteBuffer()

If buf=_buf return

_buf=buf

vsynth.FillBuffer( buf )

End

I guess it depends on whether it needs to be able to handle variable sized chunks etc – my code assumes a fixed size but I still feel like yours does more copying than it needs to.

June 28, 2016 at 8:44 am #1262

Simon Armstrong

Participant

Samples per frame is 44100/60 = 735 but typical audio buffers need to be power of two so I am thinking for simple ping pong buffers fragment size of 2048 should be safe bet for uninterrupted polling per OnRender.

How do Monkey2 timers work? Do they interrupt code or fire on app update?

June 28, 2016 at 9:40 am #1263

Mark Sibly

Keymaster

> How do Monkey2 timers work? Do they interrupt code or fire on app update?

Timers use the same mechanism as process.finished, process.stdoutready etc mentioned above.

When the timer fires, it’s handled in timer thread which basically just sticks a custom event in the SDL event queue (which can be done safely from another thread), which will cause the callback to be invoked the next time events are updated.

The nice thing is that posting the event will also wake up a WaitMsg blocked app so you don’t have to poll.

June 29, 2016 at 1:15 am #1275

Simon Armstrong

Participant

The vsynth feature set is growing including noise wave and pitch bend.

Attachments:

vsynth1.png

July 1, 2016 at 12:54 am #1449

DruggedBunny

Participant

vsynth is vcool!

September 8, 2016 at 7:31 am #3759

Diffrenzy

Keymaster

This is so cool.

Will it work on Android and iOS?

September 9, 2016 at 5:40 am #3797

Mark Sibly

Keymaster

Will it work on Android and iOS?

I’m not sure about iOS but, somewhat to my surprise, it does work on Android! Both the sdl-mixer and openal drivers too (had to increase the ‘polling’ rate of the openal one though).

I haven’t got bluetooth keyboard working on iOS yet so I haven’t really tested it there.

Author

Posts

Viewing 15 posts - 1 through 15 (of 16 total)

1 2 →

You must be logged in to reply to this topic.

Streaming Audio

Attachments:

Archives

Recent Replies

What is this?