Orx - Portable Game Engine
Welcome, Guest
Please Login or Register.    Lost Password?
Sound recording support for Orx
(1 viewing) 1 Guest
Go to bottom
TOPIC: Sound recording support for Orx
#1217
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Sound recording support for Orx 6 Years, 6 Months ago Karma: 2
Hey
for the game I'm currently developing I need sound recordind support. In the current state orx does not support this. however it is based on OpenAL, therefore it should be rather easy to implement it. I would like to implement this feature into the engine.
In order to be usefull for others too, I would like to discuss possible orx API extensions, that are independent of OpenAL, so that it could possibly work with other underlying frameworks, too.

I'm not that familiar with both orx and OpenAL.

What I though of:

We need an additional object, different than orxSOUND, like orxSOUND_CAPTURE.

This object has properties like the audio format(bits + mono/steroe) and frequency.

Also it might be stupid to have capture objects. you usually don't capture from multiple sources at the same time.

It might be better to just have an function orxSOUND_START_CAPTURE(FORMAT, FREQUENCY, BUFFERSIZE)
and
orxSOUND_STOP_CAPTURE(FORMAT, FREQUENCY)

after that orxSOUND_CAPTURE_EVENTs would be created. The payload would contain the audio data in the specified FORMAT. (BUFFERSIZE bytes at maximum)

for the actual capturing the default device on the system would be used.

then it might be usefull to add some functionality to playback those samples directly, without saving them in a file.

any suggestions?
 
Logged Logged
  The administrator has disabled public write access.
#1229
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 6 Months ago Karma: 71
Hi!

Sorry not to have replied sooner but I wanted to take some time to check orx's current sound API and see how this feature could be integrated at best.

I think your proposition makes sense and I would just like to discuss the actual function names. Let me know if what I say makes sense to you.

Right now, a sound buffer in memory is handled by the orxSoundSystem API and is called orxSOUNDSYSTEM_SAMPLE. When you're done capturing your data, that's what we should end up with. We should also add a orxSoundSystem_Save(orxSOUNDSYSTEM_SAMPLE *, const orxSTRING) to save t on disk.

Based on the orxSOUNDSYSTEM_SAMPLE, we can create an orxSOUND (unified structure that can be played through the high level orxSOUND API). Right now the creation of an orxSOUND from an orxSOUNDSYSTEM_SAMPLE is private and hidden in orxSound.c but I can expose something like orxSound_CreateFromSample(const orxSOUNDSYSTEM_SAMPLE *) that would do the trick.

Now for the recording itself, I think you're right, we need two functions, something like:
orxSOUNDSYSTEM_SAMPLE * orxSoundSystem_StartSampleRecording(const orxSTRING _zName, orxSOUNDSYSTEM_FORMAT *_pstFormat)
orxSTATUS orxSoundSystem_StopSampleRecording(orxSOUNDSYSTEM_SAMPLE *).

In this case orxSOUNDSYSTEM_FORMAT would be a public structure with direct access to rate, channels, etc.

My only concern with that is that the orxSOUNDSYSTEM_SAMPLE * should be internally flagged so as to not be useable by any other function as long as StopSampleRecording hasn't been called on it. And possibly have to add a query function such as orxSoundSystem_IsSampleRecording(const orxSOUNDSYSTEM_SAMPLE *). To solve this particular issue, we could go with another structure such as your orxSOUND_CAPTURE, but that means we should then provide something to delete it and also it won't be explicit that the StopSampleRecording would delete such an intermediate structure. I tend to prefer the first solution (keeping the number of structures low so as to make it as simple as possible) but it's not a strong preference either.

To sum it up, I'd like the capture/record interface at the orxSoundSystem level (including the events) and only the possibility to create a playable orxSOUND from the result at the orxSound level. What do you think?
If you agree to go this way, I can easily take care of creating the orxSOUND bridge and all the plugin hooks, simply leaving empty functions for you to fill up in the plugin itself. Let me know what you think!

EDIT: Sorry I made a few post-send editions so you might have to re-read this message again as it may differ from the email notification you received.
 
Logged Logged
 
Last Edit: 2010/10/28 05:30 By iarwain.
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1231
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 6 Months ago Karma: 2
sounds good to me. I agree that it's probably better to not add another structure.(orxSOUND_CAPTURE)

I'm now going to have a deeper look at the current implementation of the orx sound system.
 
Logged Logged
  The administrator has disabled public write access.
#1232
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
I'm currently struggling a bit with the build system of the orx engine itself.

I use Codelite as an IDE on a Windows 7 machine.

I'm a bit confused because you have both a SFML and a OpenAL sound plugin. which is actually used? where can I see this?

I'd guess the SFML is used. As far as I know SFML itself uses OpenAL for audio related stuff.

The CodeLite workspace has only a project for the SFML soundsystem. however this one is not built when I build the whole workspace.

however at the end everything is working...I'm a bit confused right know.

...maybe it will be clearer tomorrow^^
 
Logged Logged
  The administrator has disabled public write access.
#1233
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 71
Yep, it might not be obvious at first sight.

You can read the REAMDE file packed with orx that explains a bit what's going on (especially the "Versions" section).

Using Codelite you have the option to compile for 2 platforms (win & linux), 2 link types (static & dynamic), 2 modes (embedded & non-embedded) and 2 levels of optimization (debug & release).

That's a lot of combinations! ^^

I'm assuming the only one that doesn't look obvious is the mode (embedded/non-embedded), am I right?

Well, in non-embedded mode, you'll end up with a library for orx that only contains the core but no platform dependent code (mainly I/O). The plugins will then be loaded at runtim and you can decide which ones you want to load from command line or config (SFML, SDL, GLFW, etc...).

In embedded mode, you lose that flexibility but you gain in ease of use and speed: a given set of plugins will be compiled and embedded in orx library at compile time. The defaults plugins are currently the GLFW ones (orx 1.2+) for computers and there's only one set available for iPhone/iPad.

The list of embedded plugins is defined in src/plugins/orxPlugin_EmbeddedList.cpp


Hope this helps a bit! =)
 
Logged Logged
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1234
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
ok great, that helped alot, now I get it.

it was confusing, that the sources were not part of the codelite project. I didn't expect them to be #included.

so basically the nonembedded OpenAL plugin is missing from the codelite projects.

now I'll have a deeper look at the sound system code.
 
Logged Logged
  The administrator has disabled public write access.
#1235
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 71
Oh, thanks for the info!
I'll fix that when I'll get my computer back. I don't package the non-embedded version for releases, so I'm not testing them often (if at all for over a year! ^^).
 
Logged Logged
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1262
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
I'm currently digging into both the orxSoundSystem implementation and OpenAL programming.

So one orxSOUNDSYSTEM_SAMPLE contains a single chunk of captured audio data, right?(the size depending on some predefined buffer size)
So the sample that orxSoundSystem_StartSampleRecording returns will be filled over and over again?
How would that work, when just having a single one?

I thought of something like this:
Code:

orxSTATUS orxSoundSystem_StartSampleRecording(const orxSTRING _zName, orxSOUNDSYSTEM_FORMAT *_pstFormat)
orxSTATUS orxSoundSystem_StopSampleRecording(orxSOUNDSYSTEM_SAMPLE *)


orxSoundSystem_StartSampleRecording just initiates the capturing and doesn't return anything.

Each time a new sample is captured either some SOUND_CAPTURE event is triggered or some callback function called.

Now the user does whatever he wants with the sample and then deletes it.

so what are possible use cases for the sample data?

  • Forwarding the samples to some audio processing library(this is what I'm intending to do)

  • Streaming the samples to a file

  • Playing the samples as a stream



maybe I understood the orxSOUNDSYSTEM_SAMPLE wrong. but saving just one sample to a file is not really usefull, is it?

what do you think?
 
Logged Logged
  The administrator has disabled public write access.
#1264
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 71
tdomhan wrote:
I'm currently digging into both the orxSoundSystem implementation and OpenAL programming.

So one orxSOUNDSYSTEM_SAMPLE contains a single chunk of captured audio data, right?(the size depending on some predefined buffer size)


That's right. More precisely orxSOUNDSYSTEM_SAMPLE is simply a wrapper for the actual structure used by the underlying library. For the OpenAL plugin it holds the buffer ID (works the same way than texture IDs in OpenGL, for example) as long as the duration which can't be easily retrieved through OpenAL.

So the sample that orxSoundSystem_StartSampleRecording returns will be filled over and over again?
How would that work, when just having a single one?


I'm not sure how OpenAL works for recording, but when the recording is done, your whole sample should be in memory. If OpenAL allows to record directly do disk we can add an additional method such as orxSoundSystem_StartStreamRecording.

When the recording is over, we simply create an OpenAL buffer based on the sample, we wrap it in an orxSOUNDSYSTEM_SAMPLE and we can now use it the same way we use all others. Does it sound ok to you?

I thought of something like this:
Code:

orxSTATUS orxSoundSystem_StartSampleRecording(const orxSTRING _zName, orxSOUNDSYSTEM_FORMAT *_pstFormat)
orxSTATUS orxSoundSystem_StopSampleRecording(orxSOUNDSYSTEM_SAMPLE *)


orxSoundSystem_StartSampleRecording just initiates the capturing and doesn't return anything.


Oh yes, I don't know why, I assumed we might want more than one recording at a time which is totally silly, my bad! This option is far better. In this case I'd rather have something like:

Code:


orxSTATUS orxSoundSystem_Record(const orxSTRING _zName, const orxSOUNDSYSTEM_FORMAT *_pstFormat);
orxSOUNDSYSTEM_SAMPLE *orxSoundSystem_CreateSampleFromRecord();



We can put in the format if we want to record to memory or to disc, if this is supported by OpenAL of course. I have no idea. =)

Each time a new sample is captured either some SOUND_CAPTURE event is triggered or some callback function called.

I like the event as it'll stay consistent with the remaining for orx API. The event would be, I guess, orxSOUND_EVENT_RECORD_START, orxSOUND_EVENT_RECORD_END, and the orxSOUND_EVENT_PAYLOAD will have to be updated to contain the relevant info.

Now the user does whatever he wants with the sample and then deletes it.

so what are possible use cases for the sample data?

  • Forwarding the samples to some audio processing library(this is what I'm intending to do)

  • Streaming the samples to a file

  • Playing the samples as a stream



maybe I understood the orxSOUNDSYSTEM_SAMPLE wrong. but saving just one sample to a file is not really usefull, is it?

what do you think?


Well, if OpenAL allows to record a stream to a file, the user can choose that at the beginning. If the user choose to record to memory, he can then either save it or play it directly. Again, I'm not sure what's available for saving as i didn't look too much into that. I guess libsndfile has save support but I'd be surprised if stb_vorbis had one.

Or for now we can simply focus on just recording to memory, get a sample out of it when it's done and see later for saving/streaming to disc?

Just to sum it up, the orxSOUNDSYSTEM_SAMPLE would be created at the very end from the memory buffer that would have been recorded. But again, I need to check OpenAL as I don't know if it can do dynamic recording or if we have to give it a max size before starting to record.
 
Logged Logged
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1265
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 71
After a quick look at OpenAL/libsndfile, recording to disc shouldn't be difficult at all.

However for recording to memory, we can actually support non-bound recording and allocate new buffers on the fly, when needed. OpenAL can then create a playable buffer from all those recorded memory buffers.

I'm not sure if I'll have time tonight, but I'll try to prepare placeholders in orx's API for you to fill. I'll create a branch in the svn repository and can give you write access if you give me your sourceforge user name. Let me know.
 
Logged Logged
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1266
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
my sourceforge user name is: tdomhan.

Code:

When the recording is over, we simply create an OpenAL buffer based on the sample, we wrap it in an orxSOUNDSYSTEM_SAMPLE and we can now use it the same way we use all others. Does it sound ok to you?

What do you mean by "when the recording is over"? After orxSoundSystem_StopSampleRecording is called?


After having some thought about the whole thing, I think it's better to do it like to music playback.
This means introducing some generic stream interface. Now the SoundSystem will be able to fetch it's audio data from this stream. one possible implementation would be a filestream.

on the other hand the capturing code would be able to write the audio data continuosly to a output stream. this output stream could either be a file stream or forward it to an audio library. Or add the captured samples to an FIFO. the playback code could then access this fifo via an input stream interface.

I hope it's somehow clear what I mean and it wouldn't be like using a sledge-hammer to crack a nut oO
 
Logged Logged
  The administrator has disabled public write access.
#1267
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 71
Yes, I meant when StopRecording is called. By then we have collection of memory buffers holding the sound and they can be either saved or "transformed" to an orxSOUNDSYSTEM_SAMPLE.

Let's go back a bit, what are the features you need. I'd like to keep the API as simple and portable as possible as orx will eventually have to support it on other platforms such as iPhone.

As I see it, here's a list of the needed features:
- record for a variable length of time (ie. the time doesn't have to be specified at the beginning of the recording)
- Reuse that recorded sound either as a sample (everything is in memory) or as a stream (from disc). This is generally known before recording.

(Also note that a sample can be created from disc if needed.)

So my guess would be to have functions for asking for recording (either to memory (sample) or to disc (stream)), stop recording (sample -> orxSOUNDSYSTEM_SAMPLE is created and memory freed, disc -> file is finalized) and make the sound accessible for the rest of the API (play, pause, pitch, etc...). Do we need anything else?

Do we need general streaming, ie. play the sound that we are currently recording? If so we can send events for every recorded packet which allows the user to inspect its content, stop recording if it's blank or even altering it. This is very close to what I understood of your streaming view, with a non-intrusive approach, I guess.
 
Logged Logged
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1268
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
well I need a continuous stream of captured audio data for an infinite/unspecified amount of time. this means I'd like to get samples frequently and not at the and of the recording.
I do not need to save the audio data to a file or play it back. However I thought someone else might find this useful.


Do we need general streaming, ie. play the sound that we are currently recording?
I don't need that. And as orx does not support networking I can't think of a usecase right now. However if networking is something to be added in the future it might become relevant.

having an event for each captured sample would fulfill my needs.
It would also be fine for writing it to a file.

However it would not be the best option if someone wants to playback the captured sample directly.
 
Logged Logged
 
Last Edit: 2010/11/01 20:51 By tdomhan.
  The administrator has disabled public write access.
#1269
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 71
I guess playing what is currently recording might be interesting for adding effects, such as voice distorsion, but isn't something traditional games need, agreed.

In this case I guess we can start recording with 3 options: discard, store in memory, store on disc. You get notified of captured packets through events anyway, the packet size and the mode of captures being defined in the orxSOUNDSYSTEM_FORMAT when starting recording, which should probably be renamed to orxSOUNDSYSTEM_RECORD_INFO?

The length of packets can differ from the internal buffer used for recording of course.

As for VoIP, we'll have time to see how it works, I'd be more concerned about object replication first.

This way the API stays minimal, what do you think?
 
Logged Logged
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1270
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
yep KISS is always a good thing.

so how about this:
Code:

orxSTATUS orxSoundSystem_StartSampleRecording(const orxSTRING _zName, orxSOUNDSYSTEM_FORMAT *_pstFormat)

This starts the capturing. The format/info does not contain any information on what to do with the captured data, this means if it should be stored in memory or written to a file.
So now a orxSOUND_EVENT_RECORD_START will be generated. followed by continuos orxSOUND_EVENT_RECORD_SAMPLE events. Those will have the audio data inside the payload.

Once
Code:

orxSTATUS orxSoundSystem_StopSampleRecording(void)

is called, orxSOUND_EVENT_RECORD_END will be fired.

Ok this should be the whole, minimalistic capturing API.

Now based on this API we could add some sound recorder, which catches the events opens a file(in case of an file recorder), writes the data to the file and closes the file after the recording ended.

additionally we could add an in-memory recorder, which does the same, but writes the data to some place in the memory.

What do you think about that?
 
Logged Logged
 
Last Edit: 2010/11/01 21:12 By tdomhan.
  The administrator has disabled public write access.
#1271
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 71
Sounds good to me!

I would simply rename the sample part as it might be confusing with actual sample (which are memory sound meant to be played).

I'd go with:

Code:


orxSTATUS orxSoundSystem_StartRecording(const orxSTRING _zName, const orxSOUNDSYSTEM_RECORD_INFO *_pstInfo); <= This will contain format and can be easily extended
orxSTATUS orxSoundSystem_StopRecording();
orxSOUNDSYSTEM_EVENT_RECORD_START,
orxSOUNDSYSTEM_EVENT_RECORD_STOP,
orxSOUNDSYSTEM_EVENT_RECORD_PACKET



Should we merge the events of orxSOUNDSYSTEM and orxSOUND together, and maybe add a convenience wrapper in the orxSOUND module so that users don't have to call directly orxSOUNDSYSTEM_* functions?

Not sure about that, it was just to get more consistency.
 
Logged Logged
 
Last Edit: 2010/11/01 21:52 By iarwain.
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1272
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
Code:

I would simply rename the sample part as it might be confusing with actual sample (which are memory sound meant to be played).

ok.

Should we merge the events of orxSOUNDSYSTEM and orxSOUND together, and maybe add a convenience wrapper in the orxSOUND module so that users don't have to call directly orxSOUNDSYSTEM_* functions?
yep, that would probably be good.
 
Logged Logged
  The administrator has disabled public write access.
#1274
iarwain
Administrator
Posts: 3107
graph
User Offline Click here to see the profile of this user
Gender: Male Orx - Portable Game Engine Location: Montreal
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 71
I granted you acces to the SVN and created a branch for the sound integration here: https://orx.svn.sourceforge.net/svnroot/...nches/SoundRecording

I'll try to work a bit on the API right now so that you won't need to learn how the plugin module works.
 
Logged Logged
 
Last Edit: 2010/11/02 06:12 By iarwain.
 
Want to follow orx's development? Go subscribe to the orx-dev group at: https://groups.google.com/forum/?fromgroups#!forum/orx-dev
  The administrator has disabled public write access.
#1275
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
ok, then I can start working.


I think we need another function that checks if recording is available. It might not either if the plugin doesn't support it or there is no capturing hardware:
Code:

orxBOOL orxSoundSystem_RecordingAvailable();



Currently I'm thinking about how do handle the buffers that contain the audio samples in the event payload.
It's probably a bad idea to allocate them dynamically and let the user care about deletion. the user might not have an event handle defined...this would result in a memory leak. and constantly allocating memory is not that great performance wise either.

We could use double/tripple buffering. then the user has to make sure that he has processed/copied the sample until the next event occurs.
 
Logged Logged
  The administrator has disabled public write access.
#1276
tdomhan
Expert Boarder
Posts: 132
graphgraph
User Offline Click here to see the profile of this user
Location: Tampere, Finland
Re:Sound recording support for Orx 6 Years, 5 Months ago Karma: 2
Ok I have thought about how to do the buffering.

First the orxSOUNDSYSTEM_RECORD_INFO contains the frequency at which audio data will be fetched from the audio device. Furthermore it contains the maximum buffer size.

I think it's best to do double buffering. So once the first buffer has been filled this buffer will not be touched, until the user calls some orxSOUNDSYSTEM_RECORD_SWAP_BUFFERS method. This way the data will not be messed up while the user is still processing/copying the audio data.

If the user can't process the audio data fast enough, this means orxSOUNDSYSTEM_RECORD_SWAP_BUFFERS has not been called, when the next fetching has been triggered, we will just drop some audio samples.

the downside is that only *one* sound capturing event listener should be attached.


pro:
buffer will stay consistent while being processed by the user
if the user can't cope with the update frequency samples will just be dropped -> no delay

con:
only one listener
 
Logged Logged
  The administrator has disabled public write access.
Go to top