Community Core Audio (Alpha)

GSoC 2010 has come to the end, so I release the current version here as the achievement during this year's GSoC.
What's New in this release:
- Can recognize most English words instead of digits only.
- Added free-speaking mode(not very accuracy, so using commandpicking mode is recommend).
- Output to network by TCP.
- OSX and Linux support.

You can download CCA executable binary for your platform:
OSX : http://nuicode.com/attachments/download/177/CCA_GSoCFinal_bin_OSX.zip
Linux(32bit) : http://nuicode.com/attachments/download/179/CCA_GSoCFinal_bin_linux32.tar.gz
Windows(32bit) : http://nuicode.com/attachments/download/178/CCA_GSoCFinal_bin_win32.zip

Or check out from svn: svn checkout http://nuicode.svnrepository.com/svn/cca

ofxASR is a part of CCA, but it is a stand alone openframeworks addon.

ofxASR is host at http://code.google.com/p/ofxasr/.

And here is a thread about it at openframeworks forum: http://www.openframeworks.cc/forum/viewtopic.php?f=10&t=4202

Note that this is not CCA's "Final" version. The developing of CCA will never stop.

I'm proud of CCA. I'm proud of being a contributor of NUI Group. I will keep developing and maintaining CCA, make it stronger and stronger, and let it be downloaded by millions one day.

Thanks our NUI Group, thank you my mentor tito and Christian, thanks all people that once gave me advices and feedbacks. What a good summer!

-------------------------------------------------------------------------------
TODOs in future:
- A GUI sentence list editor. (Currently user need to edit the config file manually)
- A setting dialog.
- Automatic stop recoding(do not need press stop button by users).
- Other required features from users.

As be required by many folks, CCA have successfully compiled under Mac OS X. An advantage of OSX is that applications can be packaged as a bundle. All resources are in a single app bundle showed as the screenshot below:

The config.xml and the list of recognizable sentences are also in the bundle. Users have to use "Show Package Contents" to view and edit them.

Another good news is that CCA can recognize not only digits but also any English word and sentence now, as the following screenshot:

As seen in the screenshot, CCA is still under CommandPicking mode. That means you must edit the file Command.list in the bundle to specify which sentences can be recognized by CCA.

The FreeSpeaking mode is also available now. But this mode is very slow(may be it need a progress bar) and not accuracy. So the CommandPicking mode is recommended. In future(after the gsoc) I will do something to make the FreeSpeaking mode more useful.

The newest CCA for OSX can download at: [[http://nuicode.com/attachments/download/174/CCA_OSX_20100722.zip]]

Hi all,

CCA has released its first preview version for OS X. It was packaged as an OS X sytle application bundle. You can download it here: http://nuicode.com/attachments/download/171/CCA_OSX_20100721.zip

The usage is the same as the windows version, please check http://nuigroup.com/log/cca_preview_release/ to learn how to use it.

Unfortuntly it is very inaccurate, that means it usually gives wrong result under OS X, while the windows version is very accurate. I'm not sure if there's something wrong in my code. I guess this problem is because there might be some differences between Mac and PC's hardware. The codes are tracked at http://nuicode.com/projects/cca-alpha/repository, it will be great if anyone can fix it.

Another bug known: The application's name and its path name should not contain any space, or it may crash. This bug is from Sphinx and I will report it to Sphinx Team.

Introduction

Community Core Audio(CCA) is a GSoC Project that is has the similar GUI to CCV. CCA manages voice inputs, converts voice to text, and outputs messages to network.

Now The preview release of CCA (for windows) was available to download at: http://nuicode.com/attachments/download/169/cca_20070707.zip

Below is a screen shot:

-------------------------------------------------------------------------------

Usage¶

The current version only support command-picking mode. So just do not click the "FREE SPEAKING MODE" button in this preview version. :)
For detail of these two modes, please read: http://nuicode.com/documents/75

Select the check box "RECORD SOUND" to start recording. The waveform will be showed at the viewer window dynamically.

Unselect the check box "RECORD SOUND" or Click the "STOP" button to stop recording.

Select the check box "PLAY/PAUSE" to play, unselect it to pause. Click the "STOP" button to stop playing.

After recoding a audio, click the "SENT TO RECOGNIZE ENGINE", and the output viewer will display the sentence you just record.
The current version only support English digits because of the simple sphinx resources.

You can click the "CLEAR SCREEN" button to clear the output viewer.

-------------------------------------------------------------------------------

Configuration¶

For normally use, you do not need to do any configuration, what you need is just download and run it. However, CCA provide some options through config files.

The most important config file is $cca_path/data/config.xml. If you want to use new sphinx resources, you must specify the path of new resource files in this XML file. To learn about resource files, please read: http://nuicode.com/documents/74

The input audio sample rate was also set in config.xml. The input sample rate must be same as the sample rate of the Acoustics Model (AM). AM is a part of the resource files.

The file $cca_path/data/commandList.txt is for CommandPicking mode. See this document: http://nuicode.com/documents/75

-------------------------------------------------------------------------------

Some Technical Detail¶

A stand alone oF addon for speech recognition, ofxASR, was released for days. ofxASR is the core engine of CCA, and it can be applied on any oF application. Currently it use CMU Sphinx3 as its Automatic Speech Recognition(ASR) engine, but it also designed allowing to use other ASR engine, such as Microsoft's SAPI or Mac OSX Speech API. All engine share the same interface.

ofxASR is hosted at: http://code.google.com/p/ofxasr/

A class ofRectPrint was developed to print lines of string in a rectangle with auto scroll and scroll up/down.

-------------------------------------------------------------------------------

TODO¶

• Ship better sphinx resources that support any English words instead of digits.
• The free-speaking mode.
• Output to network.
• OSX and Linux support.

Very glad to announce that CCA has have its basic functions.
Now it can record a sound, and convert it to text.

For convenient of debug, I use a very simple Acoustic Model that can only recognize a single english digit, and the converted text was not displayed in the view window but print directly in windows cmd, as the screenshot as below:

Download the preview version to try this program (for windows): http://nuicode.com/attachments/download/164/cca_20100617.zip

Usage:
Click "RECORD" to record, and click the "SENT TO RECOGNIZE ENGINE", the converted text will printed in cmd.
Note the current version can only recognize a single english digit (one, two, ... nine), and the FREE SPEAKING MODE is unsupported now.

Now CCA have has its basic GUI, can record and play. The Windows version and the OSX version is developing synchronous.

I spent some time to compile sphinx3, explore it and wrote a hello world program. You can download it here: http://nuicode.com/attachments/download/160/SimpleSphinxDecoder_win32.zip

This helloworld is a vs2008 project, includes compiled sphinx3's lib and dll. As I described in my GSoC proposal, CCA will support two mode: free speaking mode and command picking mode. In this hello world the two mode is supported, but you must modify the config file to select the mode you wanted. That need a little knowledge of ASR, if you have no idea of how to do it, just ask me in IRC or skype. :)

BTW, compared with HTK, Sphinx is so complicated when you want to use a custom grammar instead of the default word graph.

I wrote a hello-world to show how to recognize a digit using sphinx3.

You can download it here:
http://nuicode.com/attachments/download/160/SimpleSphinxDecoder_win32.zip

I'm glad to talk with you around speech recognition techniques.

Now CCA can record sound and play.

Select the check box "RECORD SOUND" to start recording. The waveform will be showed at the viewer window dynamically.

Unselect the check box "RECORD SOUND" or Click the "STOP" button to stop recording.

Select the check box "PLAY/PAUSE" to play, unselect it to pause. Click the "STOP" button to stop playing.

Problems:
The "READ SOUND FILE" is for load a sound file instead of recording. But how to call operating system's "Open File" dialog in oF?