Members | Sign In
Legacy MOVI User Community Forum (readonly) > MOVI Question & Answers
avatar

MOVI very aggressive about matching utterances to sentence vocabulary

posted Apr 07, 2016 03:21:11 by Dylan
Hello again,

I've just started trying to program my MOVI to select various "diagnostic modes" in my Arduino project. So far I have found that, after receiving the call sign, MOVI is very "aggressive" about matching any further utterance to words trained in the "addSentence" commands in setup(). For example, with only two sentences trained: 1) "spark advance" and 2) "solenoids", MOVI will think I said "advance" when calling getResult() after a return code of RAW_WORDS from poll() if I say a word like "nothing" or "cancel", and the +'ve value I get from poll() is the sentence number for "spark advance". The false positive rate I am getting with this behaviour is undesirable.

I did try the "trick" in the WordSpotter example where one trains a "background model" with a bunch of sentences that contain frequently used English words, but then I get the opposite problem: even if I clearly say "spark advance", MOVI thinks I said some combination of the background model words instead. This only seems to serve to flip the problem on its head where I get almost everything coming back false negative.

I am working in a very quiet "lab" environment with virtually no background noise. I hope to move MOVI into my car where there is going to be a lot more ambient noise around, but for now, I think I can safely say that I'm not suffering from background noise issues. I am a native English speaker; if anything I would think my "Canadian articulation" (not maritime) should be about as optimal as it gets.

I have another post with respect to the use of an external microphone - this is unrelated as the behaviour I am describing here is observed with MOVI's built in microphone.

Is this expected behaviour from MOVI that it would very aggressively try to match utterances to trained words? Any advise on improving the strength of the phonetic matching? Would it be better if I put together some code sketch examples to evaluate?

Dylan.
[Last edited Apr 07, 2016 03:24:21]
page   1 2 next last
22 replies
avatar
GeraldFriedland said Apr 08, 2016 04:23:19
As usual, the caveat is that speech recognition is not a solved problem and many tasks will require 'tinkering'. Which, in my humble opinion, makes speech recognition a perfect area for some fun maker projects... in anyways, MOVI, as indeed any speech recognizer, will always try to match everything it 'hears' to the expected words as a first step and the expected sentences as a second step. Also, if you haven't please definitely take a look at Chapter 4 of our user manual.

If you can, a code example would indeed help, as from your description, I am pretty sure these are not microphone problems. Without having seen your code: If you keep the background model individual words and make the things to recognize three to five word sentences, you will probably get way better results.

Hope that helped...
avatar
Dylan said Apr 08, 2016 06:33:21
Thanks again Gerald. I have tried to read the user manual in detail. My feedback would be that it does not provide a lot of strong guidance in this realm.

It is fun to tinker with it to a point, then without a great deal of transparency into how the mechanisms work and why decisions are being made, it becomes difficult to remain optimistic without any grasp on what to try next.

I have completed a sketch for my application that includes all the "diagnostic commands" that I desire for now. I have coded my own timeout between callsign to successful command recognition. I have added compensation for utterances heard before the callsign that end up being falsely interpreted as commands following the call sign by adjusting the threshold at particular times. I want to field test this a bit before sending any code sketches or asking anything further on this topic. I know for sure that I'm going to have some suggestions/feedback for you with respect to the lack of sentence listening timeouts, interpretation of utterances which precede the callsign, and CALLSIGN_DETECTED event not always preceding BEGIN_LISTEN, but it's too early just yet without some further validation.

Unfortunately this is now complicated by the fact that my application requires a reliable external microphone in a somewhat noisy environment at a distance of about 2 feet. I may need to break through that problem first, we will see.

Dylan.
avatar
GeraldFriedland said Apr 08, 2016 16:40:33
Dylan,

The reason why we can't give a lot of detail in the manual is because every case is different. It's like being a lawyer or a tax accountant: Certain standard rules apply but then every project is different. This is why companies pay hundreds of thousand of dollars to get a telephone dialog system running.

In anyways, I am looking forward to seeing some code when you are ready and I probably will have some ideas. I would _in general_ recommend not to use your own timeouts and not to tinker too much with the threshold. But again, it may make sense for you. In your case, however, it will definitely make sense to make sure your sentences are phonetically wide apart. For example, "turn the light on" and "turn the light off" are two sentences that are super close together as only "n" and "f" make the difference. Our standard example "let there be light" and "go dark" shows how to make them widely different. Then, if you can include context, you can help increase the accuracy. To stay with the light example, you could just use the sentence "switch the light" and have a bool variable store the state of the light, which cuts down on one sentence to (mis-)recognize.

Another tip is that the micdebug command (see microphone thread) is probably your friend and also looking at the raw results in the console will probably help a lot.

But again, this is general advise... hope it help anyways.
[Last edited Apr 08, 2016 16:43:40]
avatar
Dylan said Apr 08, 2016 20:39:56
My code is quite extensive because it contains all sorts of diagnostics and integrations, but the main loop() routine where all the MOVI interaction takes place isn't too bad. I need to solve my microphone problem before I'd feel confident in sharing the code.

Perhaps I can pose a couple of general scenarios for your comment in the meantime, in case it causes me to change course.

Timeout problem: I find that after the callsign is received, MOVI will wait indefinitely for my voice. Because of the aggressive matching I am experiencing, I wanted to have a 5 second timeout between callsign and sentence recognition. So after receiving the -140 from poll(), if MOVI is still waiting for me to say something, I do movi.stopDialog() followed by movi.restartDialog(). This has so far been successful for me to get MOVI to just go back to listening for the callsign after 5 seconds of hearing nothing. Reasonable?

Utterances preceding callsign problem: I'm actually using a Sparkfun MP3 shield to announce results of diagnostics triggered by MOVI commands. This predates me having MOVI - I was using the other Arduino voice recognition board. MOVI's voice is reasonably impressive, but the MP3 voices are that much more life like so I'd like to keep things this way. So the first time I say the callsign, MOVI waits up to 5 seconds (my custom timeout) for my first command, then the MP3 announces the results of the requested diagnostic. The second time I give MOVI the callsign, it responds with -140 and then IMMEDIATELY with -141 choosing one of the available sentences, usually a smaller/simpler one. What I did was to do setThreshold(95) before playing an MP3 voice and then setThreshold back to the default after the MP3 is finished playing. This seems to have solved the problem for now (though I'm worried that once I have a mic that I don't have to "eat" to make work with MOVI that even with a 95 threshold, MOVI may hear the MP3). I tried the stopDialog/restartDialog thing after playing the MP3 as I did for the timeout but it made no difference to the undesired behaviour. Reasonable? Is this expected behaviour that MOVI would cache utterances prior to the callsign and use those in sentence matching?

CALLSIGN_DETECTED not sent problem: A large percentage of the time, poll() doesn't return CALLSIGN_DETECTED prior to BEGIN_LISTEN. For this one you may need to see my code to comment. For the time being, I'm using BEGIN_LISTEN as an indication that the callsign was detected.

If it helps to get an idea of how my application is intended to work here is a video with the old voice recognition board:

https://www.youtube.com/watch?v=6Y2CrRenziA

There is no need to watch the whole video, you'll get the idea after the first 30 seconds or so. I was very lucky with that run by the way, this is not meant to say that I think the other board worked well at all. On the contrary, the other board I speak of is extremely unreliable in my experience. False positives rates were very frustrating. I am hopeful based on what I saw on the "bench" using MOVI's integrated mic that MOVI can do much better.

Thanks for your advise,
Dylan.
[Last edited Apr 08, 2016 20:49:21]
avatar
GeraldFriedland said Apr 08, 2016 23:09:08
If MOVI is waiting a long time for you to finish your voice input, it means you need to increase the threshold. Too much noise is making MOVI think somebody is still speaking. Resetting MOVI's recognizer with with stopDialog() and restartDialog() is possible but it shouldn't be necessary if you tune the threshold right.

I am not really understanding your second paragraph. But rather than fiddling with setting the threshold to 95 (which pretty much will kill any recognition, no matter how loud), I wonder if you could make the loudspeakers not be heard by MOVI. I guess, you are right, this means you need to fiddle with your microphone first. MOVI doesn't 'cache' any sentences. After stopDialog() all results should be gone.

Using BEGIN_LISTEN is fine and safer (race condition)! Not sure why you wouldn't see CALLSIGN_DETECTED at all though.

Seeing your video, your best bet is a directed microphone that catches as few noise as possible from the car and other external sources. It's impressive though!
avatar
Dylan said Apr 09, 2016 00:08:44
The desire I have for MOVI not to wait too long for a sentence after a callsign stems from a legacy of false positives on the old board's "Trigger" (their word for callsign). Maybe I can do away with this whole notion with MOVI if I can count on it not to false positive the callsign. I never could figure out how the old board would get "okaymontycarlow" from casual conversation, but it would, and very regularly. Anyway, it's not that MOVI is waiting too long for me to finish my sentence after the call sign, it's waiting too long for me to start talking. At least in the old paradigm I have in my head anyway.

You may not understand the second paragraph entirely because the behaviour is very wonky. In my application, MOVI will quickly (subsecond) return BEGIN_LISTEN, END_LISTEN, RAW_WORDS, and a positive sentence number after a call sign, not nearly enough time to have said anything. This would only happen after the MP3 voice responded. I figured it must be because MOVI heard the MP3 voice speaking prior to the callsign because nothing else that I could think of made sense. If it continues to bother me once my microphone issues are resolved then I'll take a video of it to show you.

With respect to the CALLSIGN_DETECTED issue, once I'm comfortable in sharing my code you can have a look, and hopefully I can screen capture my Serial monitor into a little video to show what MOVI is returning from poll(). It's low priority though seeing as how BEGIN_LISTEN works just fine and you actually think it is preferable. I doubt I'd hit a race condition either way, it's just the indicator I use to have the MP3 player say "Yes Dylan".

This project has been a long time in the works. One could accomplish the same on a modern OBD-II car computer much more easily. I challenged myself to hack into an old 6801 based OBD-1 car computer. Now I just need MOVI to help me make it shine.

Thanks again, hopefully I can move forward again soon with a microphone solution.

Dylan.
avatar
Dylan said Apr 23, 2016 00:20:54
Okay, so I'm back again with a bunch of progress and some finer tuned questions I believe. I'm going to post them in priority (to me anyway) order here, and try to be as succinct as I can:

QUESTION 1 - Customized externally generated sound between callsign and sentence recognition:

You can totally disregard my misunderstanding above about MOVI "caching" the sound prior to the callsign. Let's start again on that one as I now understand the true nature of the problem:

If you watch my video above with me using my old voice recognition board, you'll see that I say the "callsign", then my Sparkfun MP3 says "Yes Dylan", then I say my command. Since making that video, I replaced the "Yes Dylan" with a "Whoosh Whoosh" sound because the old board was so darn bad about false positiving and I got tired of hearing my name when I didn't ask for it. For what it's worth the sound file is here:

http://m1knight.prekfun.com/m1KnightRider/download/audio/soundEffects/scannerSweep.wav

But the problem I run into with MOVI is the same regardless of what sound I play. When I say the callsign to MOVI and the Sparkfun MP3 plays "Whoosh Whoosh", MOVI comes back and thinks I said "Oh Two". When I changed "Oh Two" to "Oxygen", it comes back with "Coolant". I tried doing this (see code snip below) but it has no effect. I have a feeling MOVI does not obey setThreshold if it has already reached a state of BEGIN_LISTEN. Am I right there?

...etc...
if ((movi_res = movi.poll()) != 0 ) {

      Serial.print(F("movi.poll() = "));
      Serial.println(movi_res);

      if (movi_res == BEGIN_LISTEN) {
        movi.setThreshold(95);
        mp3.stopTrack();
        mp3.setVolume(40, 40);
        mp3.playMP3("TRIGGER.MP3");
        vr_mode = VR_MODE_SPEAKING_TRIGGER;
...etc...


The setThreshold in this context doesn't seem to be working. I put MICDEBUG ON and can usually hear the MP3 playing back from MOVI's headphone jack.

From what I understand currently, there appears to be no way to do exactly what I want given the current MOVI API. If that's the case, would you consider an enhancement to either allow the user to disable the automatic BEGIN_LISTEN after the callsign is heard (in this case I could call ask() myself when I'm ready), or a way to turn off the mic even in a sentence listening state?

QUESTION 2 - Timeout after BEGIN_LISTEN

Based on what I have come to understand from your other posts and playing a whole bunch with LowLevelInterface, your primary mechanism for controlling when MOVI stops listening for a sentence is based on not hearing any substantial sound for a period of time, tweaked with setThreshold. Hopefully I'm right there because my question kind of hinges on that assumption. Would it be reasonable to request a feature like a "setTimeout(int)" command that would have MOVI assess whatever sound it has after a limited number of seconds? Or perhaps a command to tell MOVI to END_LISTEN and assess whatever it has? setThreshold() is working for me to a point, but if I could promise that anything useful would be said in, say, 6 seconds from the point where MOVI goes "beep beep", I wonder if that would help get me closer to where I want to be. Is this worth considering? I tried to put in my own timeout that resets MOVI back to listening for the callsign as I discuss above, but I took it out because a lot of the time MOVI actually recognizes my sentence, just after a long time in a particularly noisy environment.

QUESTION 3 - Missing CALLSIGN_DETECTED

I may need to build a fully functional but trimmed down sketch to prove this one out to you fully. That's going to take some time so just in case it's something about the way I'm initializing MOVI, I thought I'd post the piece of code I have in setup(). If this isn't telling then I guess I'm into building a self contained sketch that doesn't require an SDCard, MP3 shield, LCD, etc etc.

  movi.init();

  strcpy_P(MOVI_last_words_trained, (char*)pgm_read_word(&(MOVI_sentence_table[0])));
  movi.callSign(MOVI_last_words_trained);

  int i = 0;
  while(strlen_P((char *)pgm_read_word(&MOVI_sentence_table[i])) > 0) {
    strcpy_P(MOVI_last_words_trained, (char *)pgm_read_word(&MOVI_sentence_table[i]));
    
    movi.addSentence(MOVI_last_words_trained);
    ++i;
  }

  movi.train();

  movi.welcomeMessage(false);             // silence MOVI's welcome message
//  movi.responses(false);                  // silence MOVI's responses
//  movi.beeps(false);                      // silence MOVI's beeps -- we recommend to do that once you are familiar with this example.
  movi.setThreshold(MOVI_THRESHOLD);    // uncomment and set to a higher value (valid range 2-95) if you have a problems due to a noisy environment.



I do see the -200 CALLSIGN_DETECTED coming back in the LowLevelInterface sketch, but not from movi.poll() in my own sketch.

Here's hoping I have some useful food for thought here!
Dylan.
avatar
GeraldFriedland said Apr 23, 2016 17:04:24
Dylan,

First observation:
The setThreshold in this context doesn't seem to be working. I put MICDEBUG ON and can usually hear the MP3 playing back from MOVI's headphone jack.

You are indeed correct: SetThreshold takes a while because the internal speech/non-speech model needs to be reset. So it will not be set quickly enough for ignoring the sound you are playing. You could try to recognize that particular sound (using a bogus sentence like "oh two") and then ignore it or you could set MOVI's output volume to 0 (which is immediate) and SAY a sentence about as long as you need it (and then set the output volume high again) but these are nasty workarounds. Here is the good news: The MOVI 1.1 firmware update includes a PLAY command. This means, you will be able to ask MOVI to play back that file. And of course, MOVI knows not to accidentally process it's own output. So your best bet is to stay tuned and what for us to push the update in a couple weeks (we want to have it for Maker Faire Bay Area).

Question 2:
Yes, this is a reasonable feature request. I am actually already working on something similar for MOVI 1.1. So, I need to have you wait for MOVI 1.1 on this one too. Sorry.

Question 3:
At this point you are doing everything right. I've tried, but unfortunately, I can't reproduce the problem on my end using this snippet.
avatar
Dylan said Apr 23, 2016 17:29:46
Thanks again for your response and support Gerald. Just some brief feedback:

1) I thought about listening for my "whoosh" sound, or even "Yes Dylan" as I had in my video, but then MOVI will go back to listening for the callsign if it gets that. I may try that method again using a hacked version of "ask()" (ask takes too long to get started as you have it because it drops into say() even if the string to say is empty) to get back into listening for a true sentence. You didn't like my idea of a toggle to just have MOVI report CALLSIGN_DETECTED and not automatically go to BEGIN_LISTEN? That would solve my problem if I could manually control when MOVI starts listening after the callsign, though I need to fix question #3 in that case.

2) I will wait patiently for that, thanks!

3) I'll see if I can throw together a self-contained sketch that illustrates the problem with only the base Arduino and MOVI libraries required. This will happen faster if I can turn you on to my idea in question #1 :-)

Thanks again, looking forward to the firmware update.
Dylan.
avatar
GeraldFriedland said Apr 23, 2016 22:22:20
Dylan,

I think the fix that I have in mind for 2) will also help with 1)

3) That would be great!

Gerald
avatar
Dylan said Apr 24, 2016 00:42:34
Okay, I reproduced the missing CALLSIGN_DETECTED in a small sketch. It actually wasn't so easy. Once I ripped out all the vehicle diagnostics checks etc., the problem went away. After scratching my head for a while, I figured that probably meant a timing issue. So I put a delay(10) at the very end of loop(), and there it is - missing -200 CALLSIGN_DETECTED every time I test it. 10 millis to do a bunch of stuff in loop() doesn't seem excessive to me.

First I need to confess something about how I'm using MOVI, though I would thing it would be BETTER than the planned implementation. I'm using a Mega 2560 R3 and wanted to make use of one of the 3 otherwise disused additional hardware serial lines. I chose Serial1. I hacked your library slightly to take a HardwareSerial * as the construction parameter. I'm linking those here for your consideration, and possible use with my sketch if you are inclined to test it with a Mega and wiring your MOVI to RX1/TX1. One would think that the same or worse symptoms as I am seeing would happen with SoftwareSerial.

My callsign is "okaymontycarlo". If you say Okay followed by the name of that famous region of Monaco in relatively rapid succession, MOVI generally picks it up. I'd bet the sketch will yield the same problem with any callsign, but I wanted to give it to you closest to the actual conditions I am using as possible.

movi_callsign_missing.ino

MOVIShield.cpp (modified for HardwareSerial)

MOVISheild.h (modified for HardwareSerial)

Here's where I start crossing my fingers for a fix and a feature to allow me to tell MOVI not to drop straight to BEGIN_LISTEN after CALLSIGN_DETECTED. :-)-X

Dylan.
avatar
GeraldFriedland said Apr 24, 2016 19:34:37
Thanks, Dylan. I'll take a look.

One other thing crossed my mind: If you do a sendCommand("ASK") rather than using the ask() method form the library, MOVI will immediately go into active listen mode and not "try to say something".

Gerald
avatar
Dylan said Apr 24, 2016 19:55:09
Thanks Gerald, I'll be looking forward to your observations. I have a good guess what the problem is, but I don't want to "lead the witness" inappropriately. :)

That's essentially what I meant about a hacked version of "ask()", the only difference being I envisioned adding a method to the MOVI class seeing as I already hacked it a bit for HardwareSerial. That will be exactly what I need if I can reliably catch the CALLSIGN_DETECTED and have a way to turn off the automatic BEGIN_LISTEN.

Dylan.
avatar
Dylan said May 01, 2016 04:06:54
Okay, CALLSIGN_DETECTED missing problem:

I thought it might help to increase the buffer sizes for HardwareSerial since MOVI seems kind of "chatty" with it's -230 responses. So I boosted the buffer sizes from 64 to 256. It didn't really help unfortunately. There is some kind of strange timing issue that I do not understand so I guess I'll need to wait for you to have a run at my sketch to consider a proper fix.

The problem stems from me sending "setThreshold" at the beginning and end of each MP3 that I play. The sketch I uploaded simulates this with some delays. I set debug=true in MOVI's init() routine and here is what I see in my actual sketch:

movi.poll() = -141
MOVIEvent[201]: SPEED

movi.poll() = -201
MOVIEvent[202]: #1

movi.poll() = 2
Heard: SPEED, MOVI selected trained sentence: SPEED
Got VR command 0
0
SPEED1.MP3
0
SPEED2.MP3
0
begin say: SPEED1.MP3
end say: speed1.mp3
begin say: NUMBERS/0/00/0.MP3
end say: numbers/0/00/0.mp3
begin say: SPEED2.MP3
end say: speed2.mp3
done saying non-special case
MOVIEvent[230]: Set THRESHOLD to 95 percent.

movi.poll() = -230
MOVIEvent[230]: Set THRESHOLD to 5 percent.

movi.poll() = -230
MOVIEvent[230]: Set THRESHOLD to 95 percent.

movi.poll() = -230
MOVIEvent[230]: Set THRESHOLD to 5 percent.

movi.poll() = -230
MOVIEvent[230]: Set THRESHOLD to 95 percent.

movi.poll() = -230
MOVIEvent[230]: Set THRESHOMOVIEvent[200]: CALLSIGN DETECTED <-- Here's the problem.

movi.poll() = -230
MOVIEvent[140]: ACTIVELISTEN

movi.poll() = -140
MOVIEvent[230]: Set THRESHOLD to 95 percent.

movi.poll() = -230
MOVIEvent[230]: Set THRESHOLD to 5 percent.

movi.poll() = -230
MOVIEvent[141]: END ACTIVELISTEN

movi.poll() = -141
MOVIEvent[201]: AVERAGES

This is the tail end of a callsign/command sequence followed by another callsign/command. You can see the issue on the line where I put "Here's the problem".

I even tried to put a while (movi.poll() == -230); after every setThreshold to try to clear those out, but for some reason I do not understand, the -230's seem to come through well after all of my MP3 responses are played, with the last one getting buffered somehow and then being muddled up with the next CALLSIGN_DETECTED.

I suspect there is a little bug in MOVI where it's buffering the -230 responses. I can work around it for now by hacking poll() to look for the last indexOf("MOVIEvent[") instead of the first in a string received from MOVI. For a real solid fix, I think I'll need to wait for you to run my sample sketch and consider how MOVI's firmware is reacting.

Thanks for your continued support!!
Dylan.
avatar
Dylan said May 04, 2016 04:42:24
I had a closer look at the MOVI API poll() routine on a hunch I had recalling the hack I did to use lastIndexOf. A big part of my problem here is that a single call to poll() retrieves at most one character from the serial variable that the class is initialized with. That means that in order to get a non-zero response to (for example):

MOVIEvent[230]: Set THRESHOLD to 95 percent.

It takes at least 58 calls to poll() before a non-zero result is returned.

Why would you use:

if (mySerial->available()) {

instead of:

while (mySerial->available()) {

in poll() to make it more of a hungry algorithm?

Dylan.
Login below to reply: