Many of the most compelling agents use datasources and datatables to extend their functionality beyond static, written output.  For example, if a user asks about the weather, the agent's response can be drawn from a datasource that passes parameters to a web service and retrieves the current weather conditions.  Another use of datasources and datatables lies on the input side -- we want to recognize that the user is asking about something contained in this dynamic or preloaded pool of data the agent uses for output.  There are a few ways of doing this.

Datatables

A simple datatable is just that -- a table with rows and columns, sometimes in a tab-delimited text file, preloaded into memory when the agent compiles.  We can use datatables to map one value to others, or simply as a list, and then manipulate these values to display a dynamic output.  We can also use the table as the contents of a subpattern that users match on, using a datatable subpattern.

Say we have a list of actors in a text file, with their names and IDs.  The ID we use elsewhere, but we can use the names to see if the user is asking about an actor, and do something special if they are.  The list looks like this:

  485957    Jim Calabrese

  485958    Susan Leber

  485959    Michael Yanko

Next we need to define the datatable:

datatable ActorIDTable

  load Id {index=exact}, Name {index=thawed} from file

    ActorID.txt

Now we can create a subpattern to match on a column in the datatable. 

subpattern AllActors get Name, Name in ActorIDTable {maxlength=3}

"get Name, Name in ActorIDTable" means that the subpattern will look at the ActorIDTable datatable and match on the values in the Name column (which are thawed), and return the values in the Name column.  It's the equivalent of this:

subpattern AllActors {spellcorrect=”no”}

+ jim calabrese

  return "Jim Calabrese"

+ susan leber

  return "Susan Leber"

+ michael yanko

  return "Michael Yanko"

We can make the agent's actor-recognition more robust by matching on words that look like names, and rolling it all into an actor subpattern.

subpattern AnActor

+ VAR=AllActors

+ VAR=APossibleName {score=40}

  return VAR

The last step is to create topics:

? I'm a fan of ACTOR=AnActor.

  - Yes, ACTOR deserves an Oscar.

? I'm a fan of VAR=Anything.

  - You like VAR?  I'll keep that in mind.

  User:  I am a big fan of michael yanko.

  Agent: Yes, Michael Yanko deserves an Oscar.

  User:  I'm also a fan of ice cream.

  Agent: You like ice cream?  I'll keep that in mind.

Datasources

A datasource is a tool to access external data, like a web service or database.  While there is not a "datasource subpattern" in the BuddyScript platform, it is still possible to match on the dynamic data accessed by a datasource, using a variable subpattern.

A variable subpattern matches on the keys of an object, so if we use our datasource to fill an object, we can use that object’s keys in our matching.  The return value of the subpattern is the matched key.

For example, say we have a function that calls a datasource that returns a list of TV channels a user has in her area, called ChannelLineup().  It returns an list of objects, each with the channel number and call letters for one channel.  We can manipulate that list to populate an object variable, G_USER_CHANNELS, at some point in the user session before the variable subpattern is to be used.

stored variable G_USER_CHANNELS

procedure PopulateUserChannels()

  CHANNELS = ()

  CHANNELS = ChannelLineup()

  G_USER_CHANNELS = ()

  for IDX in CHANNELS

    NUMBER   = CHANNELS[IDX].ChannelNumber

    LETTERS  = StringClean(CHANNELS[IDX].CallLetters)

    G_USER_CHANNELS[NUMBER]   = LETTERS

    G_USER_CHANNELS[LETTERS]  = NUMBER

Note that we must use StringClean to make the call letters lowercase -- otherwise the subpattern will contain illegal characters.

Next we create the subpattern:

subpattern AUserTVChannel variable G_USER_CHANNELS {style=raw score=MACRO_STRONG_SCORE}

And topics.

? Do I have CHANNEL=AUserTVChannel?

  - You do have CHANNEL.

? Do I have VAR=Anything?

  - I'm not sure if you have VAR.

  User: Do I have FOX?

  Agent: You do have fox.

  User: Do I have a car?

  Agent: I'm not sure if you have a car.

However, using a variable subpattern in this case is actually unnecessary.  A better solution is to create a datatable subpattern with all possible call letters and channel numbers, and then procedurally figure out whether the user has the channel they’ve matched on.

Consider another example.  We have a function calling a datasource that returns a list of products currently available, along with price and description.  Every week new products are added and old ones removed, so we never have quite the same list, or know what new products may be added. 

The agent must recognize when users are asking how much a product is, even if it is out of stock.  What we can do is maintain a list of all products ever sold, and add to the list when we find a new product.

subpattern APossibleProduct variable G_ALL_PRODUCTS {style=raw score=MACRO_STRONG_SCORE}

procedure DisplayProductDetails(NAME)

  PRODUCTS |= ()

  PRODUCTS = GetProductDetails(NAME)

  if !PRODUCTS || PRODUCTS == ()

    - Sorry, NAME is no longer available.

  else

    for value PRODUCT in PRODUCTS

      * Here are the products available:

      - PRODUCT.name, PRODUCT.price

  PRODUCT.detail

procedure AddProductsToPossibleProducts(NAME)

  PRODUCTS <= GetProductDetails(NAME)

  for value PRODUCT in PRODUCTS

    G_ALL_PRODUCTS[PRODUCT.name] = PRODUCT.price

? Do you have any PRODUCT=APossibleProduct?

? Do you have any VAR=Anything?

  if PRODUCT

    call DisplayProductDetails(PRODUCT)

  else

    if IsAStockedProduct(VAR)

      call AddProductsToPossibleProducts(VAR)

      call DisplayProductDetails(VAR)

    else

      - Sorry, I don't recognize VAR as a valid product.

One warning: depending on the size of the object variable, variable subpatterns are potentially big resource hogs, and can harm your agent's response time.  So use with care.

In his last post, Mirco explained how to add unit tests to your Agent: it’s an absolutely critical investment to add unit tests covering as much of the functionality of your Agent as possible, to make sure it performs the way you want throughout development and beyond.

In this post I’ll describe a complementary testing approach (you’ll still need to write unit tests!), known as fuzzing.

Fuzzing???

Fuzzing an Agent works like this: for each handle in your project the platform builds a random question that should match on it. It then asks all these questions to your Agent as a user would. Finally, it reports what questions caused execution errors, and gives you a sorted list of the slowest ones.

For instance, assuming you have the following routine in your code (not that you’d write something like that, of course):

? I was born in YEAR=Anything.

  AGE = GetCurrentYear() - YEAR

  - OK, so that makes you about AGE years old, right?

The tool would take your handle “? I was born in YEAR=Anything.”, try to randomly generate a question that would match it, and ask that question. In this case, the tool might come up with something random like “i am born in Blah blah blah” (through stemming, synonyms, etc). Of course trying to execute that question generates an error, which the tool would detect.

Step-by-step

That’s for the theory. Concretely to take advantage of this test, you simply need to launch your project in the IDE (via the query window) and type the following as your questions:

-        !d 0 => this turns off debug info display which speeds up the test substantially

-        !testmode => enters the Test mode, which offers a variety of options to tweak how the test is run (leaving the defaults is ok)

-        go => launches the test

Depending on the size of your project, the test might take a long time to finish, since it will ask at least one question for each handle of your project.

Important: if you have debug code somewhere in your code, it will be executed unless you “protect” it!

// MACRO_PROTECTED_MATCHING prevents this handle from being “fuzzed” during the test

// You wouldn’t want a random amount transferred to your test account…

+ _for_debug_transfer NUMBER=ANumber dollars to test account {MACRO_PROTECTED_MATCHING}

  call BankingDataSourceTransfer(NUMBER, G_MY_ACCOUNT, G_TEST_ACCOUNT)

Sample output

Below is a sample output of this coverage test, edited for brevity:

joeuser: !d 0

null

joeuser: !testmode

Automated Agent:
****************************

**   Entering Test Mode   **

****************************

Type: "help" or just "h" to see a list of available commands.

testmode: go

At this point you have to wait for the test to complete, which takes between a few seconds and a few hours depending on your agent and on your machine.

Automated Agent:

=======================================

Error Report:

*************************************************************

That’s a test executed for our sample routine described above

Problem encountered testing pattern 325: I was born in YEAR=Anything. (in ? I was born in YEAR=Anything.)

Sequence:

  i am born in Blah blah blah

Problem:

Exception: Exception while processing "i am born in Blah blah blah":

Error executing script associated to query "i am born in Blah blah blah"

  Error executing script labeled "? I was born in YEAR=Anything." at line 106 in file "domains:/TestProject/English/UserInfo/UserName.ddl":

    Error executing command "AGE = GetCurrentYear() - YEAR" in "domains:/TestProject/English/UserInfo/UserName.ddl", line "107"

      Can't convert expression "Blah blah blah" to a float

=======================================

30 slowest queries:

This section tells you which questions were the slowest ones to answer. If you access slow web services that you then cache in memory you might want to run the test twice to get more accurate results. Many factors may slow down a query execution rate. To “debug” slow queries, you need to check how it was analyzed by the NL engine (did it trigger “runaway rephrase rules” or involve a subpattern that happens to be slow?), if the associated script can execute quickly, etc.

588ms:  Query(ies) asked: "which gr8 dose common factor of 3 & 3"   Pattern represented="What is the Greatest Common Factor of X=AnInteger and Y=AnInteger? (in answers WhatIsTheGreatestCommonFactorOf<X, Y>)".

320ms:  Query(ies) asked: "divide the sinus of ( 12.34 ) percent with sq rt ( a million a hundred ) cubed is what in its simplest form"   Pattern represented="EXPRESSION=AComplexExpressionNoPlug =EqualsWhat [FORMAT=AResultFormatType] (in +- Calculate EXPRESSION.description)".

271ms:  Query(ies) asked: "which being the plural of white dwarf"   Pattern represented="What is the plural of NOUN=ANoun? (in answers WhatIsThePluralOfNoun<NOUN>)".

254ms:  Query(ies) asked: "we would prefer those you people call us which"   Pattern represented="I would prefer that you call me that. (in answers CallMeThat)".

238ms:  Query(ies) asked: "sum body calls me which"   Pattern represented="People call me that. (in answers CallMeThat)".

… edited for brevity…

107ms:  Query(ies) asked: "which percents of 3 is pi"   Pattern represented="(what|which) =nsPercent of WHOLE=AnAtomicExpression is PART=AnAtomicExpression (in answers WhatPercentOf<WHOLE>Is<PART>)".

107ms:  Query(ies) asked: "be u top of the line at math"   Pattern represented="Are you good at math? (in answers CanYouHelpMeWithMath)".

105ms:  Query(ies) asked: "my names is it" => "call us mrs godowns" => "pass" => "aye" => "nah thank yo u sorry" => "if u insist" => "that s wrong" => "alright" => "nada" => "thats wrong" => "we might prefer which you calls us sir plasmapheresis"   Pattern represented="my =nName is it (in answers CallMeThat)".

=======================================

Total of 2115 questions asked in 24050 milliseconds.

  Breakdown of execution time:

    Rephrasing:         8863 milliseconds 36.852390852391% of processing time

    Matching:           6329 milliseconds 26.316008316008% of processing time

    Preprocessing:      1 milliseconds 0.0041580041580042% of processing time

    ScriptExecution:    6154 milliseconds 25.588357588358% of processing time

    Postprocessing:     2248 milliseconds 9.3471933471933% of processing time

    Presentation layer: 100 milliseconds 0.41580041580042% of processing time

    Rest:               355 milliseconds 1.4760914760915% of processing time

and 2114 answers received (99.95%)

Average speed for queries answered: 11.381921438713 milliseconds (that's 87.858627858628 QPS)

1 error(s) during the test.

Complete test took 13885 milliseconds to process.

=======================================

Conclusion

Having a thorough test strategy is critical to the quality of your Agent. You want to find bugs before your users do! Nothing replaces good unit tests (see Mirco’s post), but fuzzing is a good way to detect cases you didn’t think about, or to identify slow queries. It’s a good idea to run a fuzzing test on your Agent when you reach critical milestones (before you make a beta available, before you launch, before releasing a major update, etc).