An advanced look at Web Services and Datasources
Back in February, we showed a basic example of accessing a web service and using a datasource to consume information from a web service. (Refer to: http://windowsliveagents.spaces.live.com/blog/cns!5BCD45E519E07634!443.entry) Today, we’ll take a further look at a real-world web service, and some further things you can do with a datasource to make your application perform better, and to potentially reduce coding effort.
Many agents rely on rich content in order to create a solid user experience. Many times, this involves accessing an external data store, such as a web service. With some web services, you can parameterize your request in the URL itself, for example http:/URL?<input>. More often, web services involve using request and response model via a SOAP API.
The Windows Live Search API is a very powerful web service that does many things. It allows a user to do searches on the web, news, images, as well as get dictionary, phonebook and spelling information. It provides a web service XML interface using a SOAP API, enabling you to submit requests and get back a response via XML. For more information on the XML, refer to this link: http://msdn.microsoft.com/en-us/library/bb251794.aspx
Using any number of XML tools, we can take a look at the WSDL of the Live Search API web service to see how a typical request and response is crafted. The WSDL endpoint is this:
http://soap.search.msn.com/webservices.asmx?wsdl
For purposes of discussion, let’s say that we are going use a Web News search, searching for “Baron Davis”. The query of “Baron Davis” is put into the Query element. There are numerous other elements that are part of the response. They are highlighted in red. The XML response would look like this:
<?xml version="1.0" encoding="utf-16"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap:Body>
<Search xmlns="http://schemas.microsoft.com/MSNSearch/2005/09/fex">
<Request>
<AppID>your APP ID</AppID>
<Query>Baron Davis</Query>
<CultureInfo>en-us</CultureInfo>
<SafeSearch>Moderate</SafeSearch>
<Flags>None</Flags>
<Location>
<Latitude>0</Latitude>
<Longitude>0</Longitude>
</Location>
<Requests>
<SourceRequest>
<Source>News</Source>
<Offset>0</Offset>
<Count>10</Count>
<FileType />
<ResultFields>All DateTime</ResultFields>
<SearchTagFilters>
<string />
</SearchTagFilters>
</SourceRequest>
</Requests>
</Request>
(Note that strangely enough, in the ResultFields element, putting in the request “All” usually returns back all fields In the case of a News request, this does not return back all fields. “All DateTime” will return all fields, plus the datetime in this case.)
The XML response looks like this, with output fields in bold:
<?xml version="1.0" encoding="utf-16"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<SearchResponse xmlns="http://schemas.microsoft.com/MSNSearch/2005/09/fex">
<Response>
<Responses>
<SourceResponse>
<Source>News</Source>
<Offset>0</Offset>
<Total>31185</Total>
<Results>
<Result>
<Title>NBA free agent roundup: Baron Davis to L.A.?</Title>
<Description>Baron Davis has a verbal agreement with the Clippers, Gilbert Arenas likely is staying in Washington, and Elton Brand has a decision to make as NBA free agency opened with some surprises yesterday. The dominoes began to tumble when Davis surprised ... </Description>
<Url>http://www.newsday.com/sports/basketball/knicks/ny-spnba025749266jul02,0,2266274.story </Url>
<DisplayUrl>http://www.newsday.com/sports/basketball/knicks/ny-spnba025749266jul02,0,2266274.story</DisplayUrl>
<Source>Newsday</Source>
<DateTime>
<Year>2008</Year>
<Month>7</Month>
<Day>2</Day>
<Hour>15</Hour>
<Minute>26</Minute>
<Second>29</Second>
</DateTime>
</Result>
<Result>
<Title>Later Daze, Baron: Davis leaves Warriors for Clippers</Title>
<Description>del.icio.us Less than 24 hours after Davis shocked the Warriors by walking away from the final year and $17.8 million left on his contract, the franchise guard delivered a second strike Tuesday afternoon. Davis agreed in principle with the Clippers ... </Description>
<Url>http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2008/07/02/SPCF11ICKB.DTL</Url>
<DisplayUrl>http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2008/07/02/SPCF11ICKB.DTL</DisplayUrl>
<Source>San Francisco Gate</Source>
<DateTime>
<Year>2008</Year>
<Month>7</Month>
<Day>2</Day>
<Hour>15</Hour>
<Minute>19</Minute>
<Second>20</Second>
</DateTime>
</Result>
</Results>
</SourceResponse>
</Responses>
</Response>
</SearchResponse>
</soapenv:Body>
</soapenv:Envelope>
For purposes of display, only 2 items are displayed, but in the real-life example, there are 31,185 articles on “Baron Davis” that were found. The 31,185 is indicated in the <TOTAL> element in the response. Note that not all 31,185 rows are retrieved. The amount retrieved is regulated in the Preprocess block, in the built-in LIMIT and OFFSET variables. More on this later, when we describe the datasource.
Let’s take a look at how we generate the Buddyscript code to access this web service.
A typical datasource accessing XML might have header information that looks like this:
datasource dsSample(ARG1, ARG2, ARG3) => A1, B1, C1
http
http://someurl/sample.xml
simple xml
.
.
.
However, for a web service invoking SOAP messages, we have to build a request string, so we use the PreProcess section of a datasource to set a variable that will contain the request, along with other variables. Thus, the beginning of the datasource would look like this instead:
datasource LiveSearchAPI(SEARCH, CULTURE_INFO) => Title, Description, Url, Source, NewsYear, NewsMonth, NewsDay, NewsHour, NewsMinute, NewsSecond {expire="in 1 hour" continue_on_error="true"}
preprocess
if LIMIT>10 || LIMIT<=0
MAXRESULTS = 10
else
MAXRESULTS = LIMIT
FIELDLIST = "Title Description Url Source DateTime"
POST_DATA = BuildSearchAPIPostData(SEARCH, "News", OFFSET, MAXRESULTS, CULTURE_INFO, FIELDLIST)
http
http://soap.search.msn.com:80/webservices.asmx
header
Accept: application/soap+xml
postdata {encode=no}
POST_DATA
simple xml
.
.
.
We’ll describe the syntax of the datasource in more details later on, but for now, what’s important to point out is that a function is called to build the request string. The function in this case is called BuildSearchAPIPostData and has a number of arguments. Let’s take a look at the function:
function BuildSearchAPIPostData(SEARCH, TYPE, OFFSET, COUNT,CULTURE_INFO,
FIELDLIST, RADIUS, LATITUDE, LONGITUDE)
if (LATITUDE eq "")
LATITUDE = 0
if (LONGITUDE eq "")
LONGITUDE = 0
if (RADIUS eq "")
RADIUS = 5
POST_DATA = '<?xml version="1.0" encoding="UTF-8" standalone="no"?>\n'
POST_DATA = StringConcat(POST_DATA, '<SOAP-ENV:Envelope xmlns:SOAP-ENV="http:/\/schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:tns="http://schemas.microsoft.com/MSNSearch/2005/09/fex" > <SOAP-ENV:Body>')
POST_DATA = StringConcat(POST_DATA, ' <tns:Search xmlns:tns="http:/\/schemas.microsoft.com/MSNSearch/2005/09/fex"> <tns:Request>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:AppID>', GetLiveSearchAPIAppID(), '</tns:AppID>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:Query>', SEARCH , '</tns:Query>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:CultureInfo>', CULTURE_INFO, '</tns:CultureInfo> <tns:SafeSearch>Moderate</tns:SafeSearch> <tns:Flags>None</tns:Flags>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:Location>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:Latitude>', LATITUDE, '</tns:Latitude>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:Longitude>', LONGITUDE, '</tns:Longitude>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:Radius>', RADIUS, '</tns:Radius>\n')
POST_DATA = StringConcat(POST_DATA, ' </tns:Location>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:Requests> <tns:SourceRequest>\n')
POST_DATA = StringConcat(POST_DATA, ' <tns:Source>', TYPE, '</tns:Source>\n') // Web / Ads / InlineAnswers / PhoneBook / WordBreaker / Spelling
POST_DATA = StringConcat(POST_DATA, ' <tns:Offset>', OFFSET, '</tns:Offset> <tns:Count>', COUNT, '</tns:Count> <tns:ResultFields>', FIELDLIST, '</tns:ResultFields>\n')
POST_DATA = StringConcat(POST_DATA, ' </tns:SourceRequest> </tns:Requests>\n')
POST_DATA = StringConcat(POST_DATA, ' </tns:Request> </tns:Search>\n')
POST_DATA = StringConcat(POST_DATA, '</SOAP-ENV:Body> </SOAP-ENV:Envelope>\n')
return POST_DATA
The function essentially uses the variable POST_DATA to build the request string. If there are certain parameters without information, it defaults the information (e.g. LATITUDE). The function also calls another function to get the Windows Live APP ID. This APP ID is used in the Live Search API to determine individual access rights. (See the Windows Live API for more information on obtaining this APP ID.) The function itself simply looks like this:
function GetLiveSearchAPIAppID()
return "ABCDE12345"
where “ABCDE12345” is the App ID.
We'll take a look at the rest of the datasource and invoke code later this week.
Outils 
Aide
