Definition of Reporting Terms in Usage Reporting

When your agent launches, you will be given access to the Knowledge Management Server, which includes a Usage Reporting section. The URL is https://YourProjectName.console.agents.live.com.

When you navigate to the Usage Reporting site, you will see date range options in the left pane, and report results in the right pane.  The default date is the present day.  You can switch days by clicking on the calendar on the left.

If you click on “Custom Range” on the upper left side of the page, you can view usage reports for a custom date range. Please not that in this view, New & Unique Users will be shown as N/A. Due to a design limitation, we currently cannot provide accurate measurements of new and unique users for custom reporting periods.

At the top of the page, you may see the following sections of reporting: Volume Summary, User Demographics, Languages Used, Activity Usage, Compliance, Category Analysis, Topic Analysis, and Clickthroughs. The default reporting view is Volume Summary. The sections of Usage Reporting you see, depends on how you have set up the usage_config.xml your project uses. For more information on customizing the usage_config.xml, click here.

Definition of Reporting Terms: 

Volume Summary Section

Total Queries:             

The total number of queries in all sessions within the specified time period. For example, if the time period includes 100 sessions, there could be 1000 or more total queries.

Total Sessions:           

The total number of sessions within the specified time period.

Unique Users:             

The number of unique users within the specified time period. Each user is counted once, regardless of how often they interact with the agent. For example, if one person has ten separate sessions with the agent during the specified time period, they are still identified as one unique user.

New Users:                  

The number of users whose initial session with the agent occurred during the specified time period.

Average Queries Per Session:

Total queries within the specified time period divided by total sessions within the specified time period.

Average Sessions Per Unique Users:  

Number of unique users within the specified time period divided by total number of sessions within the specified time period.

Category & Topic Analysis Sections

Category Analysis is broken out into two sections: Category Distribution Per Query and Category Distribution Per Session. Category Distribution Per Query breaks down how many queries for the chosen date range matched to a category. Category Distribution Per Session breaks down how many times a query matched on a Category during a session.

Topic Analysis is only broken out into one section: Topic Distribution (By Query). This measures how many times users’ queries matched to a specific topic for the chosen date range.

For information on how to log Category and Topic Analysis in your project, click here.

Clickthrough Section

The Clickthrough section breaks down how many times a user is presented a link and how many times a user clicks on the link.

Total Impressions

Number of times a user is presented a link in their conversation with the bot.

Total Clicks

Number of times a user clicks on the link he / she is presented.

For information on tracking ClickThroughs in your project, click here.

Languages Used Section

Agent detects what languages users speak to the agent in.

Activity Usage Section

Tracks how many users accept or reject an invitation to open the Activity Window.

Compliance Section

Conversations Stopped

Number of conversations interrupted after detecting user was typing sensitive topics beyond the scope of the agent.

Sensitive Sequences Rejected

Number of answers that were blocked by the output filter, displaying an error message to the user.

Sensitive Sequences Trusted

Number of answers that would have been blocked by the output filter, but were let through because of the use of the tag <trusted>.

Answers Invoking Trust

Number of messages that were displayed with the tag <trusted>…</trusted> in them. It includes both messages that would have passed the output filter or failed it.