ODA’s release 22.08 brought yet again a significant number of new features.
One of those is the new Skill Quality Report, which enables us to assert on the Skill’s intent and utterances quality, by providing metrics and suggestions,
You can use the following reports (accessed by clicking 
in the left navbar) throughout the skill development process:
The Overview Report – This report’s metrics and graphs allow you quickly assess the size and shape of your training data by breaking down each of the skill’s intents by the number of utterances that they have. You can use this report as your data set grows throughout out the development process to make sure that the number of intents, the number of utterances per intent, and the word count for each utterance are always compliant with our guidelines.
The Report – Breaks down the intents by anomalies, utterances that might be inappropriate for their intent because they are out of scope or mislabeled, or are potentially applicable but difficult to classify because they are unsual.
Skill Quality Report
I picked up one of my older Skills, a custom HCM Skill, with plenty of intents to make the report more interesting.
This report can tell you at a glance if:
Your skill has the minimum of two intents, but has not exceeded the recommended number of 300 intents.
Your training does not exceed 25,000 utterances.
Each intent has a minimum of two utterances (with the recommended minimum of 12 utterances to provide predictable training).
The average utterances length is between three and 30 words.
The Overview gives you a graphical representation of the intents and the number of utterances. It is good practice to have a balanced set, without the extreme of one intent having 5 utterances and the next intent 500.
Below there is more information with potential warning signs. In my Skill, there are 2 warning. One for a very small intent, having only 5 utterances (really a bad practice, we should aim for 30-40 to start with), and an utterance with 1 word only (I suspect it is the Greeting intent)
Furthermore, it does explain why some intent are marked with issues.
Report
There are a couple of pre-requisites in order to be able to run the report.
Skill must be 21.12 or higher, model must use Trainer Tm and Intents must be enabled (this last one feels strange but technically it’s correct)
Once all of that is checked, then we can go ahead.
Like the Overview report, this report breaks down intents by utterances. The utterances tallied in this report, however, are anomalies, utterances that might be inappropriate for their intent because they are out of scope or mislabeled, or are potentially applicable but unique. Anomalies can cause the boundaries between intents to become blurred, particularly if you’ve extended your skill to cover new domains or have partitioned a large intent into smaller intents.
Below this graph we have the listed anomalies.
We can explore each of those anomalies and visualize why they were listed. In most cases it’s to the proximity with another utterance from another intent. This blurs the intent borders and makes it harder to properly resolve the utterance.
Based on the anomaly’s relationship to the other utterances in the report, you may find that the utterance is simply misclassified or not pertinent to the skill at all. Other anomalies, however, may only be anomalous to the other utterances in that they are implicit requests – they are asking the same questions as the other training data, but in a different way. For example, “I can’t log into my account” might be evaluated as an anomaly for an agent transfer intent whose training utterances cluster around more straightforward requests like “I need to talk to a live agent.” Adding similar utterances will add depth to the model.