The magic of extracting unknown values with Machine Learning Entities

In a previous post I wrote about Entities and their importance in conversational AI. In this post I will cover Machine Learning (ML) Entities and the use cases where we should be using them. But first a small introduction to why one would need such a type of entities.

Custom Entities

One of the most important types of custom entities is Value List , which as the name indicates, provides a list of values.

We define a limited set of data, including synonyms and we assign that entity to an intent. Whenever any of those values, or associated synonyms appear in the conversation, the NLP engine is able to identify and extract them.

Dynamic Entities are pretty much the same as Value Lists, with the added feature of being dynamic, which means they can be updated in real time with a REST API. Both contain a known set of values and are widely used while building entities.

Another example is Regular Expression, which use regex to be able to match certain patterns (Order numbers or particular identifiers are very common use cases). This type also requires the knowledge of what we are looking for, in this case the knowledge of the format.

But here is the main question: What if we have an unknown set of data?

Or what if we need something more flexible, not dependent on patterns or a fixed set of data and its synonyms?

Enter…ML Entities

This is where ML Entities play a role, because ML Entities allow us to extract known and unknown values!!

The mechanism behind this is completely different than any of the other Entities and relies on the power of the NLP engine. We train the entity with utterances and annotations. This means the model does not learn from the values themselves, but instead by how and where these values are used in an utterance.

See below a visual representation.

In the above example we have a ML Entity for TV Series, which is a perfect use case for an ML Entity, as it impossible to have a complete list of all TV Series.

The NLP does not care about A-Team or Gilmore Girls, the only relevant information is how those values appear in an utterance, and that is how those values are extracted. Good quality training data is fundamental for this to work properly!

Important to note is that ML Entities are a complement to the existing set and not a magical solution for every use case.

We can also complement ML Entities with Value Lists, which makes sense when we have some known values, but not all.

Gathering of data

Gathering data is critical for ML Entities, as we need a large set of utterances and annottations.

Apart from the necessary steps of design and clearly defining the scope of those entities we do need to be able to generate quality utterances and annotations.

ODA’s Data Manufacturing tool gives us the ability to create jobs so that we can gather utterances (paraphrasing job), and after that we can create a annotation job (to annotate the entities in those utterances).