Lingjia Tang

US Patent:

20210241066, Aug 5, 2021

Filed:

Jan 15, 2021

Appl. No.:

17/149786

Inventors:

- Ann Arbor MI, US
Jason Mars - Ann Arbor MI, US
Lingjia Tang - Ann Arbor MI, US
Michael A. Laurenzano - Ann Arbor MI, US
Johann Hauswald - Ann Arbor MI, US
Yiping Kang - Ann Arbor MI, US
Yunqi Zhang - Ann Arbor MI, US

International Classification:

G06N 3/00
G06N 20/00
G06N 5/04
G06F 40/30
G06F 16/332
G06K 9/62

Abstract:

A system and method of configuring a graphical control structure for controlling a machine learning-based automated dialogue system includes configuring a root dialogue classification node that performs a dialogue intent classification task for utterance data input; configuring a plurality of distinct dialogue state classification nodes that are arranged downstream of the root dialogue classification node; configuring a graphical edge connection between the root dialogue classification node and the plurality of distinct state dialogue classification nodes that graphically connects each of the plurality of distinct state dialogue classification nodes to the root dialogue classification node, wherein (i) the root dialogue classification node, (ii) the plurality of distinct classification nodes, (iii) and the transition edge connections define a graphical dialogue system control structure that governs an active dialogue between a user and the machine learning-based automated dialogue system.

US Patent:

20210166138, Jun 3, 2021

Filed:

Jan 15, 2021

Appl. No.:

17/149998

Inventors:

- Ann Arbor MI, US
Anish Mahendran - Ann Arbor MI, US
Parker Hill - Ann Arbor MI, US
Jonathan K. Kummerfeld - Ann Arbor MI, US
Michael A. Laurenzano - Ann Arbor MI, US
Lingjia Tang - Ann Arbor MI, US
Jason Mars - Ann Arbor MI, US

International Classification:

G06N 5/04
G06N 20/00

Abstract:

Systems and methods for automatically detecting annotation discrepancies in annotated training data samples and repairing the annotated training data samples for a machine learning-based automated dialogue system include evaluating a corpus of a plurality of distinct training data samples; identifying one or more of a slot span defect and a slot label defect of a target annotated slot span of a target training data sample of the corpus based on the evaluation; and automatically correcting one or more annotations of the target annotated slot span based on the identified one or more of the slot span defect and the slot label defect.

US Patent:

20210004539, Jan 7, 2021

Filed:

Sep 1, 2020

Appl. No.:

17/009695

Inventors:

- Ann Arbor MI, US
Stefan Larson - Ann Arbor MI, US
Christopher Clarke - Ann Arbor MI, US
Kevin Leach - Ann Arbor MI, US
Jonathan K. Kummerfeld - Ann Arbor MI, US
Parker Hill - Ann Arbor MI, US
Johann Hauswald - Ann Arbor MI, US
Michael A. Laurenzano - Ann Arbor MI, US
Lingjia Tang - Ann Arbor MI, US
Jason Mars - Ann Arbor MI, US

International Classification:

G06F 40/35
G06N 5/04
G06N 20/00
G06F 40/284

Abstract:

Systems and methods for constructing an artificially diverse corpus of training data includes evaluating a corpus of utterance-based training data samples, identifying a slot replacement candidate; deriving distinct skeleton utterances that include the slot replacement candidate, wherein deriving the distinct skeleton utterances includes replacing slots of each of the plurality of distinct utterance training samples with one of a special token and proper slot classification labels; selecting a subset of the distinct skeleton utterances; converting each of the distinct skeleton utterances of the subset back to distinct utterance training samples while still maintaining the special token at a position of the slot replacement candidate; altering a percentage of the distinct utterance training samples with a distinct randomly-generated slot token value at the position of the slot replacement candidate; and constructing the artificially diverse corpus of training samples based on a collection of the percentage of the distinct utterance training samples.

US Patent:

20200401914, Dec 24, 2020

Filed:

Jun 8, 2020

Appl. No.:

16/895147

Inventors:

- Ann Arbor MI, US
Anish Mahendran - Ann Arbor MI, US
Parker Hill - Ann Arbor MI, US
Jonathan K. Kummerfeld - Ann Arbor MI, US
Michael A. Laurenzano - Ann Arbor MI, US
Lingjia Tang - Ann Arbor MI, US
Jason Mars - Ann Arbor MI, US

International Classification:

G06N 5/04
G06N 20/00

Abstract:

Systems and methods for automatically detecting annotation discrepancies in annotated training data samples and repairing the annotated training data samples for a machine learning-based automated dialogue system include evaluating a corpus of a plurality of distinct training data samples; identifying one or more of a slot span defect and a slot label defect of a target annotated slot span of a target training data sample of the corpus based on the evaluation; and automatically correcting one or more annotations of the target annotated slot span based on the identified one or more of the slot span defect and the slot label defect.

US Patent:

20200380964, Dec 3, 2020

Filed:

May 18, 2020

Appl. No.:

16/876198

Inventors:

- Ann Arbor MI, US
Michael A. Laurenzano - Ann Arbor MI, US
Johann Hauswald - Ann Arbor MI, US
Lingjia Tang - Ann Arbor MI, US
Jason Mars - Ann Arbor MI, US

International Classification:

G10L 15/18
G06N 20/00
G06N 5/04
G10L 15/22
G10L 15/04

Abstract:

Systems and methods for building a response for a machine learning-based dialogue agent includes implementing machine learning classifiers that predict slot segments of the utterance data based on an input of the utterance data; predict a slot classification label for each of the slot segments of the utterance data; computing a semantic vector value for each of the slot segments of the utterance data; assessing the semantic vector value of the slot segments of the utterance data against a multi-dimensional vector space of structured categories of dialogue, wherein the assessment includes: for each of a distinct structured categories of dialogue computing a similarity metric value; selecting one structured category of dialogue from the distinct structured categories of dialogue based on the computed similarity metric value for each of distinct structured categories; and producing a response to the utterance data.

US Patent:

20200364410, Nov 19, 2020

Filed:

Jul 30, 2020

Appl. No.:

16/944124

Inventors:

- Ann Arbor MI, US
Lingjia Tang - Ann Arbor MI, US
Michael A. Laurenzano - Ann Arbor MI, US
Johann Hauswald - Ann Arbor, US
Parker Hill - Ann Arbor MI, US
Yiping Kang - Ann Arbor MI, US
Yunqi Zhang - Ann Arbor MI, US

International Classification:

G06F 40/30
G06N 5/04
G06N 20/00

Abstract:

A system and method for intelligently configuring a machine learning-based dialogue system includes a conversational deficiency assessment of a target dialog system, wherein implementing the conversational deficiency assessment includes: (i) identifying distinct corpora of mishandled utterances based on an assessment of the distinct corpora of dialogue data; (ii) identifying candidate corpus of mishandled utterances from the distinct corpora of mishandled utterances as suitable candidates for building new dialogue competencies for the target dialogue system if candidate metrics of the candidate corpus of mishandled utterances satisfy a candidate threshold; building the new dialogue competencies for the target dialogue system for each of the candidate corpus of mishandled utterances having candidate metrics that satisfy the candidate threshold; and configuring a dialogue system control structure for the target dialogue system based on the new dialogue competencies, wherein the dialogue system control structure governs an operation of an automated dialogue agent.

US Patent:

20200272855, Aug 27, 2020

Filed:

Apr 30, 2020

Appl. No.:

16/864120

Inventors:

- Ann Arbor MI, US
Yunqi Zhang - Ann Arbor MI, US
Jonathan K. Kummerfeld - Ann Arbor MI, US
Parker Hill - Ann Arbor MI, US
Johann Hauswald - Ann Arbor MI, US
Michael A. Laurenzano - Ann Arbor MI, US
Lingjia Tang - Ann Arbor MI, US
Jason Mars - Ann Arbor MI, US

International Classification:

G06K 9/62
G06N 5/04
G06N 20/00
G06F 16/332

Abstract:

Systems and methods of intelligent formation and acquisition of machine learning training data for implementing an artificially intelligent dialogue system includes constructing a corpora of machine learning test corpus that comprise a plurality of historical queries and commands sampled from production logs of a deployed dialogue system; configuring training data sourcing parameters to source a corpora of raw machine learning training data from remote sources of machine learning training data; calculating efficacy metrics of the corpora of raw machine learning training data, wherein calculating the efficacy metrics includes calculating one or more of a coverage metric value and a diversity metric value of the corpora of raw machine learning training data; using the corpora of raw machine learning training data to train the at least one machine learning classifier if the calculated coverage metric value of the corpora of machine learning training data satisfies a minimum coverage metric threshold.

US Patent:

20200257856, Aug 13, 2020

Filed:

Feb 6, 2020

Appl. No.:

16/783604

Inventors:

- Ann Arbor MI, US
Parker Hill - Ann Arbor MI, US
Kevin Leach - Ann Arbor MI, US
Sean Stapleton - Ann Arbor MI, US
Jonathan K. Kummerfeld - Ann Arbor MI, US
Johann Hauswald - Ann Arbor MI, US
Michael A. Laurenzano - Ann Arbor MI, US
Lingjia Tang - Ann Arbor MI, US
Jason Mars - Ann Arbor MI, US

International Classification:

G06F 40/30
G06N 20/00
G06N 7/00
G06F 40/284

Abstract:

Systems and methods for synthesizing training data for multi-intent utterance segmentation include identifying a first corpus of utterances comprising a plurality of distinct single-intent in-domain utterances; identifying a second corpus of utterances comprising a plurality of distinct single-intent out-of-domain utterances; identifying a third corpus comprising a plurality of distinct conjunction terms; forming a multi-intent training corpus comprising synthetic multi-intent utterances, wherein forming each distinct multi-intent utterance includes: selecting a first distinct in-domain utterance from the first corpus of utterances; probabilistically selecting one of a first out-of-domain utterance from the second corpus and a second in-domain utterance from the first corpus; probabilistically selecting or not selecting a distinct conjunction term from the third corpus; and forming a synthetic multi-intent utterance including appending the first in-domain utterance with one of the first out-of-domain utterance from the second corpus of utterances and the second in-domain utterance from the first corpus of utterances.