Safari Books Online is a digital library providing on-demand subscription access to thousands of learning resources.
An Efficient Unification-Based Multimodal Language Processor for Multimodal Input Fusion Among multimodal fusion approaches, early fusion is only useful for pairs of inputs that are closely coupled. Hence, a late fusion component is needed to handle the diverse input modalities in a multimodal system. PUMPP, a multimodal lan- guage processor which integrates symbol strings recognized by signal recognizers from multiple modalities, is proposed to meet this requirement. It targets applications in which complementary multimodal inputs can be recognized to symbols and occur sequentially or simultaneously. By sophisticatedly disclosing the peculiarities of multimodal utterances from our point of view, we established the prerequisite of PUMPP. In a multimodal utterance, the linear order of inputs from multiple modalities is variable; however, the linear order from one modality is invariable. To tackle issues about exponential computational complexity in multimodal parsing, PUMPP al- gorithm is designed to parse two parallel input may not derive a unique parsing result because one multimodal utterance can have several se- mantic interpretations. In future work, we plan to introduce probabilistic parsing to mitigate the ambiguity in parsing results. To utilize the PUMPP result, we propose a concept, subsumption on hybrid logic, to hinge multimodal input fusion and output generation in an agent-based multimodal presentation system. In it, the subsumption on feature structure is adapted to hybrid logic to check the generaliza- tion of one hybrid logic formula over another one. That enables a system to respond to multimodal utterances flexibly. The preliminary experiment result supports its flexibility on the system perfor- mance. We also observed the overall performance of the system is closely related to other modules in the system. There are a couple of directions that can be explored in the future. The first is to research how