copyright INRIA / Photos C. Lebedinsky

Multimodality modeling

Our goal is to obtain realistic strutured models from multimodal -possibly dynamic- data to be used in AR systems for interaction management, visualization or annotation. Two projects are described

Designing a multimodal acquisition system

  • Design of the system, calibration, synchronization and registration procedures:
    • Coupling electromagnetic sensors and ultrasound images for tongue tracking: acquisition set up and preliminary results, ISSP 2006 Pdf file .
    • Details on the system are available here .
  • Registration and processing of articulatory data

Building a realistic augmented head

We address the problem of obtaining realistic facial animation within the augmented head application. The main idea of this work is to transfer the dynamics learned on the sparse meshes of the face onto a 3D dense mesh acquired with a scanner.

  • Realistic face animation from sparse stereo meshes, Audiovisual Speech Processing 2007, Pdf file
  • Realistic face animation for audiovisual speech applications: a densification approach driven by sparse stereo meshes, Mirage 2009, Computer Vision / Computer Graphics Collaboration Techniques and Applications Pdf file .
  • A video showing the realistic animation.

Surgical workflow analysis

The focus of this work is the development of statistical methods that permit the modeling and monitoring of surgical processes, based on signals available in the operating room. The goal is to combine low-level signals with high-level information in order to detect events and trigger pre-defined actions. A main application is the development of context-aware operating rooms, providing adaptive user interfaces, better synchronization within the surgery department and automatic documentation.
  • On-line Recognition of Surgical Activity for Monitoring in the Operating Room , Proceedings of the 20th Conference on Innovative Applications of Artificial Intelligence (IAAI 2008) Pdf file
  • A video showing the annotation of a surgery from the model learned on exemplary recordings.