Sign Language Page

Interaction Laboratory


Sign language recognition group
[poster](in Japanese)

Sign language

shuwas.jpg Sign language is a "Visual language" in which a sentence is expressed by facial expression and motion and shape of hands and fingers. It plays an important role as a way of communication of persons with hearing impairment when such a person supports himself/herself and has a social life.

Background and subject

5 percent of the population of Japan
It is said 6 that there exist 6 million people with hearing moderate impairment (where audiometric ISO value is greater than 40dB) in Japan. Since many peaple without hearing impairment cannot use sign language, peaple with hearing impairment may have to put up with discomfort.

Automatic recognition

The long term aim of the group is development of automatic recognition system for signlanguage. The group researches sign language recognition from a movie.


[1] Recognition of sign language words
We are developing an automatic recognition system of a sign language word motion by Hidden Markov Models (HMMs), which are trained by motion and shape of hands extracted by image processing. Since hands and a face often overlap together in a sign language motion, it is difficult to extract shapes of hands by simple image processing. We utilize template matching and detector trained with appearance of a hand to estimate position and shape of a hand.

We use a HMM with topology customized for each sign language word to reflect variety of motions for the word, though previously proposed methods often use a single topology for all words. We also research automatic training data synthesis, which synthesizes a training data from other samples that have partially similar motion or shape of a hand.

[2] Recognition of facial expression
We use neural network to recognize facial expressions from feature points extracted from a face on a movie. We have developed a program that can recognize happy, angry, sad, doubtful and impassive facial expressions. However, extraction of feature points on a face may be failed because hands may overlap a face in a movie of practical sign language. We try to improve the program so that it works on such a case.



    Assistant professor Tadashi Matsuo ( matsuo [at] )
    Bachelor student Yuka Nakajima ( ynakajima [at] )