Математичне та програмне забезпечення комп’ютерної системи ідентифікації елементів української жестової мови

Abstract

Дисертація присвячена розробленню математичного та програмного забезпечення комп’ютеризованої системи ідентифікації жестів української жестової мови у реальному часі. Розроблено нову модифікацію методу навчання нейронних мереж зворотним поширенням похибки, яка дозволила на 27% зменшити кількість помилок розпізнавання кінців пальців долоні та на 44% зменшити час навчання, порівняно з методом спряжених градієнтів. Розроблено новий метод вибору навчальних прикладів для навчання нейромережевого класифікатора, який дозволив застосувати технологію інтерактивного навчання з відео. Розроблену модель псевдодвовимірної неперервної деформації зображення використано для порівняння форми долоні з еталоном, що дало змогу підвищити відсоток правильного розпізнавання форми долоні з 87% до 94%. Розроблена комп’ютеризована система ідентифікації жестів української жестової мови на тестовому наборі з 85 жестів правильно розпізнає 92% жестів. Диссертация посвящена разработке математического и программного обеспечения компьютеризированной системы идентификации жестов украинского жестового языка в реальном времени. Разработана новая модификация метода обучения нейронных сетей обратным распространением ошибки, которая позволила на 27% уменьшить количество ошибок распознавания концов пальцев ладони и на 44% уменьшить время обучения, по сравнению с методом сопряженных градиентов. Разработан новый метод выбора обучающих примеров для обучения нейронных сетей, который позволил использовать технологию интерактивного обучения нейронных сетей с видео. Разработанная модель псевдодвумерной непрерывной деформации изображения использована для сравнения формы ладони с эталоном, что позволило увеличить процент правильно распознанных форм ладони с 87% до 94%. Разработанная компьютеризированная система идентификации жестов украинского жестового языка на тестовом наборе их 85 жестов правильно распознает 92% жестов.The thesis is dedicated to development of computational methods and software of the computerized real-time Ukrainian sign language identification system. The problem of sign language to text translation is not solved not only in Ukraine but all over the world. A valuable contribution to development of foreign languages sign recognition models was made by T. Starner, H. Ney, J. Zieren, A. Fitzgibbon, H. Stern, T. Coogan, C. Vogler, R. Bowden, A. Farhadi, R.-H. Liang. The proposed recognition model differs from existing foreign language recognition models by hand shape recognition in motion. The system utilizes one video camera as a sensor. The software of the system consists of real-time video processing application “IMPROC”, gesture identification application “Sign”, computerized “Ukrainian sign language trainer”. New algorithms and methods are proposed for the purpose of the system development. The task of Ukrainian gesture language recognition and translation to written language was analyzed. The advantages and disadvantages of known foreign sign language recognition systems were studied. The most difficult task in real-time sign language recognition is hand shape identification. Most of the systems do not consider signs that differ by hand shape only. The structure of software solution for this task was proposed. The proposed system structure utilizes fingertip position information and pseudo 2-dimentional continuous image deformation model for hand shape recognition. Two skin segmentation methods for hands tracking were developed. The first method is based on neural network classifier. The second is based on diffusion light equation. Fingertip recognition is done by means of the neural network classifier. New modification of back propagation neural network training algorithm was developed. Via the modification the fingertip recognition error rate was reduced by 27% and the teaching time was reduced by 44% comparing with conjugate gradient method. New method for sample selection for neural networks teaching was developed. The method allows use of new technology for interactive neural networks teaching from video. The fast pseudo 2-dimensional continuous image deformation model was developed for hand shape recognition. The proposed model is faster than pseudo 2-dimentional hidden Markov models (P2HMM) and pseudo 2-dimentional hidden Markov models with deformation model (P2HMMDM) and could be used even if one sample for every hand shape is available. For better hand shape extraction the new penalty function is proposed to compare image pixels and hand shape sample pixels. The proposed function utilizes two penalty functions – function of fuzzy penalty for pixel difference between image pixel and opaque sample pixel and function of fuzzy penalty for resemblance of image pixel and the nearest opaque sample pixel in case of pixel comparison to transparent sample pixel. By the means of proposed methods the percent of properly recognized hand shapes increased from 87% up to 94%. The sign identification method is based on hidden Markov models. The one-dimensional hidden Markov model with 8 states is used. Such number of states is enough to describe complex gestures with several hand shapes. For every sign a model for left and right hand is created. The Baum-Welch algorithm is used for HMM training and forward algorithm is used for gesture verification for model fitness. The developed sign identification software utilizes proposed algorithms. The best achieved result of separate signs recognition is 92% of test set that contains 85 signs. The result achieved is close to sign language recognition results achieved by foreign scientists. For the precise comparison of results testing on common test set is required, but it is impossible because open foreign sign language test sets are captured by monochrome camera and are unsuitable for processing by methods proposed in this thesis. The sign identification module is implemented in interactive “Ukrainian sign language trainer”. The proposed trainer version consists of sign language dictionary with gesture video records. The trainer has means for video rendering with different speed and frame-by-frame review. The interactive part of the trainer is developed for sign language gesture verification. Special functions allows student to see his own image from camera. Is helps to synchronize student signs and signs form vocabulary. The gesture recognition algorithm identifies proper and improper gesture execution and signalizes about it to user. The interactive “Ukrainian sign language trainer” is inculcated in Lviv consulting center for special need children based on “School-gymnasium “Syhivska” and Lviv specialized secondary boarding school for hearing impaired children.