검색 상세

Development of a Notation Method for Hand Gesture Vocabularies based on a 3D Free Hand Gesture Taxonomy

초록/요약

In this study, I developed a notation method that can organize and notate hand gestures in a systematic manner. The specific objectives are as follows: 1) the development of a taxonomy of hand gestures and a notation method; 2) the development of a text-based notation method of hand gestures and its’ verification; 3) the development of learning models for pattern classifications of various hand gesture vocabularies matched with one command and verification of the models. For organizing hand gestures in a systematic manner I defined basic elements of hand gestures and then derived sub-elements of the basic elements by analyzing related studies. Subsequently I developed the 3D free hand gesture taxonomy based on the elements and sub-elements, and devised the notation method based on a combination of the elements that matched an integer code for easy notation. In this study, I defined that a hand gesture is a successive combination of posture(s) and movement(s). A posture is defined as a particular hand shape represented by a single image that consists of seven elements: number of hand(s) involved, hand type, special relationship between the hands (only for two-handed gestures), hand location, hand shape, hand orientation and arm posture. Movement refers to elements that make dynamic gestures, and consists of two elements: path movement and wrist movement. The text-based notation method including all of the elements in the taxonomy was devised to notate hand gestures by hand and decode the hand gestures using the notated codes. Through an additional experiment of deriving gestures from users and a pilot test, the additional elements were added into the taxonomy and also the notation method was modified to include them. Finally, the usefulness of the notation method was verified by training participants to notate hand gestures and by asking another set of participants to decode the notated gestures. As a result, except for arm posture (shoulder angle (SA), elbow angle (EA)), fleiss’ kappa of all of the elements was 1 which shows almost perfect agreement among participants. In the case of SA and EA for both hand, fleiss’ kappa was 0.59 (moderate agreement) to 0.99 (almost perfect agreement). This results shows that the same hand gestures can be notated and decoded across the users. However, this research only dealt with 11 commands (22 gestures) of a music player for the experiments, so further experiments with additional commands should be used for verification of the taxonomy and notation method. Finally, a future study will regard additional elements such as the size or speed of hand gestures that I have not considered in this study. Based on the notation method, I developed the learning models to classify patterns of gesture vocabularies for smart-home appliances and then verified the learning models. First, 1200 hand gestures for a total of 24 commands of 7 products, TV, light, air-conditioner, faucet, window(s), blind(s), and door(s) were derived from 70 participants. Based on the user-defined gestures a hand gesture library was established. Second, the notation method was modified to allow use of user-defined gestures as input data for computer analysis. Then the user-defined gestures in the library were notated using the modified method. Third, a hand-gesture learning model for each target product was developed using artificial neural networks. Finally, the developed models were validated experimentally in tests on a new group of users. The total hit rates were more than 98% for all of the models used for 75% seen data and 25% unseen data. These results show that the selected elements including the notation method are efficient to classify various gestures, and also that one command can be mapped to several gesture vocabularies. Due to the inaccuracy of currently-available gesture-recognition equipment, the researcher notated the user-defined gestures by observation only. The suggested models assumed 100% accuracy of the equipment, whereas in reality the final hit rate for the models is dependent on the accuracy of the equipment. Also the models did not consider the speed of hand(s) because a researcher notated the gesture visually. In future studies the above issues should be considered to verify the models. The hand gesture taxonomy and the notation method establish a foundation for a systematic approach for organizing hand gesture vocabularies. It has made the following contributions. First, this research has provided a thorough process for developing the hand gesture taxonomy and the notation method so that further research on improving the taxonomy and notation method can be conducted more easily. Second, textual records help the experimenter to find what he/she wants to identify at a glance without any extra system, such as a video player or a computer. In this respect, the notation method can be seen as a complement to or an alternative method for video recording. Third, pattern classifications by a computer are available because the notation method is based on the combination of elements matched with integer codes. In addition, if all of the elements suggested in the taxonomy can be recognized by equipment, the notation method could be useful to enhance the recognition rate of hand gestures. This is because the notation method is based on the small number of elements which can be combined to form a large number of hand gesture vocabularies (scalability). In short, the result of this study may suggest a good starting point for the further research on organizing and designing hand gesture vocabularies.

more

목차

Chapter 1. Introduction 1
1.1. Background 1
1.1.1. Hand gestures for natural user interface in smart-home 1
1.1.2. Benefits of hand gesture interaction 3
1.2. Motivation 6
1.3. Thesis objectives 9
1.4. Thesis structures 11
Chapter 2. Related Works 13
2.1. Classification of hand gestures 13
2.1.1. Definitions of hand gesture vocabulary 13
2.1.2. Semantic gestures 16
2.1.3. Descriptive gestures 20
2.2. Gesture recognition 24
2.2.1. Template matching method 24
2.2.2. Model based method 25
2.2.3. Appearance based method 28
2.2.4. Statistical analysis method 30
2.3. Artificial neural network for hand gesture recognition 31
2.3.1. Artificial neural networks 31
2.3.2. Learning Rule 33
Chapter 3. Development of a 3D free hand gesture taxonomy 35
3.1. Objectives 35
3.2. Methods and Procedure 35
3.3. A 3D free hand gesture taxonomy 36
3.3.1. Posture 37
3.3.2. Movement 43
Chapter 4. Development of a notation method 46
4.1. Objectives 46
4.2. The notation method of hand gestures 46
4.3. Encoding gesture elements 49
4.3.1. Code for posture 49
4.3.2. Code for movement 53
Chapter 5. Development of a text-based notation method 56
5.1. Objectives 56
5.2. Method 56
5.3. Refinement of the taxonomy and the notation method 57
5.3.1. Experiment 1: Acquisition of hand gestures 57
5.3.1.1. Participants 57
5.3.1.2. Methods and Procedures 57
5.3.1.3. Results 58
5.3.2. The text-based notation method 59
5.3.3. Experiment 2: Pilot test for notating user-defined gestures 60
5.3.3.1. Participants 60
5.3.3.2. Methods and procedure 61
5.3.3.3. Results 63
5.3.4. A final 3D free hand gesture taxonomy and text-based notation method 65
5.4. Validation of the notation method 66
5.4.1. Methods and procedure 66
5.4.2. Participants 68
5.4.3. Results 68
5.4.3.1. The agreement among the participants in the 1st experiment 69
5.4.3.2. The agreement among the participants in the 2nd experiment 70
5.4.4. Discussion 71
Chapter 6. Establishment of a hand gesture library 75
6.1. Objectives 75
6.2. Finding target commands 75
6.3. Acquisition of hand gestures from users 77
6.3.1. Methods 77
6.3.1.1. Participants 77
6.3.1.2. Procedures 78
6.4. Establishment of hand gesture library 79
Chapter 7. Modifying the notation method and notating gestures in the library 83
7.1. Objectives 83
7.2. Scope of hand gestures 83
7.3. Selecting main gesture elements 84
7.4. Modifying the notation method 85
7.4.1. A gesture notation method for classification 85
7.4.1.1. Modifying the encoding rule of hand shape 88
7.5. Notating user-defined hand gestures in the library 89
Chapter 8. Development of a hand gesture learning model 91
8.1. Objectives 91
8.2. Methods 91
8.3. Learning model for target products 93
8.3.1. Learning model for TV 93
8.3.2. Learning model for Light 95
8.3.3. Learning model for Air-conditioner 96
8.3.4. Learning model for Faucet 98
8.3.5. Learning model for Window(s) 100
8.3.6. Learning model for Blind 101
8.3.7. Learning model for Door(s) 102
8.4. Comparison of classification methods for pattern classification 104
Chapter 9. Evaluating and updating the developed learning models 106
9.1. Objectives 106
9.2. Validating the learning models 106
9.2.1. Methods 106
9.2.2. Results and Analysis 110
9.3. Updating the learning models 113
9.3.1. Methods 113
9.4. Results 115
9.4.1. Modified learning model for faucet 115
9.4.2. Modified learning model for window(s) 117
9.4.3. Modified learning model for blind 119
9.4.4. Modified learning model for door(s) 121
Chapter 10. General discussion 124
10.1. Development of a taxonomy of hand gestures and a notation method 124
10.1.1. The advantage of the notation method 124
10.1.2. Extensibility of the notation method 126
10.2. Development of learning models for gesture pattern classification 127
Chapter 11. Overall conclusion 130
Reference 137

more
반출 Meta View 목록