Human Centered Transportation Systems

Research on Driver’s Fatigue Detection

Preeti Bajaj, Narendra Narole, Mandalapu Sarada Devi
Intelligent Transportation Research Group
G. H. Raisoni College of Engineering, Nagpur, India


A focused group in various areas of Intelligent Transportation is working at G. H. Raisoni College of Engineering. The work undertaken relates to Drivers fatigue detection, their expression and mental state detection, designing multicore, embedded controller for in vehicle control, automotive network controller design, VANET etc. The work carried out in this article is the efforts of few masters and doctoral student carried out at the center. International statistics shows that a large number of road accidents are caused by driver fatigue. Therefore, a system that can detect driver fatigue and issue timely warning could help to prevent many accidents, and consequently save money and reduce personal suffering. In this article, various methods to detect drivers fatigue have been developed, which can help to increase vigilance of the driver and make him alert from fatigue state. These methods are non-intrusive technique and intelligence is provided by using soft computing tools.

Key words: Fatigue Detection, Fuzzy Logic Neural network, Genetic Algorithms, Support Vector Machine (SVM).

1. Introduction

The increasing number of traffic accidents due to a diminished driver’s vigilance level has become a serious problem for society. Statistics show that 20% of all the traffic accidents are due to drivers with a diminished vigilance level [1]. Furthermore, accidents related to driver hypo-vigilance are more serious than other types of accidents, since sleepy drivers often do not take correct action prior to a collision. For this reason, developing systems for monitoring driver’s level of vigilance and alerting the driver, when he is drowsy and not paying adequate attention to the road, is essential to prevent accidents. The prevention of such accidents is a major focus of effort in the field of Human Centre Transportation System. Lot of work has been cited in this area. The author along with her doctoral and post-graduate students is working in this area. The work carried out at the research center in last two years and has been reported in this article.

Driver’s state of vigilance can be characterized by driver performance with a focus on the vehicle behavior or based on driver’s physiological / physical state. The driver fatigue can be measured by intrusive and non-intrusive method. This article is focused on non-intrusive methods of fatigue detection. Possible non intrusive techniques for detecting drivers fatigue are

Research has carried out with the following parameters

Fatigue in driver can be well observed from changes in their facial features like eyes, head, mouth and face [2]. Some common assumptions in previous face related works were: frontal facial views, constant illumination, and the fixed lighting source. Unfortunately these assumptions are not realistic. In the application of real world facial expression understanding, one has to consider at least three issues: capturing the full features in a variety of lighting conditions and head motion, multiple and non rigid object tracking, and the self-occlusion of features.

2. Fatigue Detection from Eye Movement

To start the research, fatigue detection was carried out with the simple skin color based algorithm. In this approach a facial image is given as input and the skin color based algorithm [3] is applied to detect the face in the image. The top and sides of the face are detected to narrow down the area of where the eyes exist. Using the sides of the face, the centre of the face is detected, which is used further as a reference while comparing the left and right eyes. Moving down from the top of the face, horizontal averages (average intensity value for each x coordinate) of the face area are calculated. Large changes in the averages are used to define the eye area. Using the horizontal average values of both sides of the face the open or closed state of the eyes are detected. If the eyes are found closed for 5 consecutive frames, the system draws the conclusion that the driver is in micro sleep also the system detects the blink frequency and if it not as per the normal limit the warning signal is generated.

2.1 Face Detection

a)Original image (b) Gray scale image (c) Detected face

(a)Original image (b) Gray scale image (c) Detected face
Figure 1 Face Detection

Each pixel in the given image is classified as a skin pixel or a non-skin pixel. The different skin regions in the skin-detected image are identified by using connectivity analysis to whether each region identified is a face or not as shown in the Figure 1.

2.2 Eye detection and eye state estimation

The next step in locating the eyes would be finding the intensity changes on the face. This is done using the gray scale image and not the RGB image. The first step is to calculate the average intensity for each x – coordinate. These average values are found for both the eyes separately. When the plot of these average values was observed it was found that there are two significant intensity changes. The first intensity change is the eyebrow, and the next change is the upper edge of the eye, as shown in the figure. Thus with the knowledge of the two valleys the position of the eyes in the face was found. The state of the eyes (whether it is open or closed) is determined by distance between the first two intensity changes (valleys) found in the above step. When the eyes are closed, the distance between the x – coordinates of the intensity changes is larger if compared to when the eyes are open as shown in Figure 2.

Figure 2. Average intensity variation on the face when eyes are open and close

2.3 Drowsiness detection

For drowsiness detection each frame is observed to check whether the eyes are closed or open. Thus, the eye blink frequency is determined and if the eye blinks increases beyond the normal limit, the alarm is activated. Also if the eyes are found closed for consecutive 5 to 6 frames the system decides that the eyes are closed and give a fatigue alert. Thus in this approach the fatigued state of the driver is detected with the help of the horizontal intensity variation on the face. One similarity among all faces is that eyebrows are significantly different from the skin in intensity, and that the next significant change in intensity, in the y-direction, is the eyes. This facial characteristic is the centre of finding the eyes on the face, which will allow the system to monitor the eyes and detect long periods of eye closure.

3. Fatigue Detection with Support Vector Machine (SVM)

SVM (Support Vector Machine) is a useful technique for data classification [4]. A classification task usually involves with training and testing data which consist of some data instances. Each instance in the training set contains one “target value" (class labels) and “several attributes" (features). The goal of SVM is to produce a model which predicts target value of data instances in the testing set which are given only the attributes. Complete Block diagram representation of proposed system is as shown in Figure 3.

Figure 3 Proposed System Using SVM

3.1 Mouth Detection Algorithm

In this approach mouth detection procedure classifies images based on the value of simple features. This image-based detection algorithm works on uncompressed images and has proven to be robust under various lighting conditions. The method is based on a cascade of boosted classifiers of simple Haar-wavelet like features on different scales and positions[5]. The features are brightness and contrast-invariant and consist of two or more rectangular region pixel-sums that can be efficiently calculated by the canny integral image. The feature set is over complete and an adaptation of the AdaBoost learning algorithm is proposed to select and combine features into a linear classifier. To speed up detection a cascade of classifiers is used such that every classifier can reject an image. All classifiers are trained to reject part of the candidates such that on average only a low amount of features are used per position and scale. After all possible mouth candidates are obtained, a grouping algorithm reduces groups of mouth candidates into single positive detections. Figure 4 displays the list of Images used, Mouth Detection and extracted Mouths using the classifier.

Figure 4 Mouth Detection and Extraction

3.2 Fatigue Detection

Fatigue Detection has got two phases, one is training phase and the other one is detection phase. Figure 5 and Figure 6 lists the Training set of mouths used for SVM.

Figure. 5 Normal Mouths for SVM Training

Figure.6 Yawning Mouths for SVM Training

SVM requires that each data instance is represented as a vector of real numbers. Hence, if there are categorical attributes, then those have to convert into numeric data first & scaling them before applying to SVM is very important. In this part of research few videos and about 20 yawning images and more than 1000 normal images are selected as the data set. From each video, 10 yawning images and 10 normal images are given to cascade of classifiers for training. Same samples of Video are trained by SVM to classify normal and yawing mouths.

Then the proposed approach is tested and the results of classification are presented in table 1.

Table1: Results of classification of Normal and yawning mouths

4. Fatigue Detection Using Fuzzy Based System

In this approach Fuzzy inference system make use of states of eyes and mouth as inputs to analyze the fatigue state of the driver. The system is based on the assumption that the driver's head exhibits very small displacements in the time required to grab and process consecutive input frames. During tracking, error-detection is performed in order to recover from possible tracking failure. If an error occurs, the system gets out of tracking mode and recalculates the locations of the eyes from the active image frame. The fatigue detection phase is closely related to the tracking phase. The information obtained from processing each consecutive input frame in the eye-tracking phase and mouth tracking phase are fed directly to the fatigue detection phase if no tracking error has occurred. This phase is responsible of extracting the information from the images of the eyes and mouth those will determine any signs of micro-sleeps and yawning. At each frame, when the eyes are localized, the system determines whether the eyes are open or not. Similarly At each frame, when the mouth is localized, the system determines whether the mouth is open or not. The numbers of consecutive frames are counted during which the mouth is opened, if this number gets too large yawning is considered.

4.1 The Fatigue Detector

As the driver's fatigue increases, his/her eye blinks and mouth opening tends to last longer. The system can determine the blink rate by counting the number of consecutive frames in which the eyelashes remain closed. The area of black pixel region in open mouth and closed mouth are different. Similarly the area of the black pixel region in the case of open eye is different from that of closed eye. These areas of eye and mouth regions are given inputs to the fuzzy inference system. The fatigue detection system is Mamdani fuzzy inference system with two inputs and one output as shown in Figure 7 (a) (b) & (c). The eye and mouth tracker outputs are given as inputs to the fuzzy inference system. Three partitions are made on eye tracker output so it has three membership functions named with the linguistic variables blink, sleepy and slept and two partitions are made on the mouth tracker output so it has two membership functions named with the linguistic variables normal and yawn. The sample rules could be

If (Eye Blink) and (Mouth is Normal) then (Output is FIT)(1)

If (Mouth is yawn) then (Output is Fatigue )(1)

Figure.7.(a) Input1(Eye state)

Figure.7 (b)Input2 (Mouth state)

Figure.7(c) Output of the fuzzy system (fatigue state)

To compute the final output centroid defuzzification is proposed. Figure 8 and Figure 9 shows the fatigue detector when the person is normal and fit for driving & also shows how the system signals a "fatigue alert" when the eyes have been closed for many consecutive input frames and/ or mouth is open for many consecutive frames (3-4 seconds)

Figure.8 Output1

Figure 9 Output2

This Fuzzy based Fatigue Monitoring system is tested with drivers of different skin color, with facial hair and of different gender. The system is able to complete the eye localization and tracking at 15 frames per second. For small head-movements, the system rarely loses track of the eyes and mouth. Under these circumstances, the system was able to detect prolonged eye blinks and yawning in 95% of the times and it produced occasional false alarms. For better results system need is to add the capability of auto-zoom on the eyes, once they are localized. This would avoid the trade-off between having a wide field of view in order to locate the eyes and a narrow field of view in order to detect fatigue

5. Real time Fatigue Detection with Hybrid Algorithm

The block diagram representation of the Neural Genetic system is as shown in Figure 10. The skin color based algorithm is applied to detect the face to the input image [6 - 10].

Figure 10: Block Diagram of Neuro-Genetic System

Using the side of the face, eyes portion and mouth portion of the image is separated from which the open or closed state of the eyes are detected along with yawning count. These two parameters are given as the input to a Neuro genetic system which generates the fatigue probability. In this system AVI file is used, which is converted to frames and are selected for further processing & the next step will be of eye and lip detection.

5.1 Face parameter Detection

For Yawning Count, [11] first process is to identify the lip pixels and measure the lip pixel count. The lip pixel can be identified by using R/G ratio which has different values for skin and for lips, in this system for lips count measurement only lower 50 % portion of frame is considered The lip count will have different value in different frames of video for 50% of image because when person is yawning, it may cause disappearance of upper lip portion. Thus Lip count will get affected and by using proper threshold value of lip count the state of driver can be easily differentiated i.e. yawn, no yawn state of driver as shown in Figure 11. The total 300 frames in the group of 100 are considered and corresponding eye close count, yawn count & fatigue probability is calculated.

Figure 11: Driver with during various condition

i) Neuro genetic System

In this system a feed-forward neural network with a back-propagation learning algorithm is used[12]. This is illustrated in Figure 12. In real-world applications, the back-propagation algorithm might converge to a set of sub-optimal weights from which it cannot escape. As a result, the neural network is often unable to find a desirable solution to a problem at hand.

Figure.12: Neural Network Structure used in the proposed system

Initial Structure of Neural network defined is two layer feed forward network having an input (ranging from 0 to 8), followed by a layer of 10 TANSIG neurons, followed by a layer with 1 PURELIN neuron. TRAINLM back propagation is used. The network is trained and simulated. Sample input and target data used for training of neural network is as given below

Input Data = [Eye Blink Count    Yawning Count]

p = [  40    5   ;   50     7   ;   30    4   ;   50    12   ;    35   7   ]

Target assigned for the above input

t = [.5 0 0 0 0;    .7 0 0 0 0;    .8 0 0 0 0;    .2 0 0 0 0;   .1 0 0 0 0];

The data of the eye close count and yawn count are fed to the neural network. The training process of the neural network begins with the selection of any ten random neuron numbers in the specified limit. This is done to find the best possible number of neurons to be present in the hidden layer with the help of the genetic algorithm. Ten randomly generated number act as members of initial population. Those random decimal numbers are converted in their four bit binary equivalent number. In crossover operation two LSB digits and MSB digits get swapped with the corresponding bits of next successive member, this is a repetitive process between groups of two successive members, which results in a new generation of population after crossover. These members will act as a neuron number used in hidden layer of neural network. After applying these newly generated members as a neuron number; best four members are further selected, where the selection criterion is based on the reciprocal of the error value between desired target value and actual output generated by the neural network. In mutation process these four best members which act as a neuron in hidden layer of Neural network are used and random changes are made in their binary equivalent. Neuron generated after cross over and mutation operation can become zero sometimes which results in loss of hidden layer and to avoid this probability of error, three neurons are taken as a default value. Thus this resultant optimized structure network is trained and the inputs are provided, then it carried out the calculation for the drowsiness on the basis of the training provided to it. The output about the state of the driver is then displayed with corresponding fatigued probability and comparative analysis is perform with the fixed layer Neural network having ten numbers of hidden layer neurons.

Initial Randomly Generated Population

Initial members = (1111) – Decimal Equivalent = [15]
Initial members = (0100) – Decimal Equivalent = [04]
Initial members = (1010) – Decimal Equivalent = [10]
Initial members = (1000) – Decimal Equivalent = [08]

Population after Crossover Operation

Crossed members = (1100) – Decimal Equivalent = [12]
Crossed members = (0111) – Decimal Equivalent = [07]
Crossed members = (1000) – Decimal Equivalent = [8]
Crossed members = (1010) – Decimal Equivalent = [10]

In mutation, random changes are made so the four members generated after crossover and mutations are totally random in nature. The comparative result between fixed structured NN and Genetic process based optimized structured NN is as shown in Figure 13, after performing number of iteration the best result is obtained which is approximately close to the desired target.

Figure 13: Results

6. Conclusion

The experimental results indicate that the proposed expert systems are effective in increasing safety while driving. In this research work the fuzzy inference system is able to automatically detect fatigue in the driver by monitoring the eyes for micro-sleeps and mouth for yawning also SVM is successively used to classify the mouth regions to detect yawning and Neuro Genetic system can be used to give more intelligence for detecting the fatigue of the driver.

7. References

[1] Quong Wang, Jingyu Yang, Mingwu Ren, Yujie Zheng, “Driver Fatigue Detection: A Survey”, in Proc Intelligent control and Automation, Dalion, China, pp 8587- 8591, 2006.
[2] L.M. Bergasa, J.Nuevo, M.A. Sotalo, and M. Vazquez, “Real-time system for monitoring driver Vigilance”, in Proc. Intelligent Vehicle Symp., Parma, Italy, pp.78-83, 2004.
[3] N. Parmar, “Drowsy Driver Detection System”. Engineering Design Project Thesis, Ryerson University, 2002.
[4] Christopher J. C. Burges. “A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery”, 2(2):121{167, 1998
[5] Boser, B. E., I. Guyon, and V. Vapnik (1992). “A training algorithm for optimal margin classifiers”. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pp. 144{152. ACM Press.
[6] S. Jin , S.-Y. Park J.-J. Lee “Driver Fatigue Detection using Genetic Algorithm.” 11th International Symposium on Artificial Life and Robotics, Oita, Japan, January 23–25, 2006
[7] R.S. Parsai, Dr.P.R.Bajaj “Intelligent Monitoring System for Driver’s Alertness (A vision based approach)”, KES 2007 International Conf.Sept-07, Italy
[8] Sanjay Kr. Singh1, D. S. Chauhan2, Mayank Vatsa3, Richa Singh3*, “A Robust Skin Color Based Face Detection Algorithm” Tamkang Journal of Science and Engineering, Vol. 6, No. 4, pp. 227-234 (2003) 227, Department of Computer Science and Engineering, Institute of Engineering and Technology, Jaunpur−222002, India
[9] K. Yammamoto and S.Higuchi, J.Soc “Development of drowsiness warning System”. Automotive Eng.Japan,pp. 127-133
[10] M.Yang, D.J.Kriegman and N.Ahuja, “Detecting faces in images: A survey”, IEEE Trans. Pattern Analysis and achine Intell.24, 2002, pp. 34-58.
[11] N. Eveno, A. Caplier and P.Y. Coulon, “A New Color Transformation For Lips Segmentation ” . In Proc. MMSP’01, pp. 3-8, 2001
[12] F. Smach, M. Atri, J. Mitéran and M. Abid, “Design Of A Neural Networks Classifier For Face Detection”, Journal of Computer Science 2 (3): 257 260, 2006,ISSN 1549-3636,© 2006 Science Publications

About the Authors

Dr. Preeti Bajaj holds a PhD degree in Electronics Engineering. Currently she is Professor of Electronics & Principal of G.H.Raisoni College of Engineering, Nagpur, India. Her research interests include Intelligent system design, Applications of Fuzzy logic in areas of Intelligent Transportation Systems. She is co-chair of the IEEE Technical committee on Human Centered Transportation System of SMCS .She is Senior Member- IEEE. She also Chair SMCS Chapter, IEEE Bombay Section, Region-10, Asia Pacific. She has worked as TPC member for more than 30 International Conferences. She is General Chair for International Conference on Emerging Trends in Engineering & Technology (ICETET). She has over 65 publications in International Conferences and Journals. She has offered consultancy to National Highways Authority of India for designing and proposing various Intelligent Transportation projects on National Highways in India.

Mandalapu Saradadevi received B.Tech degree in Electronics & Communication Engineering from V.R.Siddhardha Engineering college, Vijayawada and M.E in Electronics & Communication Engineering from Andhra university, Visakhapatnam. She is currently pursuing PhD under Dr. Preeti Bajaj She has been with B.D.College of Engineering, Sevagram, India since 1994. She is working on Design and Development of some methodology for driver monitoring system with non-intrusive approach.

Narendra G. Narole, received the B.E. and M.Tech degrees in Electronics Engineering from GHRCE, Nagpur, R. T. M. Nagpur University in 2001 an 2008, respectively. His research interest includes signal processing, Artificial Intelligent and their application in areas of Intelligent Transportation. He is a student member of IEEE.