Attention-Emotion-Fatigue Detection Based On Face Mesh Features Using Deep Learning

Abstract

Extended Abstract Research Problem/Questions – Modern applications in work safety, traffic management, and psychotechnical evaluation demand reliable real?time assessments of an individual’s cognitive state. Despite significant progress in single-parameter analyses (such as fatigue detection or emotion recognition), a gap remains in developing an integrated system that simultaneously evaluates emotion, fatigue, and attention. This study aims to address this gap. In addition, considering the amount of paper work required in deployment of the classical widespread manual D2 Attention Test [18] in measuring the attention level, this study aims to be one of the first studies to pioneer the automation of this test. Short Literature Review – Earlier studies show that a large portion of occupational and traffic accidents are related to attention deficit [1],[2], fatigue [3],[4] and high emotional state [5]. However, research has typically focused on individual aspects—usually only fatigue [6], [7], only attention [8] or only emotion [9] are analyzed separately. While these methods have yielded promising results, they often neglect the interdependencies between cognitive factors. Recent advances in deep learning and computer vision have enabled more nuanced detection capabilities; however, studies that merge these modalities remain scarce. Methodology – The proposed system is developed in Python by using several specialized libraries. – Fatigue Detection: Mediapipe is used for real-time detection of facial landmarks, enabling the extraction of parameters such as EAR, MAR, MOE (Mouth Opening Extent), and PERCLOS. These parameters are then fed to an LSTM model for fatigue classification. – Emotion Analysis: Deepface, supported by a convolutional neural network (CNN) pretrained on FER-2013 dataset and fine-tuned with domain-specific images, identifies subtle facial expressions. In addition, K-Means clustering algorithm was implemented to cluster the intensity levels (low, medium, and high) within each emotion class. – Attention Measurement: The system incorporates the D2 attention test framework, wherein eye-tracking data obtained via Mediapipe is used to assess fixation durations and detect attention lapses. – User Interface and Calibration: A user-friendly interface is developed with PyQt5. A calibration phase establishes individual baseline metrics to account for inter-subject variability. The integrated approach employs feature extraction, time series analysis, and clustering techniques (e.g., K-means) to quantify and categorize the cognitive states reliably. Results and Conclusions – Experimental evaluations demonstrate that the integrated system achieves approximately 94% accuracy in fatigue detection. This result offers a practical and effective solution for real-time applications with its high accuracy, low computational cost and person-independent generalization ability. On the other hand, attention measurement module shows stable performance and high correlation with the classical manual deployment scores. This result is promising especially considering that this is one of the first studies to pioneer the automation of this test. The emotion analysis module, optimized through hyperparameter tuning and CNN architecture search, reaches a mean average precision (mAP) close to 88%. These results validate that a multi-faceted approach not only enhances detection accuracy but also provides a comprehensive analysis of an individual’s cognitive state. The findings show significant potential for deploying such systems in environments where real-time monitoring is critical for safety and performance improvement.Bu çalışma, bireylerin anlık dikkat düzeyi, duygu durumu ve yorgunluk seviyesini entegre bir şekilde analiz eden yapay zeka tabanlı bir sistem geliştirmeyi amaçlamaktadır. Python diliyle geliştirilen sistemde, yüz ifadeleri ve göz hareketleri MediaPipe, DeepFace ve OpenCV gibi kütüphanelerle analiz edilerek; yorgunluk, dikkat ve duygu durumuna dair ölçümler elde edilmiştir. Kullanıcı dostu bir arayüz PyQt5 ile tasarlanmış, böylelikle tüm analizlerin arka planda gerçek zamanlı olarak gerçekleştirilmesi sağlanmıştır. Yorgunluk tespiti için EAR(Eye Aspect Ratio), MAR (Mouth Aspect Ratio), MOE (Mouth Opening Extent), PERCLOS (Percentage of Eye Closure) ve Blink Rate (göz kırpma oranı) gibi metrikler, LSTM tabanlı bir modelle analiz edilmiştir. Dikkat seviyesi, dijitalleştirilmiş D2 Dikkat Testi üzerinden göz takibiyle ölçülmüş ve geleneksel testle yüksek korelasyon göstermiştir. Böylelikle literatürde yaygın olarak kullanılan D2 Dikkat Testi'nin göz takip teknolojileriyle dijital ortama aktarılmasının mümkün olduğu gösterilmiştir. Bu yönüyle, dikkat testlerinin dijitalleşmesine öncülük eden ilk uygulamalardan biri olma özelliği taşımaktadır. Duygu analizi ise CNN ve K-Means kümeleme yöntemleriyle gerçekleştirilmiş, ve duygu sınıfının yanında duygu yoğunluğunun da belirlenebilmesi sağlanmıştır. Gerçekleştirilen deneylerde sistemin yorgunluk tespitinde %94, duygu analizinde %88 doğruluk oranı yakaladığı ve dikkat ölçümünde güvenilir sonuçlar verdiği görülmüştür. Çalışma, geleneksel tek yönlü analizlerin ötesine geçerek, entegre ve gerçek zamanlı bir analiz imkanı sunmakta ve dikkat testlerinin otomasyonu konusunda literatüre katkı sağlamaktadır.