Pekiştirmeli öğrenme ile robot kol yörünge kontrolü
Yükleniyor...
Dosyalar
Tarih
2022
Yazarlar
Dergi Başlığı
Dergi ISSN
Cilt Başlığı
Yayıncı
Erişim Hakkı
info:eu-repo/semantics/openAccess
Özet
Robotların kullanımı son yıllarda teknolojinin gelişmesine bağlı olarak yaygınlaşmakta ve görevlerini otonom olarak gerçekleştirmeleri ile ilgili çalışmalar da gün geçtikçe artmaktadır. Verilen işleri insanlara nazaran daha güçlü ve hassas olarak yapabilme potansiyeli olan robot kolları, endüstriyel tesislerin yanı sıra ameliyathanelerde ve uzay görevleri gibi farklı alanlarda da kullanılarak geniş bir alanda hizmet vermektedir. Robot kol kontrolü problemine literatürde mevcut bulunan model tabanlı yaklaşımlar, fiziksel sistemi ifade eden matematiksel modelin oluşturulması ve kontrol algoritmalarının geliştirilmesini gerektirmektedir. Bu matematiksel modellerin eldesi her zaman kolay olmamakla birlikte çözümlerinin yapılamaması gibi durumlar bulunabilmekte ve bu yöntemler istenen performansı gösterememektedir. Günümüz teknolojisinin son bahis konularından olan yapay zekâ bu durum için bir çözüm ve kolaylık sağlama potansiyeli barındırmaktadır. Bu alanda etkili bir çözüm makine öğrenmesinin bir alt dalı olan Pekiştirmeli Öğrenmedir (PÖ). PÖ, herhangi işlenmiş bir veriye ihtiyaç duymadan, keşif yaparak ve bu keşifte ortaya konulan davranışları belirli kriterler doğrultusunda değerlendirerek öğrenme süreciyle optimum davranışları öğrenir. Böylece matematiksel modellemelere ihtiyaç duymadan bir kontrol seçeneği sunar. Bu yönüyle PÖ endüstriyel robotlardan insansız hava araçlarına kadar birçok alanda araştırılmış ve geliştirilmeye devam etmektedir. Bu çalışmada matematiksel modelden bağımsız olarak robot kol yörünge kontrol problemi için PÖ algoritmaları kullanılmıştır. Sürekli durum ve eylem uzayları için geliştirilmiş model-bağımsız, politika-dışı, aktör-kritik PÖ algoritmaları, derin deterministik politika gradyanı (DDPG), ikiz gecikmeli politika gradyanı (TD3) ve soft aktör-kritik (SAC) kullanılmıştır. Bu algoritmalar kendi aralarında eğitim süreçleri ve eğitilmiş ajanların simülasyonu aracılığıyla görevleri yerine getirmelerindeki doğru konumlandırma, istenilen sürede yerine getirme gibi çeşitli parametrelerce kıyaslanmıştır. Robot kolun bulunduğu çevreye sabit engeller konularak engelli çevre oluşturulmuş ve bu engelli çevre için eğitim süreçleri ve simülasyonlar tekrarlanmıştır. Çevrede bulunan engel ile çarpma gibi bozucu bir sinyalin eğitim süreçlerine ve ajanların davranışlarına etkileri gözlemlenmiştir. Algoritmalar ile ajanların hazırlanması, eğitim ve test süreçleri Matlab programı kullanılarak gerçekleştirilmiştir. Robot kol modeli Matlab kütüphanesinden alınmış gerekli modifikasyonlar ve çevre modellemesi yine Matlab'ın Simulink/Simscape arayüzü kullanılarak yapılmıştır. Sonuç olarak algoritmaların verilen görevleri yüksek bir başarı oranı ile yerine getirmeyi öğrenebildiği görülmüştür.
Robots have become widespread in recent years due to the development of technology, and studies on performing given tasks autonomously are increasing daily. Robot arms, which have the potential to achieve given tasks more powerfully and more precisely than humans, are used in a wide range of areas, such as operating rooms and space missions, as well as industrial facilities. Model-based approaches to the robot arm control problem in the literature require creating a mathematical model expressing the physical system and developing control algorithms. However, these mathematical models are not always easy to obtain, there may be situations such as being unable to solve them, and these methods do not show the desired performance. Artificial intelligence, which is one of the last topics of today's technology, has the potential to provide a solution and convenience for this situation. A practical solution in this area is Reinforcement Learning (RL), a sub-branch of machine learning. Reinforcement learning learns the optimum behaviours through the learning process by making discoveries and evaluating the behaviours revealed in this discovery according to specific criteria, without needing any processed data. Thus, it provides a control option without the need for mathematical models. In this respect, RL has been researched and developed in many areas, from industrial robots to unmanned aerial vehicles. This study used RL algorithms for the robot arm trajectory control problem, regardless of the mathematical model. Deep deterministic policy gradient (DDPG), twin-delayed policy gradient (TD3) and soft actor-critic (SAC) algorithms which are model-free, off-policy, actor-critic reinforcement learning algorithms that developed for continuous state and action spaces, were used. These algorithms were compared among themselves by various parameters such as correct positioning and fulfilment in the desired time in performing tasks through training processes and simulation of trained agents. A second environment with obstacles was created by placing fixed obstacles in the environment where the robot arm is located, and training processes and simulations were repeated for this environment. The effects of a disruptive signal, such as hitting an obstacle in the environment, on the training processes and the behaviour of the trained agents were observed. The agents' preparation, training and testing processes with the algorithms were carried out using the Matlab program. The robot arm model was taken from the Matlab library, and the necessary modifications and environment modelling were made using Matlab's Simulink/Simscape interface. As a result, it has been seen that the algorithms can learn to perform the given tasks with a high success rate.
Robots have become widespread in recent years due to the development of technology, and studies on performing given tasks autonomously are increasing daily. Robot arms, which have the potential to achieve given tasks more powerfully and more precisely than humans, are used in a wide range of areas, such as operating rooms and space missions, as well as industrial facilities. Model-based approaches to the robot arm control problem in the literature require creating a mathematical model expressing the physical system and developing control algorithms. However, these mathematical models are not always easy to obtain, there may be situations such as being unable to solve them, and these methods do not show the desired performance. Artificial intelligence, which is one of the last topics of today's technology, has the potential to provide a solution and convenience for this situation. A practical solution in this area is Reinforcement Learning (RL), a sub-branch of machine learning. Reinforcement learning learns the optimum behaviours through the learning process by making discoveries and evaluating the behaviours revealed in this discovery according to specific criteria, without needing any processed data. Thus, it provides a control option without the need for mathematical models. In this respect, RL has been researched and developed in many areas, from industrial robots to unmanned aerial vehicles. This study used RL algorithms for the robot arm trajectory control problem, regardless of the mathematical model. Deep deterministic policy gradient (DDPG), twin-delayed policy gradient (TD3) and soft actor-critic (SAC) algorithms which are model-free, off-policy, actor-critic reinforcement learning algorithms that developed for continuous state and action spaces, were used. These algorithms were compared among themselves by various parameters such as correct positioning and fulfilment in the desired time in performing tasks through training processes and simulation of trained agents. A second environment with obstacles was created by placing fixed obstacles in the environment where the robot arm is located, and training processes and simulations were repeated for this environment. The effects of a disruptive signal, such as hitting an obstacle in the environment, on the training processes and the behaviour of the trained agents were observed. The agents' preparation, training and testing processes with the algorithms were carried out using the Matlab program. The robot arm model was taken from the Matlab library, and the necessary modifications and environment modelling were made using Matlab's Simulink/Simscape interface. As a result, it has been seen that the algorithms can learn to perform the given tasks with a high success rate.
Açıklama
Anahtar Kelimeler
Pekiştirmeli Öğrenme, Engelden Kaçınma, Robot Kol, DDPG, TD3, SAC, Reinforcement Learning, Obstacle Avoidance, Robotic Arm
