Aleksandr I. Panov

2023

1. TNNLS

   When to Switch: Planning and Learning For Partially Observable Multi-Agent Pathfinding

   Alexey Skrynnik, Anton Andreychuk, Konstantin Yakovlev, and 1 more author

   IEEE Transactions on Neural Networks and Learning Systems, 2023

   Bib PDF

   @article{Skrynnik2023,
     dimensions = {true},
     title = {When to Switch: Planning and Learning For Partially Observable Multi-Agent Pathfinding},
     doi = {10.1109/TNNLS.2023.3303502},
     pages = {(In Press)},
     journal = {{IEEE} Transactions on Neural Networks and Learning Systems},
     shortjournal = {{TNNLS}},
     author = {Skrynnik, Alexey and Andreychuk, Anton and Yakovlev, Konstantin and Panov, Aleksandr},
     year = {2023},
     langid = {english},
     keywords = {appl, group1}
   }

2. RAL

   Policy Optimization to Learn Adaptive Motion Primitives in Path Planning With Dynamic Obstacles

   Brian Angulo, Aleksandr Panov, and Konstantin Yakovlev

   IEEE Robotics and Automation Letters, 2023

   Abs Bib PDF

   This paper addresses the kinodynamic motion planning for non-holonomic robots in dynamic environments with both static and dynamic obstacles – a challenging problem that lacks a universal solution yet. One of the promising approaches to solve it is decomposing the problem into the smaller sub-problems and combining the local solutions into the global one. The crux of any planning method for non- holonomic robots is the generation of motion primitives that generates solutions to local planning sub-problems. In this work we introduce a novel learnable steering function (policy), which takes into account kinodynamic constraints of the robot and both static and dynamic obstacles. This policy is efficiently trained via the policy optimization. Empirically, we show that our steering function generalizes well to unseen problems. We then plug in the trained policy into the sampling-based and lattice-based planners, and evaluate the resultant POLAMP algorithm (Policy Optimization that Learns Adaptive Motion Primitives) in a range of challenging setups that involve a car-like robot operating in the obstacle-rich parking-lot en- vironments. We show that POLAMP is able to plan collision- free kinodynamic trajectories with success rates higher than 92%, when 50 simultaneously moving obstacles populate the environment showing better performance than the state-of-the- art competitors.

   @article{Angulo2023,
     dimensions = {true},
     title = {Policy Optimization to Learn Adaptive Motion Primitives in Path Planning With Dynamic Obstacles},
     volume = {8},
     issn = {2377-3766},
     url = {https://ieeexplore.ieee.org/document/10003648/},
     doi = {10.1109/LRA.2022.3233261},
     pages = {824--831},
     number = {2},
     journal = {{IEEE} Robotics and Automation Letters},
     author = {Angulo, Brian and Panov, Aleksandr and Yakovlev, Konstantin},
     year = {2023},
     eprint = {2212.14307},
     keywords = {robotics, group1}
   }

3. ICONIP

   HPointLoc: Point-based Indoor Place Recognition using Synthetic RGB-D Images

   Dmitry Yudin, Yaroslav Solomentsev, Ruslan Musaev, and 2 more authors

   In Neural Information Processing. Lecture Notes in Computer Science, 2023

   Abs Bib HTML PDF

   We present a novel dataset named as HPointLoc, specially designed for exploring capabilities of visual place recognition in indoor environment and loop detection in simultaneous localization and mapping. The loop detection sub-task is especially relevant when a robot with an on-board RGB-D camera can drive past the same place (“Point") at different angles. The dataset is based on the popular Habitat simulator, in which it is possible to generate photorealistic indoor scenes using both own sensor data and open datasets, such as Matterport3D. To study the main stages of solving the place recognition problem on the HPointLoc dataset, we proposed a new modular approach named as PNTR. It first performs an image retrieval with the Patch-NetVLAD method, then extracts keypoints and matches them using R2D2, LoFTR or SuperPoint with SuperGlue, and finally performs a camera pose optimization step with TEASER++. Such a solution to the place recognition problem has not been previously studied in existing publications. The PNTR approach has shown the best quality metrics on the HPointLoc dataset and has a high potential for real use in localization systems for unmanned vehicles. The proposed dataset and framework are publicly available: https://github.com/metra4ok/HPointLoc.

   @inproceedings{Yudin2023,
     dimensions = {true},
     title = {HPointLoc: Point-based Indoor Place Recognition using Synthetic RGB-D Images},
     volume = {13625},
     url = {https://link.springer.com/chapter/10.1007/978-3-031-30111-7_40},
     doi = {10.1007/978-3-031-30111-7_40},
     booktitle = {Neural Information Processing. Lecture Notes in Computer Science},
     author = {Yudin, Dmitry and Solomentsev, Yaroslav and Musaev, Ruslan and Staroverov, Aleksei and Panov, Aleksandr I},
     editor = {Tanveer, Mohammad and Agarwal, Sonali and Ozawa, Seiichi and Ekbal, Asif and Jatowt, Adam},
     year = {2023},
     keywords = {dataset, indoor localization, rgb-d image, synthetic, visual place recognition},
     pages = {471--484}
   }

4. Robotics

   Skill Fusion in Hybrid Robotic Framework for Visual Object Goal Navigation

   Aleksei Staroverov, Kirill Muravyev, Konstantin Yakovlev, and 1 more author

   Robotics, 2023

   Abs Bib HTML PDF

   In recent years, Embodied AI has become one of the main topics in robotics. For the agent to operate in human-centric environments, it needs the ability to explore previously unseen areas and to navigate to objects that humans want the agent to interact with. This task, which can be formulated as ObjectGoal Navigation (ObjectNav), is the main focus of this work. To solve this challenging problem, we suggest a hybrid framework consisting of both not-learnable and learnable modules and a switcher between them—SkillFusion. The former are more accurate, while the latter are more robust to sensors’ noise. To mitigate the sim-to-real gap, which often arises with learnable methods, we suggest training them in such a way that they are less environment-dependent. As a result, our method showed top results in both the Habitat simulator and during the evaluations on a real robot. Video and code for our approach can be found on our website: https://github.com/AIRI-Institute/skill-fusion (accessed on 13 July 2023).

   @article{Staroverov2023,
     dimensions = {true},
     title = {Skill {Fusion} in {Hybrid} {Robotic} {Framework} for {Visual} {Object} {Goal} {Navigation}},
     volume = {12},
     issn = {2218-6581},
     url = {https://www.mdpi.com/2218-6581/12/4/104},
     doi = {10.3390/robotics12040104},
     journal = {Robotics},
     author = {Staroverov, Aleksei and Muravyev, Kirill and Yakovlev, Konstantin and Panov, Aleksandr I},
     year = {2023}
   }

2022

1. BrainInf

   Hierarchical intrinsically motivated agent planning behavior with dreaming in grid environments

   Evgenii Dzhivelikian, Artem Latyshev, Petr Kuderov, and 1 more author

   Brain Informatics, 2022

   Abs Bib PDF

   Biologically plausible models of learning may provide a crucial insight for building autonomous intelligent agents capable of performing a wide range of tasks. In this work, we propose a hierarchical model of an agent operating in an unfamiliar environment driven by a reinforcement signal. We use temporal memory to learn sparse distributed representation of state–actions and the basal ganglia model to learn effective action policy on different levels of abstraction. The learned model of the environment is utilized to generate an intrinsic motivation signal, which drives the agent in the absence of the extrinsic signal, and through acting in imagination, which we call dreaming. We demonstrate that the proposed architecture enables an agent to effectively reach goals in grid environments.

   @article{Dzhivelikian2022a,
     dimensions = {true},
     title = {Hierarchical intrinsically motivated agent planning behavior with dreaming in grid environments},
     volume = {9},
     issn = {2198-4018},
     url = {https://braininformatics.springeropen.com/articles/10.1186/s40708-022-00156-6},
     doi = {10.1186/s40708-022-00156-6},
     pages = {8},
     number = {1},
     journal = {Brain Informatics},
     author = {Dzhivelikian, Evgenii and Latyshev, Artem and Kuderov, Petr and Panov, Aleksandr I},
     year = {2022},
     keywords = {slap, group1}
   }

2. Access

   Hierarchical Landmark Policy Optimization for Visual Indoor Navigation

   Aleksei Staroverov, and Aleksandr Panov

   IEEE Access, 2022

   Abs Bib PDF

   In this paper, we study the problem of visual indoor navigation to an object that is defined by its semantic category. Recent works have shown significant achievements in the end-to-end reinforcement learning approach and modular systems. However, both approaches need a big step forward to be robust and practically applicable. To solve the problem of insufficient exploration of the scenes and make exploration more semantically meaningful, we extend standard task formulation and give the agent easily accessible landmarks in the form of the room locations and those types. The availability of landmarks allows the agent to build a hierarchical policy structure and achieve a success rate of 63% on validation scenes in a photo- realistic Habitat simulator. In a hierarchy, a low level consists of separately trained RL skills and a high level deterministic policy, which decides which skill is needed at the moment. Also, in this paper, we show the possibility of transferring a trained policy to a real robot. After a bit of training on the reconstructed real scene, the robot shows up to 79% SPL when solving the task of navigating to an arbitrary object.

   @article{Staroverov2022,
     dimensions = {true},
     title = {Hierarchical Landmark Policy Optimization for Visual Indoor Navigation},
     volume = {10},
     issn = {2169-3536},
     url = {https://ieeexplore.ieee.org/document/9795006/},
     doi = {10.1109/ACCESS.2022.3182803},
     pages = {70447--70455},
     journal = {{IEEE} Access},
     author = {Staroverov, Aleksei and Panov, Aleksandr},
     year = {2022},
     keywords = {robotics, group1}
   }

2021

1. KBS

   Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations

   Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, and 3 more authors

   Knowledge-Based Systems, 2021

   Publisher: Elsevier B.V.

   Abs Bib PDF

   Deep reinforcement learning (RL) shows impressive results in complex gaming and robotic environments. These results are commonly achieved at the expense of huge computational costs and require an incredible number of episodes of interactions between the agent and the environment. Hierarchical methods and expert demonstrations are among the most promising approaches to improve the sample efficiency of reinforcement learning methods. In this paper, we propose a combination of methods that allow the agent to use low-quality demonstrations in complex vision-based environments with multiple related goals. Our Forgetful Experience Replay (ForgER) algorithm effectively handles expert data errors and reduces quality losses when adapting the action space and states representation to the agent’s capabilities. The proposed goal-oriented replay buffer structure allows the agent to automatically highlight sub-goals for solving complex hierarchical tasks in demonstrations. Our method has a high degree of versatility and can be integrated into various off-policy methods. The ForgER surpasses the existing state-of-the-art RL methods using expert demonstrations in complex environments. The solution based on our algorithm beats other solutions for the famous MineRL competition and allows the agent to demonstrate the behavior at the expert level.

   @article{Skrynnik2021,
     dimensions = {true},
     title = {Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations},
     volume = {218},
     issn = {09507051},
     url = {https://linkinghub.elsevier.com/retrieve/pii/S0950705121001076},
     doi = {10.1016/j.knosys.2021.106844},
     pages = {106844},
     journal = {Knowledge-Based Systems},
     author = {Skrynnik, Alexey and Staroverov, Aleksey and Aitygulov, Ermek and Aksenov, Kirill and Davydov, Vasilii and Panov, Aleksandr I.},
     year = {2021},
     note = {Publisher: Elsevier B.V.},
     keywords = {slap, group1}
   }

2. CSR

   Hierarchical Deep Q-Network from imperfect demonstrations in Minecraft

   Alexey Skrynnik, Aleksey Staroverov, Ermek Aitygulov, and 3 more authors

   Cognitive Systems Research, 2021

   Abs Bib PDF

   We present hierarchical Deep Q-Network (HDQfD) that took first place in MineRL competition. HDQfD works on imperfect demonstrations utilize hierarchical structure of expert trajectories extracting effective sequence of meta-actions and subgoals. We introduce structured task dependent replay buffer and adaptive prioritizing technique that allow the HDQfD agent to gradually erase poor-quality expert data from the buffer. In this paper we present the details of the HDQfD algorithm and give the experimental results in Minecraft domain.

   @article{Skrynnik2021a,
     dimensions = {true},
     title = {Hierarchical Deep Q-Network from imperfect demonstrations in Minecraft},
     volume = {65},
     issn = {13890417},
     url = {https://www.sciencedirect.com/science/article/pii/S1389041720300723?via%3Dihub},
     doi = {10.1016/j.cogsys.2020.08.012},
     pages = {74--78},
     journal = {Cognitive Systems Research},
     author = {Skrynnik, Alexey and Staroverov, Aleksey and Aitygulov, Ermek and Aksenov, Kirill and Davydov, Vasilii and Panov, Aleksandr I.},
     year = {2021},
     eprint = {1912.08664v2},
     keywords = {slap, group1}
   }

3. Access

   Hybrid Policy Learning for Multi-Agent Pathfinding

   Alexey Skrynnik, Alexandra Yakovleva, Vasilii Davydov, and 2 more authors

   IEEE Access, 2021

   Abs Bib PDF

   In this work we study the behavior of groups of autonomous vehicles, which are the part of the Internet ofVehicles systems. One of the challenging modes of operation of such systems is the case when the observability of each vehicle is limited and the global/local communication is unstable, e.g. in the crowded parking lots. In such scenarios the vehicles have to rely on the local observations and exhibit cooperative behavior to ensure safe and efficient trips. This type of problems can be abstracted to the so-called multi- agent pathfinding when a group of agents, confined to a graph, have to find collision-free paths to their goals (ideally, minimizing an objective function e.g. travel time). Widely used algorithms for solving this problem rely on the assumption that a central controller exists for which the full state of the environment (i.e. the agents current positions, their targets, configuration of the static obstacles etc.) is known and they cannot be straightforwardly be adapted to the partially-observable setups. To this end, we suggest a novel approach which is based on the decomposition of the problem into the two sub-tasks: reaching the goal and avoiding the collisions. To accomplish each of this task we utilize reinforcement learning methods such as Deep Monte Carlo Tree Search, Q-mixing networks, and policy gradients methods to design the policies that map the agents’ observations to actions. Next, we introduce the policy-mixing mechanism to end up with a single hybrid policy that allows each agent to exhibit both types of behavior – the individual one (reaching the goal) and the cooperative one (avoiding the collisions with other agents). We conduct an extensive empirical evaluation that shows that the suggested hybrid-policy outperforms standalone stat-of-the-art reinforcement learning methods for this kind of problems by a notable margin.

   @article{Skrynnik2021b,
     dimensions = {true},
     title = {Hybrid Policy Learning for Multi-Agent Pathfinding},
     volume = {9},
     issn = {2169-3536},
     url = {https://ieeexplore.ieee.org/document/9532001/},
     doi = {10.1109/ACCESS.2021.3111321},
     pages = {126034--126047},
     journal = {{IEEE} Access},
     author = {Skrynnik, Alexey and Yakovleva, Alexandra and Davydov, Vasilii and Yakovlev, Konstantin and Panov, Aleksandr I.},
     year = {2021},
     keywords = {appl, group1}
   }

2020

1. Access

   Real-Time Object Navigation with Deep Neural Networks and Hierarchical Reinforcement Learning

   Aleksey Staroverov, Dmitry A. Yudin, Ilya Belkin, and 3 more authors

   IEEE Access, 2020

   Abs Bib PDF

   In the last years, deep learning and reinforcement learning methods have significantly improved mobile robots in such fields as perception, navigation, and planning. But there are still gaps in applying these methods to real robots due to the low computational efficiency of recent neural network architectures and their poor adaptability to robotic experiments’ realities. In this paper, we consider an important task in mobile robotics - navigation to an object using an RGB-D camera.We develop a new neural network framework for robot control that is fast and resistant to possible noise in sensors and actuators. We propose an original integration of semantic segmentation, mapping, localization, and reinforcement learning methods to improve the effectiveness of exploring the environment, finding the desired object, and quickly navigating to it. We created a new HISNav dataset based on the Habitat virtual environment, which allowed us to use simulation experiments to pre-train the model and then upload it to a real robot. Our architecture is adapted to work in a real-time environment and fully implements modern trends in this area.

   @article{Staroverov2020b,
     dimensions = {true},
     title = {Real-Time Object Navigation with Deep Neural Networks and Hierarchical Reinforcement Learning},
     volume = {8},
     issn = {2169-3536},
     url = {https://ieeexplore.ieee.org/document/9241850/},
     doi = {10.1109/ACCESS.2020.3034524},
     pages = {195608--195621},
     journal = {{IEEE} Access},
     author = {Staroverov, Aleksey and Yudin, Dmitry A. and Belkin, Ilya and Adeshkin, Vasily and Solomentsev, Yaroslav K. and Panov, Aleksandr I.},
     year = {2020},
     keywords = {robotics, group1}
   }

2016

1. CSR

   Multilayer cognitive architecture for UAV control

   Stanislav Emel’yanov, Dmitry Makarov, Aleksandr I. Panov, and 1 more author

   Cognitive Systems Research, 2016

   Abs Bib PDF

   Extensive use of unmanned aerial vehicles (UAVs) in recent years has induced the rapid growth of research areas related to UAV production. Among these, the design of control systems capable of automating a wide range of UAV activities is one of the most actively explored and evolving. Currently, researchers and developers are interested in designing control systems that can be referred to as intelligent, e.g. the systems which are suited to solve such tasks as planning, goal prioritization, coalition formation etc. and thus guarantee high levels of UAV autonomy. One of the principal problems in intelligent control system design is tying together various methods and models traditionally used in robotics and aimed at solving such tasks as dynamics modelling, control signal genera- tion, location and mapping, path planning etc. with the methods of behaviour modelling and planning which are thoroughly studied in cognitive science. Our work is aimed at solving this problem. We propose layered architecture — STRL (strategic, tactical, reactive, layered) — of the control system that au- tomates the behaviour generation using a cognitive approach while taking into account complex dynamics and kinematics of the control object (UAV).We use a special type of knowledge representation — sign world model — that is based on the psychological activity theory to describe individual behaviour planning and coalition formation processes. We also propose path planning methodology which serves as the mediator between the high-level cognitive activities and the reactive control signals generation. To generate these signals we use a state-dependent Riccati equation and specific method for solving it. We believe that utilization of the proposed architecture will broaden the spectrum of tasks which can be solved by the UAV’s coalition automatically, as well as raise the autonomy level of each individual member of that coalition.

   @article{Emelyanov2016,
     dimensions = {true},
     title = {Multilayer cognitive architecture for {UAV} control},
     volume = {39},
     issn = {1389-0417},
     url = {http://linkinghub.elsevier.com/retrieve/pii/S1389041716000048},
     doi = {10.1016/j.cogsys.2015.12.008},
     pages = {58--72},
     journal = {Cognitive Systems Research},
     author = {Emel’yanov, Stanislav and Makarov, Dmitry and Panov, Aleksandr I. and Yakovlev, Konstantin},
     year = {2016},
     keywords = {strl, group1}
   }