A Robust Distributed Hierarchical Online Learning Approach for Dynamic MEC Networks

We consider a resource allocation and offloading decision-making problem in a mobile edge computing (MEC) network. Since the locations of user equipments (UEs) vary over time in practice, we consider a dynamic network, where the UEs could leave or join the network coverage at any location. Since the joint offloading decision that minimizes the network cost also varies with the topology, the expected best offloading decision for the previous topology would not match the new topology. Consequently, the system suffers from recurring cost peaks due to the topology change. Thus, we propose a robust distributed hierarchical online learning approach to enhance the algorithm's robustness and reduce the cost peaks. Specifically, the UEs learn the utility of each offloading decision via deep Q networks (DQNs) from their interaction with the MEC network. Meanwhile, the computational access points (CAPs) train their deep neural networks (DNNs) online with the real-time data collected from the UEs to predict their corresponding Q-value vectors. Therefore, the UEs and CAPs form a hierarchical collaborative-learning structure. When the topology changes, each UE downloads its Q-value vector as the Q-bias vector and learns its difference from the actual Q-value vector via its DQN. With different agents learning distributedly, both the peak and sum costs are reduced as the joint offloading decision could start from a near-local-optimal point. In simulations, our robust approach successfully reduces the peak cost and sum cost by up to 50% and 30%, respectively. This demonstrates the need for a robust learning algorithm design in a practical dynamic MEC network.