Optimal Scheduling over Time-Varying Channels with Traffic Admission Control: Structural Results and Online Learning Algorithms

• Published in: IEEE transactions on wireless communications Vol. 12; no. 9; pp. 4434 - 4444
• Main Authors: Phan, Khoa T., Tho Le-Ngoc, van der Schaar, Mihaela, Fangwen Fu
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.09.2013; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Access methods and protocols, osi model; Admission control; Algorithms; Applied sciences; Approximation methods; Channels; Delays; Distance learning; Equations; Exact sciences and technology; Heuristic algorithms; Learning; Markov decision process (MDP); Mathematical model; Mathematical models; On-line systems; Online; Operations research; Optimization; Scheduling; structural results; Studies; Systems, networks and services of telecommunications; Telecommunications; Telecommunications and information theory; Teleprocessing networks. Isdn; Teletraffic; Throughput; traffic admission control; Transmission and modulation (techniques and equipments); Access control; Markov process; E-learning; Averaging method; Markov decision; Information rate; learning; traffic admission control; Time variation; Arrival process; Traffic control; Convexity; Optimal planning; Learning algorithm; Queue; structural results; Fading channels; Markov decision process (MDP); Time variable channel; Teletraffic; Scheduling; Information transmission; Transmission time; Queueing system; On line processing; Algorithm analysis; Monotonicity
• ISSN: 1536-1276; 1558-2248
• DOI: 10.1109/TW.2013.081913.121525

This work studies the joint scheduling- admission control (SAC) problem for a single user over a fading channel. Specifically, the SAC problem is formulated as a constrained Markov decision process (MDP) to maximize a utility defined as a function of the throughput and queue size. The optimal throughput- queue size trade-off is investigated. Optimal policies and their structural properties (i.e., monotonicity and convexity) are derived for two models: simultaneous and sequential scheduling and admission control actions. Furthermore, we propose online learning algorithms for the optimal policies for the two models when the statistical knowledge of the time-varying traffic arrival and channel processes is unknown. The analysis and algorithm development are relied on the reformulation of the Bellman's optimality equations using suitably defined state-value functions which can be learned online, at transmission time, using time-averaging. The learning algorithms require less complexity and converge faster than the conventional Q-learning algorithms. This work also builds a connection between the MDP based formulation and the Lyapunov optimization based formulation for the SAC problem. Illustrative results demonstrate the performance of the proposed algorithms in various settings.