Journal's Facebook pageDelay-Aware Resource-Efficient Interleaved Task Scheduling Strategy in Spark
- School of Software, Henan University
Kaifeng, Henan, China, 475000
zhangyanhao@henu.edu.cn, wangcongyang@henu.edu.cn, hexin@henu.edu.cn, jyyu@henu.edu.cn, zr@henu.edu.cn, 122648935@qq.com
Abstract
For solving the low CPU and network resource utilization in the task scheduler process of the Spark and Flink computing frameworks, this paper proposes a Delay-Aware Resource-Efficient Interleaved Task Scheduling Strategy(RPTS). This algorithm can schedule parallel tasks in a pipelined fashion, effectively improving the system resource utilization and shortening the job completion times. Firstly, based on historical data of task completion times, we stagger the execution of tasks within the stage with the longest completion time. This helps optimize the utilization of system resources and ensures the smooth completion of the entire pipeline job. Secondly, the execution tasks are categorized into CPU-intensive and non-CPU-intensive phases, which include network I/O and disk I/O operations. During the non-CPU-intensive phase where tasks involve data fetch, parallel tasks are scheduled at suitable intervals to mitigate resource contention and minimize job completion time. Finally, we implemented RPTS on Spark 2.4.0 and conducted experiments to evaluate its performance. The results show that compared to DelayStage, RPTS reduces job execution time by 3.18% to 6.48% and improves CPU and network utilization of the cluster by 6.33% and 7.02%, respectively.
Key words
Job execution time, delay-aware, Spark, task scheduler
How to cite
Zhang, Y., Wang, C., He, X., Yu, J., Zhai, R., Song, Y.: Delay-Aware Resource-Efficient Interleaved Task Scheduling Strategy in Spark. Computer Science and Information Systems
