文章基本信息

标题：An Optimized Algorithm for Reduce Task Scheduling
本地全文：下载
作者：Zhang, Xiaotong ; Hu, Bin ; Jiang, Jiafu 等
期刊名称：Journal of Computers
印刷版ISSN：1796-203X
出版年度：2014
卷号：9
期号：4
页码：794-801
DOI：10.4304/jcp.9.4.794-801
语种：English
出版社：Academy Publisher
摘要：In this paper, we propose a novel algorithm to solve the starving problem of the small jobs and reduce the process time of the small jobs on Hadoop platform. Current schedulers of MapReduce/Hadoop are quite successful in achieving data locality and scheduling the reduce tasks with a greedy algorithm. Some jobs may have hundreds of map tasks and just several reduce tasks, in which case, the reduce tasks of the large jobs require more time for waiting, which will result in the starving problem of the small jobs. Since the map tasks and the reduce tasks are scheduled separately, we can change the way the scheduler launches the reduce tasks without affecting the map phase. Therefore we develop an optimized algorithm to schedule the reduce tasks with the shortest remaining time (SRT) of the map tasks. We apply our algorithm to the fair scheduler and the capacity scheduler, which are both widely used in real production environment. The evaluation results show that the SRT algorithm can decrease the process time of the small jobs effectively.
关键词：mapreduce;hadoop;schedule;SRT