文章基本信息

标题：Scheduling Data Intensive Workloads through Virtualization on MapReduce based Clouds [
本地全文：下载
作者：B.Thirumala Rao ; L.S.S.Reddy
期刊名称：International Journal of Distributed and Parallel Systems
印刷版ISSN：2229-3957
电子版ISSN：0976-9757
出版年度：2012
卷号：3
期号：4
DOI：10.5121/ijdps.2012.3411
出版社：Academy & Industry Research Collaboration Center (AIRCC)
摘要：MapReduce has become a popular programming model for running data intensive applications on the cloud. Completion time goals or deadlines of MapReduce jobs set by users are becoming crucial in existing cloud-based data processing environments like Hadoop. There is a conflict between the scheduling MR jobs to meet deadlines and “data locality” (assigning tasks to nodes that contain their input data). To meet the deadline a task may be scheduled on a node without local input data for that task causing expensive data transfer from a remote node. In this paper, a novel scheduler is proposed to address the above problem which is primarily based on the dynamic resource reconfiguration approach. It has two components: 1) Resource Predictor: which dynamically determines the required number of Map/Reduce slots for every job to meet completion time guarantee; 2) Resource Reconfigurator: that adjusts the CPU resources while not violating completion time goals of the users by dynamically increasing or decreasing individual VMs to maximize data locality and also to maximize the use of resources within the system among the active jobs. The proposed scheduler has been evaluated against Fair Scheduler on virtual cluster built on a physical cluster of 20 machines. The results demonstrate a gain of about 12% increase in throughput of Jobs.
关键词：Cloud Computing; Data Locality; MapReduce; Virtualization; Hadoop