文章基本信息

标题：Spark for Social Science
本地全文：下载
作者：Graham MacDonald ; Alex Engler ; Jeffrey Levy 等
期刊名称：International Journal of Population Data Science
电子版ISSN：2399-4908
出版年度：2018
卷号：3
期号：5
页码：1-1
DOI：10.23889/ijpds.v3i5.1044
出版社：Swansea University
摘要：Urban has developed an elastic and powerful approach to the analysis of massive datasets using Amazon Web Services’ Elastic MapReduce (EMR) and the Spark framework for distributed memory and processing. The goal of the project is to deliver powerful and elastic Spark clusters to researchers and data analysts with as little setup time and effort possible, and at low cost. To do that, at the Urban Institute, we use two critical components: (1) an Amazon Web Services (AWS) CloudFormation script to launch AWS Elastic MapReduce (EMR) clusters (2) a bootstrap script that runs on the Master node of the new cluster to install statistical programs and development environments (RStudio and Jupyter Notebooks). The Urban Institute’s Spark for Social Science Github page holds code used to setup the cluster and tutorials for learning how to program in R and Python.