文章基本信息

标题：Comparative analysis of methods for prediction continuous numerical features on big datasets
本地全文：下载
作者：Eduard Kinshakov ; Yuliia Parfenenko ; Vira Shendryk 等
期刊名称：Technology Audit and Production Reserves
电子版ISSN：2706-5448
出版年度：2021
卷号：6
期号：2
页码：15-17
DOI：10.15587/2706-5448.2021.244003
语种：English
出版社：PC Technology Center
摘要：The object of research is the process of choosing a method for predicting continuous numerical features on big datasets. The importance of the study is due to the fact that today in various subject areas it is necessary to solve the problem of predicting performance indicators based on data collected from different sources and presented in different formats, which is the task of big data analysis. To solve the problem, the methods of statistical analysis were considered, namely multiple linear regression, decision trees and a random forest. An array of extensive data was built without specifying the subject area, its preliminary processing, analysis was carried out to establish the correlation between the features. The processing of the big data array was carried out using the technology of parallel computing by means of the Dask library of the Python language. Since working with big data requires significant computing resources, this approach does not require the use of powerful computer technology. Prediction models were built using multiple linear regression methods, decision trees and a random forest, visualization of the prediction results and analysis of the reliability of the constructed models. Based on the results of calculating the prediction error, it was found that the greatest prediction accuracy among the considered methods is the random forest method. When applying this method, the prediction accuracy for a dataset of numerical features was approximately 97%, which indicates a high reliability of the constructed model. Thus, it is possible to conclude that the random forest method is suitable for solving prediction problems using large data sets, it can be used for datasets with a large number of features and is not sensitive to data scaling. The developed software application in Python can be used to predict numerical features from different subject areas, the prediction results are imported into a text file.
关键词：machine learning;data analysis;big data;linear regression;decision tree;random forest