METHODOLOGY OF DATA ANALYSIS AND PRE-PROCESSING FOR SOLVING MACHINE LEARNING PROBLEMS
DOI:
https://doi.org/10.34132/mspc2025.01.14.07Keywords:
Key words, data analysis and preprocessing methodology, feature generation methods, data gap processing methods, outlier processing methods, nonlinearity and non-stationarity identification methods, and normalization methods.Abstract
The paper describes and investigates the methodology of data analysis and pre-processing for solving machine learning problems. The methodology combines the following groups of methods based on a systematic approach: methods for processing data gaps, methods for processing abnormal values, methods for generating features, methods for identifying nonlinearities and non-stationarities, and methods for normalization. The method of generating features is investigated. Methods for selecting and generating features are divided into three main groups: filtering methods, wrapper methods, and embedded methods. The feature generation method consists of five steps for effectively selecting the most relevant features of the data set. It offers a better approach to feature selection. This leads to improved model performance with fewer features and reduced computational costs. To experimentally verify the methodology of data analysis and pre-processing, the creation of a red wine quality classification system was considered. To solve the problem of wine quality classification, modeling was carried out using various algorithms. The effectiveness of the method for solving data analysis and preprocessing problems for solving classification problems was proven.
References
Roonizi, K. (2023) Kalman filter/smoother-based design and implementation of digital IIR filters, Signal Processing, vol. 208, 108958. URL: https://doi.org/10.1016/j.sigpro.2023.108958
Bidyuk, P., Kalinina, I. & Gozhyj, A. (2022) An Approach to Identifying and Filling Data Gaps in Machine Learning Procedures. Lecture Notes on Data Engineering and Communications Technologies, vol. 77, (pp. 164–176). [in Switzeland]
Kiptoon, D. (2023) Understanding Feature Engineering in Machine Learning. [Online]. URL: https://medium.com/@jdkiptoon/understanding-feature-engineering-in-machine-learning-59fc343a29c9 (application date: 15.08.2024).
Kalinina, I. & Gozhyj, A. (2022) Modeling and forecasting of nonlinear nonstationary processes based on the Bayesian structural time series. Applied Aspects of Information Technology, vol. 5, no. 3, (pp. 240-255). URL: https://doi.org/10.15276/aait.05.2022.17.
