Hybrid machine learning: lasso and random forests

Abstract

This research investigates a hybrid machine learning approach that combines Least Absolute Shrinkage and Selection Operator (LASSO) with Random Forests to solve regression problems. LASSO is employed to capture the primary relationship between the response variable and independent variables, while Random Forests are harnessed to capture both nonlinear relationships and noise components within the data. Additionally, we investigate the impact of incorporating second-order LASSO with Random Forests into the regression framework. To evaluate the efficacy of our hybrid methods, we conduct a comprehensive comparative analysis against traditional LASSO, Random Forests, and Local Linear Random Forests (LLF) using five benchmark datasets from the existing literature. Remarkably, our proposed methods exhibit substantial enhancements in predictive accuracy, except under high-noise conditions. Furthermore, this study explores the implications of varying noise levels and redundant predictor variables for prediction errors, underscoring the robustness of our proposed methods in effectively managing redundancy in the presence of noise. The proposed methods are applied to address research questions using the Research and Development Survey, conducted by the National Center for Health Statistics, Centers for Disease Control and Prevention.

The authors' abstract, as published at the source. Journal of Applied Statistics, 2026 · DOI ↗