Deep Model Intervention for Representation Learning of Tabular Data
Abstract
Weight pruning methods compress neural network models to comply with high memory re-quirements after accepting marginal loss in supervised image classification performance. However,similar concepts have not been explored in unsupervised learning or the classification of heteroge-neous tabular data. Furthermore, the classification of tabular data with DNNs is often challengedby the superior performance of traditional machine learning methods. This thesis provides oneof the first investigations into weight pruning methods for an unsupervised autoencoder modelon tabular data sets. A novel weight perturbation method is proposed that periodically perturbsDNN weights during its unsupervised pre-training stage. The proposed weight perturbation rou-tine sets some target weights to zero (resets) at a constant interval and then allows the perturbedweights to regrow during pretraining. The pretrained model is evaluated in a downstream classi-fication task using six tabular data sets. The proposed weight perturbation algorithm outperformsdropout learning or weight regularization (L1 or L2) for four out of six tabular data sets. Un-like dropout learning, the proposed weight perturbation routine additionally achieves 15% to 40%sparsity across six tabular data sets, resulting in compressed pretrained models. While traditionalweight pruning methods trade off classification performance for model compression, the proposedperturbation method achieves performance gain instead. The proposed routine also helps DNNmethods outperform traditional machine learning methods on tabular data. The findings suggestthat weight compression methods should focus on targeting weights that may contribute to overfit-ting to achieve model compression and performance gain simultaneously.
Subject Area
Computer science|Artificial intelligence
Recommended Citation
Sakib Abrar,
"Deep Model Intervention for Representation Learning of Tabular Data"
(2023).
ETD Collection for Tennessee State University.
Paper AAI30312402.
https://digitalscholarship.tnstate.edu/dissertations/AAI30312402