Machine learning for propensity score matching and weighting : comparing different estimation techniques and assessing ifferent balance diagnostics

Abstract

Using an extensive simulation exercise, we address two open issues in propensity score analyses: how to estimate propensity scores and how to assess covariates balance. We compare the performance of several machine learning algorithms and the standard logistic regression in terms of bias and mean squared errors of matching and weighing estimators based on the estimated propensity score. Additionally, we profit of the simulation framework to assess the ability of several measures of covariate balance in predicting the quality of the propensity score estimators in terms of bias reduction. Among the different techniques we considered, random forests performed the best when propensity scores were used for matching. In the case of weighting, both random forests and boosted tree outperformed other techniques. As for the performance of the several diagnostics of covariate balance we considered, we found that the simplest and most commonly used one, the Absolute Standardized Average Mean difference of covariates (ASAM), predicts well the bias of causal estimators. However, our findings suggest the use of a stringent rule: researchers should aim (at least) at obtaining an average ASAM lower than 10% and/or a low proportion of covariates with ASAM exceeding the 10% threshold. Balancing interactions among covariates is also desirable.