Gradient Boosting Machine Learning Accurately Predicts Digital Health

Researchers studying how to keep users on telehealth and mobile health platforms have found that advanced machine learning methods can predict when customers are likely to stop using these services. These methods can identify complex patterns in user behavior and activity. In a study published in the journal Discover Health Systems, researchers Safar Fazli and Mohammad Javad Mohammadi tested three leading machine learning methods: XGBoost, LightGBM, and CatBoost. They used real records from an active digital healthcare company. Losing customers is a serious problem for digital healthcare organizations. It reduces regular business income and can also interrupt the ongoing medical care that patients need for better long-term health. The researchers found that after carefully adjusting the models using a tool called Optuna, all three methods performed well on anonymous platform data. XGBoost performed best overall, with an accuracy of 84.1 percent. It was also better at identifying users who were at risk of leaving the platform.

Digital healthcare platforms work differently from ordinary online shops and subscription services. This creates special challenges when trying to predict which users will leave. These platforms offer many healthcare services, including online doctor appointments, virtual consultations, and medical tests that people can take at home. In ordinary software businesses, losing a customer mainly means losing income. In digital healthcare, it can also break the connection between patients and their ongoing care, making it harder for them to follow treatment plans. People's use of healthcare apps is influenced by urgent medical needs, personal worries, and privacy concerns, rather than simply by convenience. Because of this, ordinary financial records and personal details do not provide enough information to predict when someone might leave. More detailed measures of user activity are more useful. These include average session length, how often people log in, and whether they attend appointments. Such measures can help show whether patients find the service useful, reliable, and trustworthy over time.

To test prediction methods in this special healthcare setting, the researchers collected an existing dataset directly from the internal computer systems of an established digital health company. The company asked the researchers not to reveal its name or identifying details, to protect its business information. The dataset covered twelve consecutive months of patient activity, from April 2024 to April 2025. It included activity through mobile apps and desktop websites. The first data collection contained 3,500 customer records. Each record represented one person who used the platform's medical or administrative services. After removing profiles that did not have the information needed to identify whether a customer had left, the researchers kept 3,343 records for the study. They defined customer churn as completely stopping use of the service, with no activity for three consecutive months after the person's last recorded activity. In the final group, 46 percent of customers had left and 54 percent were still active. This created a moderately uneven split between the two groups, similar to patterns seen in real businesses.

The research team created a set of features combining user behavior, personal details, and payment and treatment information. Behavioral features included average session length, logins per month, active days per month, how often the app was updated, and how long users stayed with the platform. Personal details included customer age, the Iranian province where each customer lived, and whether they used a personal computer or mobile phone. Payment and healthcare information included total money paid, type of treatment, whether treatment was ongoing, prescription status, follow-up appointments, and number of diagnoses. The researchers prepared the data by filling in missing values. They used the median for missing numerical values to reduce the effect of unusually high or low numbers. They used the most common value for missing category information to keep the original distribution of categories as similar as possible. They also changed category information into numerical form using one-hot encoding for XGBoost and LightGBM. CatBoost used its own built-in method for handling categories. To address the uneven number of customers in each group, the researchers used stratified cross-validation and different weighting settings for each model. These included scale positive weight in XGBoost and LightGBM, LightGBM's unbalance setting, and CatBoost's balanced class weight setting.

The researchers tested three different gradient boosting methods using Stratified 5-Fold Cross-Validation with a fixed random seed. This helped make sure the data was divided into groups in a repeatable way. In each round, 80 percent of the records were used to train the model, while the remaining 20 percent were used to test it. XGBoost builds decision trees by expanding them level by level. It also uses L1 and L2 penalties to reduce overfitting, which happens when a model learns the training data too closely and performs less well on new data. LightGBM grows trees leaf by leaf and uses two techniques called Gradient-Based One-Side Sampling and Exclusive Feature Bundling to make calculations faster when there are many features. The researchers used Optuna to find better settings for each model. Optuna uses a method called Bayesian Tree-structured Parzen Estimators to guide the search for useful settings. The researchers adjusted factors such as tree depth, the number of estimators from 100 to 2000, learning rates from 0.01 to 0.3, data sampling rates, column sampling, and regularization values. Their main goal was to improve Recall and F1-Score for the smaller group of customers who left, so the models would be better at finding users at risk of leaving.

Tests using the models' original settings showed that all three methods could classify customers reasonably well, but they differed in how well they detected people who left. XGBoost had the highest accuracy at 82.5 percent. Its precision was 83 percent for customers who stayed and 82 percent for customers who left. Its recall showed that it correctly identified 80 percent of customers who actually left and 84 percent of those who stayed. Its average precision across the two groups was 82 percent. CatBoost performed almost as well, with 82.3 percent accuracy and precision and recall scores of 82 percent for both groups. LightGBM had an accuracy of 81.7 percent. However, its recall was uneven: it correctly identified 85 percent of customers who stayed but only 78 percent of those who left. The researchers also examined confusion matrices, which show correct and incorrect predictions. They found that all three models made more false negative predictions than false positive predictions. This means that, with their original settings, the models were more likely to miss customers who had actually left than to wrongly label customers as having left.

Adjusting the models with Optuna improved their performance and made their results more stable. After this process, the three methods produced similar overall results. XGBoost performed best overall, reaching 84.1 percent accuracy and a low cross-validation standard deviation of 1.2 percent. Its precision was 86 percent for customers who stayed and 82 percent for customers who left. It correctly identified 85 percent of customers who left, achieved an F1-Score of 83 percent, and had an Area Under the ROC Curve (AUC) of 0.91. LightGBM reached 83.8 percent accuracy, with a standard deviation of 1.3 percent. Its recall was 84 percent for both groups, and it had the highest ability to distinguish between the groups, with an AUC of 0.92. CatBoost also reached 83.8 percent accuracy and 84 percent recall for both groups. However, its cross-validation standard deviation was higher, at 2.0 percent, and its AUC was 0.91. The researchers found that CatBoost improved less after tuning than XGBoost and LightGBM. They suggested that this was because CatBoost already had strong default settings, built-in methods to prevent overfitting, and its own way of handling category information.

The researchers examined which features had the greatest effect on the models' predictions. They found that the results differed depending on how each algorithm measured feature importance. XGBoost measures importance by counting how often a feature is used to split a decision tree. In this model, Device Type was the most important feature. This suggests that using a personal computer or a mobile app may create different experiences that affect whether users stay. Prescription status was the next important feature, followed by average session length, update frequency, and logins per month. LightGBM measures importance by adding up how much each feature reduces prediction errors. It ranked average session length, active days per month, and logins per month as its most important features, followed by total amount paid. CatBoost uses permutation importance, which measures how much predictions change when a feature's values are mixed up. It also ranked average session length as its most important feature. Across all three models, measures of user activity were more important than personal details such as age or location. This shows that how deeply people use a platform is a major factor in predicting whether they will leave.

To understand how individual features affected the predicted risk of leaving, the researchers also used a method called SHAP analysis, which is based on game theory. In the XGBoost model, update frequency had the widest range of SHAP values and the strongest individual effect on the model's predictions. Across all three models, higher values for average session length, logins per month, active days per month, time spent using the app over the user's relationship with the platform, follow-up appointments, and number of diagnoses were linked to a lower predicted chance of leaving. This suggests that regular and longer use of the platform is strongly linked to customer retention. Using a mobile platform and being older were also linked to a lower chance of leaving. However, the researchers found an unexpected pattern for update frequency. In all three models, more frequent updates were linked to a higher chance of leaving, while fewer updates were linked to better retention. The authors stressed that SHAP values show relationships used by a prediction model, not proven cause-and-effect relationships. They suggested that frequent software updates might introduce technical problems or sudden changes to the interface. These changes could reduce users' trust and lead them to stop using the platform.

The researchers connected their findings to the Customer Lifecycle framework to show how healthcare managers could use them to improve customer retention at different stages. In this framework, customer churn is seen as a gradual weakening of customer retention and loyalty, rather than something that happens suddenly. A drop in session length, monthly logins, and active days can act as an early warning that users are becoming less satisfied and less involved in their healthcare. Other factors, such as ongoing prescriptions and the total amount a patient has paid, may make it harder for patients to switch to another service and help keep them in the retention stage. The authors also described several limits of their study. The data came from one private digital health provider and represented a single cross-sectional dataset. The researchers also did not carry out formal statistical tests to determine whether the differences in model accuracy were statistically significant. They recommended that future studies test deep learning methods that process sequences of information, such as recurrent neural networks and transformers. They also suggested using data from several healthcare organizations to see whether the results apply more widely. Finally, they recommended customer grouping methods such as K-means clustering to better understand the different reasons why people stop using digital health services.

Subject of Research: Machine learning prediction of customer churn in digital health platforms

Article Title: Using gradient boosting machine learning models to predict customer churn in digital health platforms

Article References: Fazli, S., & Mohammadi, M. J. (2026). Using gradient boosting machine learning models to predict customer churn in digital health platforms. Discover Health Systems, 5(1), Article 51. https://doi.org/10.1007/s44250-026-00387-y

Image Credits: AI Generated

DOI: 10.1007/s44250-026-00387-y

Keywords: Machine Learning, Digital Health, XGBoost, LightGBM, CatBoost, Customer Churn, Explainable AI, SHAP, Healthcare Analytics, Using, gradient, boosting