Open Data Machine Learning Framework Predicts Which Media Platform Users

Creating useful features was a key part of the study. The researchers turned the raw records of user activity into four groups of information: behavior, timing, content preferences, and demographics. Behavioral features included each user's average rating and how much their ratings varied. These measures showed how consistent a user's ratings were. Timing features were especially important. They included rating frequency, calculated by dividing the number of ratings by the length of time a user was active, and the average time between ratings. Longer gaps between interactions may mean that a user is losing interest. The researchers also measured how many days had passed since a user's last interaction and created a binge index, which recorded the highest number of ratings a user submitted in one day. Content-related features measured users' genre preferences. A genre affinity vector showed their interest in different movie genres, while entropy measured how varied those interests were. The researchers also included demographic information such as age group, gender, and occupation, converting these categories into a format the models could use. They scaled all numerical features using min-max normalization and used Pearson correlation analysis to check whether features such as average rating and rating variation contained overlapping information.

After preparing the features, the researchers trained and compared ten different algorithms. Logistic Regression provided a simple starting point. It used a mathematical function called a sigmoid to estimate the chance that a user would stay. A Decision Tree provided a simple model that could handle more complex patterns by dividing the data into groups using rules that reduce differences within each group. A Support Vector Machine with a radial basis function kernel tested whether learning clear boundaries between groups could improve predictions. A Random Forest built many decision trees using randomly selected samples and combined their predictions through majority voting. This helped reduce the risk of a single tree learning the training data too closely. A Multilayer Perceptron served as the basic neural network model. It learned complex relationships through hidden layers and activation functions. The researchers also tested four gradient boosting methods. Classical Gradient Boosting builds trees one after another, with each new tree trying to correct earlier mistakes. XGBoost adds controls to reduce overfitting and uses efficient tree pruning. LightGBM grows trees by expanding the leaf that offers the greatest improvement, helping it work efficiently. CatBoost is designed to handle category-based information, such as gender, occupation, and genre features.

The tenth model, TabPFN 2.5, works differently from the tree-based boosting methods. It is a foundation model for table-based data, rather than a group of decision trees. It uses knowledge learned from a large collection of artificially created table-based tasks to make predictions on a new dataset. The researchers included it to test whether this newer type of model could improve predictions using the features they had created. They followed a careful testing process. They split the dataset into 80 percent for training and 20 percent for testing. They also used stratified five-fold cross-validation on the training data, meaning they divided it into five parts while keeping the proportions of retained and churned users similar in each part. Importantly, they respected the order of events in time rather than randomly mixing earlier and later interactions. They built features using only interactions from a defined observation period, then used behavior during the following 30 days to determine whether users stayed. This was intended to reduce information leakage, which happens when information that would not have been available at prediction time accidentally helps a model. The researchers used grid search with cross-validation to choose the best settings for each model and repeated the experiments with the same random seed so others could reproduce the results.

The results showed that the newest model performed best. On the separate test dataset, TabPFN 2.5 achieved the highest score on every reported measure: 0.814 accuracy, 0.789 precision, 0.741 recall, an F1-score of 0.764, an AUC of 0.866, and a specificity of 0.862. Among the traditional models that combine decision trees, classical Gradient Boosting performed best. It reached 80.2 percent accuracy, an F1-score of 0.752, and an AUC of 0.852. XGBoost, LightGBM, and CatBoost came close behind. The researchers also used cross-validation results to test whether differences between models were statistically meaningful, mainly by comparing AUC scores. Gradient Boosting performed significantly better than the simpler methods, including Logistic Regression, Decision Tree, Support Vector Machine, and Random Forest. However, its results were not significantly different from those of the Multilayer Perceptron or the improved boosting models. TabPFN 2.5 performed significantly better than Gradient Boosting, matching the pattern seen in the test results. ROC curves and precision-recall curves showed a similar ranking. TabPFN 2.5 and XGBoost offered the best overall balance between precision and recall, while normalized confusion matrices showed that TabPFN 2.5 performed best across the two user groups.

The researchers mainly used the easier-to-interpret Gradient Boosting model to find which features had the greatest influence on predictions. Measures of behavior and timing were the most important. These included how often users rated movies, how much time passed between ratings, and how varied their genre choices were. How recently users had been active was also an important factor. Users who stayed tended to rate movies more often, leave shorter gaps between ratings, and explore a wider range of genres, especially drama and action. The basic statistics also showed that retained users had more consistent and varied rating habits and shorter gaps between interactions than users who left. Demographic details helped somewhat too. Younger users and people working or studying in student or technical fields were more likely to stay. Female users had a slightly higher rate of leaving than male users, but gender and location were much weaker predictors than behavior. The researchers explained that different types of models capture these patterns in different ways. Linear models can measure the separate effects of rating frequency and recent activity, while tree-based and boosting models can detect thresholds and more complex connections between features. The team also tested inactivity limits of 7, 14, and 60 days. The models' absolute performance changed depending on the chosen limit, but their relative rankings remained similar, with Gradient Boosting consistently among the strongest models.

The authors caution against treating these results as a direct picture of how people use media today. MovieLens 1M records user activity from about two decades ago, so the study is best viewed as a repeatable starting point for developing and testing retention models, rather than a direct description of current media platforms or modern viewing habits. The researchers stress that the main contribution is not a new algorithm, but a clear and open framework that allows researchers to test predictions using public data, understandable features, and consistent experimental methods. Future studies could add streaming data such as viewing time and clicks, use sequence models such as LSTMs or Transformers to study how user behavior changes over time, and test whether the methods work on data from today's services. Researchers could also run ablation studies, which remove groups of features to see how much each group contributes to prediction quality. For media companies, the main lesson is that how people use a platform—including how often they interact, how consistent they are, and how varied their interests are—reveals much more about whether they will stay than their personal background alone. Watching for changes in behavior and reaching out to users at the right time may therefore reduce losses from cancellations more effectively than targeting people based mainly on demographic details.

Subject of Research: Predicting whether users will stay by studying their online behavior, using the MovieLens 1M dataset and ten machine learning models

Article Title: Predictive modeling of user retention based on online behavior in media service platforms

Article References: Chen, J., & Chen, D. (2026). Predictive modeling of user retention based on online behavior in media service platforms. Discover Artificial Intelligence, 6(1), Article 1008. https://doi.org/10.1007/s44163-026-02043-z

Image Credits: AI generated

DOI: 10.1007/s44163-026-02043-z

Keywords: user retention, predicting user departures, machine learning, MovieLens, TabPFN, gradient boosting, media platforms, behavior data, feature creation, reproducibility, foundation models for table-based data, Discover Artificial Intelligence