Research report: Artificial intelligence-based analysis of digital media

The study introduces a system called the Hybrid Textual Sentiment Engagement Learning Model, or HTSELM. It combines two different types of information to make predictions. The first part looks at text. It turns words into numerical values using a method called TF-IDF, then uses a linear support vector machine to classify documents. This combination is a well-established method that still works very well on datasets of moderate size. The second part looks at structured information, including the platform, location, language, campaign identifiers, time-related information and engagement counts. It uses a method called HistGradientBoosting, which builds a series of decision-making models using groups of numerical values. This method can find complex patterns and relationships between categories and numbers.

Instead of combining all the information into one large set of features, the model trains each part separately. Each part produces probabilities for the possible classes, and the model combines these probabilities using a weighted average. The researchers choose the weights using a separate validation dataset. Giving both parts equal weight actually reduced performance. Combining the two parts helped only when the text-based part, which was much stronger, received most of the weight. On a separate test dataset, HTSELM achieved an accuracy of 0.9392, a macro-F1 score of 0.9398, a Matthews correlation coefficient of 0.9043 and a macro ROC-AUC of 0.9923.

In an exact five-fold cross-validation test, the average macro-F1 score was 0.9398, with a standard deviation of only 0.0069. This suggests that the results were stable and did not depend on one lucky split of the data. Compared with a support vector machine that used text alone, the improvement was small: the macro-F1 score increased by 0.0048. However, the improvement appeared in all five cross-validation folds. The most important part of the study may not be these performance numbers, but the careful checks the researchers carried out before reporting them.

Wang studied the Social Media Engagement Dataset, a computer-generated collection of 12,000 records with 28 original columns. The study found that the released variable called sentiment_score directly determines the target variable, sentiment_label. Negative labels match scores below about −0.20, neutral labels match a narrow range around zero, and positive labels begin above about +0.20. The engagement_rate column also matched a formula based on likes, shares, comments and impressions perfectly, apart from small differences caused by rounding. If a model used these variables during training, it would not really be making predictions. It would simply be recovering answers that were already built into the data.

To avoid this problem, the study removed sentiment_score, emotion_type, user_past_sentiment_avg and toxicity_score before training the sentiment classifier. In the engagement experiment, it also removed engagement_rate and all variables measured after people had responded. This created what the author calls a leakage-controlled baseline: an honest starting point rather than an artificially high score. Tests that removed different features also showed how much the model depended on the words in the posts. The explanation analysis supported the same conclusion. Important TF-IDF terms included phrases such as "okay," "wouldn't recommend" and "highly recommend." These phrases clearly express positive or negative feelings and strongly influenced the model's decisions. By comparison, permutation tests showed that structured features had small effects spread across information about brands, products, languages, campaigns and locations.

In simple terms, once information that indirectly reveals people's feelings was removed, the extra metadata provided surprisingly little help in identifying sentiment in this dataset. Tests of how well the model handled changes to the data added more detail. Adding random noise to numerical features had almost no effect on performance. Macro-F1 scores ranged from 0.9390 to 0.9423 at different noise levels.

However, removing features or damaging the text caused a gradual but meaningful drop in performance. When 20 percent of the features were affected in missing-feature tests, the macro-F1 score fell to 0.8441. Adding noise at the word-token level reduced it to 0.8599. This suggests that the practical reliability of these systems depends much more on having complete, accurate text than on having highly precise numerical information alongside it. The confusion matrix showed that the model handled neutral posts reasonably well, correctly identifying 0.9069 of the neutral cases. Most remaining mistakes involved confusing negative posts with positive ones, or positive posts with negative ones.

The study's second experiment produced perhaps its most concerning result. The researchers tried to predict engagement categories using only information available before people responded. This included text, platform, timing and campaign information, but excluded counts and other information based on responses. Every model performed close to the one-third success rate expected from random guessing among three categories. The best model, a logistic regression using structured features, achieved an accuracy of just 0.3479. HTSELM itself reached only 0.3413 accuracy, with a Matthews correlation coefficient of 0.0118, which is effectively zero.

Wang argues that this poor result is scientifically useful. It shows that the information available before a response in this dataset does not provide a reliable basis for predicting engagement. It also helps prevent future researchers from confusing the recovery of a formula already present in the data with genuine forecasting. The author clearly states that the findings have limits. The dataset was created by a computer rather than collected from real-world activity, so its writing patterns, platform distributions and user behavior may not reflect what happens on actual social media.

The study's baseline models deliberately leave out transformer models such as BERT and RoBERTa. These models might better understand context, including sarcasm, negation and the mixing of different languages in the same text. However, the study says that any comparison with these models should first apply the same checks to prevent data leakage. Statistical tests also showed that the evaluation had limits because it used only a small number of results. Paired t-tests across five folds favored HTSELM over the text-only models, but Wilcoxon and McNemar tests did not find statistically significant differences. This reflects the limited ability of the tests to detect small effects with so little data, rather than evidence that the results directly contradict each other. The study treats explainability as a key part of the research process, not something added at the end. Along with performance scores, it reports model coefficients, the importance of individual features measured by permutation tests, and the proportions of different types of errors. This approach follows human-centered AI principles, which call for openness about where data comes from, what models cannot do and where they should or should not be used.

Subject of Research: Using machine learning to analyze social media sentiment and engagement while preventing data leakage

Article Title: Artificial intelligence-based analysis of digital media engagement and user sentiment using machine learning techniques

Article References: Wang, X. (2026). Artificial intelligence-based analysis of digital media engagement and user sentiment using machine learning techniques. Discover Artificial Intelligence, 6(1), Article 1448. https://doi.org/10.1007/s44163-026-02430-6

Image Credits: Created using AI

DOI: 10.1007/s44163-026-02430-6

Keywords: artificial intelligence, machine learning, sentiment analysis, social media, target leakage, engagement prediction, late fusion, explainable AI, TF-IDF, gradient boosting, digital media analytics, data leakage