Hierarchical Taxonomy Framework Tracks Shifting Artificial Intelligence

Researchers have created a framework that measures and compares how topics change over time across different types of artificial intelligence (AI) documents. AI is growing quickly and involves many different groups. Academic researchers work on new theories, engineers build software models, and industry professionals discuss how AI is used in business. These groups share their work in different formats, such as scientific papers, open-source software repositories, and industry newsletters. Each source uses different words, levels of technical detail, and publishing schedules. Traditional tools for measuring scientific research have trouble bringing these sources together. They often examine only one source at a time or use simple topic lists that cannot show how topics change across different levels or communities. Published in EPJ Data Science, the new method solves these problems by placing documents from different sources into shared, multi-level topic systems. This makes it possible to compare them at different levels of detail without limiting the analysis to one source or creating new topic labels without a clear basis.

The framework has three main parts: predefined topic systems arranged as directed acyclic graphs, collections of incoming documents from different sources, and methods for assigning documents to specific parts of these systems. In these graphs, the top-level nodes represent broad subject areas. Lower-level child nodes represent more specific technical topics. When a document is assigned to a particular node, the framework also counts it under every higher-level parent node along the same branch. This allows researchers to calculate measurements at several levels of detail. Broad categories keep stable overall statistics, while lower levels show more specialized technical developments. The framework can also use several separate topic systems at the same time. This allows researchers to classify documents by different features, such as model architecture, the type of problem being solved, the learning method, and the area of application. These separate dimensions do not have to be forced into one topic structure.

The authors created five measurements to study topic patterns across these topic graphs. The first two measure how popular topics are and how much technical detail documents provide. The first measurement, topic popularity, calculates the share of all labeled documents in a source that belong to a particular node or any of its lower-level branches. This shows how important a subject is within a particular community. The second measurement, average detail depth, shows how technically specific a source is when discussing topics within a branch of the topic system. For each lower-level node, its relative depth is calculated by dividing its distance from the root by the greatest depth found among the lower-level nodes in that branch. The framework then calculates the average relative depth of all assigned document labels in the branch. The result ranges from zero to one hundred percent. Zero percent means all documents are assigned to the root, while one hundred percent means all assignments reach the final leaf nodes. This shows how deeply technical communities discuss their subjects and how specific their descriptions are.

The third measurement, topical diversity, looks at how evenly documents are spread across the lower-level topics under a parent node, rather than being concentrated in only a few subjects. Traditional measurements, such as Shannon entropy and the Gini coefficient, can be difficult to use with topic trees because parent nodes contain more documents than their children. This can make the calculated entropy too low. To address this problem, the authors create an equal baseline by dividing the number of documents assigned to a parent node by the number of leaf nodes in its branch. When calculating diversity, each lower-level node can contribute no more than this baseline amount. This prevents large branches from increasing the score too much. The capped document counts are added together and divided by the product of the number of lower-level nodes and the baseline count. This produces a diversity score between zero and one. The score does not depend on the total number of documents or the number of lower-level nodes. Researchers can therefore compare how evenly topics are distributed across different levels and branches of a topic system.

The fourth measurement looks for real changes in topic activity over time within a single document source, while filtering out random variation. It uses a statistical method called the chi-squared goodness-of-fit test. This test compares the observed number of documents about a topic across time periods of equal length with what would be expected if activity were evenly spread over time. The test is used only for topics with at least five expected documents in each time period, because it needs enough observations to produce reliable results. To prevent the results from depending too much on where the time periods begin and end, the researchers repeat the test using several different starting points. They then combine the resulting p-values using Simes' method. The tests use time periods ranging from six to twenty-four months. Simes' method combines the results across all these time scales into a final score, calculated as the negative logarithm of the combined p-value. Higher scores point to meaningful changes in topic activity, helping researchers find topics that are becoming more or less active without assuming that their growth follows a particular pattern.

The fifth measurement, temporal uniqueness, checks whether a topic changes over time in one selected source differently from how it changes across all the other sources combined. Traditional chi-squared tests for comparing groups do not work well here because some comparison sources have too few documents about certain topics. This can produce expected counts that are too small for reliable testing. To solve this problem, the authors created a statistical model that examines monthly event counts over the period shared by all sources. They combine the other sources into one comparison group and use two related Poisson generalized linear models with log link functions and cubic spline basis functions. The simpler model gives all sources a shared pattern over time but allows each source to have its own starting level. The more complex model adds an extra spline for the selected source, allowing its pattern over time to differ. A likelihood-ratio test compares how well the two models fit the data, using a chi-square distribution to evaluate the test result. The researchers test spline complexity levels from four to ten degrees of freedom and combine the p-values with Simes' method. The final score is the negative logarithm of the p-value. At least ten observed events are required.

To test the framework, the researchers collected data from three current AI sources: arXiv preprints, Hugging Face model cards, and the Deep Learning Weekly newsletter. These sources represent academic research, the release of open-source models, and information selected for AI practitioners. The arXiv dataset included eight AI-related categories from 2013 onward and used paper titles and abstracts. The Hugging Face dataset included model cards published from March 2022 onward and used their original creation dates. The Deep Learning Weekly dataset included individual entries published from April 2019 onward. To compare changes in topics without letting large increases in publication numbers dominate the results, the team used balanced sampling. They randomly selected thirty documents per month from each source, matching the newsletter's minimum monthly volume of thirty-one entries. The documents were assigned to four separate topic systems: Model architecture, AI problem type, Learning paradigm, and Application domain. Each system contained between fifty and two hundred nodes, arranged across three to five levels. These systems covered major AI concepts while allowing the researchers to measure topic patterns.

Documents were assigned to topics using the LLMDap pipeline. This system guides a large language model through the topic graph using a method called constrained decoding. Starting at the root, the model examines the available child nodes. At each step, it can choose a child node, stop moving further down the graph, or mark a top-level topic as not relevant. This ensures that the model only produces valid category choices. The researchers used GPT-4.1-nano with a decoding temperature of 0.4 for all three sources. To check how reliable the assignments were, they tested fifteen randomly selected documents, with five from each source. They examined 121 assigned labels across the four topic systems. Using GPT-4.1-nano, they judged 76 labels correct, 29 incorrect, and 16 uncertain. This gave an estimated accuracy of between 63% and 76%. They repeated the evaluation on the same documents using GPT-5-mini. This time, 90 labels were correct, 9 were incorrect, and 13 were uncertain, giving an estimated accuracy of between 80% and 92%. These findings show that stronger models can improve classification accuracy considerably. They also suggest that the overall measurements remain reliable even when some individual topic assignments are wrong.

The measurements showed clear differences between the three sources in the topics they covered, the amount of technical detail they provided, and how evenly their topics were distributed. For topic popularity, Hugging Face focused heavily on transformer architectures, natural language processing, and adapting or fine-tuning models. This reflects the way its repository is organized and used. In contrast, arXiv papers covered a wider range of model architectures and gave more attention to computer vision and supervised learning. Deep Learning Weekly had the strongest connection to application areas, with only 2% of its entries left unassigned in the Application domain topic system. Changes over time showed important shifts in model architecture. Convolutional networks became more popular and then declined after reaching a peak around 2020. Transformer architectures continued to become more popular. The detail depth scores showed that Hugging Face documents provided more specific information about model architectures than the abstract-level text available from arXiv. For topical diversity, Deep Learning Weekly had the highest scores across most topics, suggesting that its editorial selection covered subjects more evenly. Hugging Face had lower diversity scores because its data included many fine-tuned versions of already popular model architectures.

Statistical tests covering all 403 topic nodes showed that most topics remained stable over time, but some important subfields experienced significant changes. At a significance score of three, which corresponds to a p-value of 0.001, arXiv had the largest number of topics with significant changes over time. This was mainly because its dataset covered a longer period. During the time period shared by all three sources, Hugging Face had the highest number of topics with unique changes over time. To help people explore these findings, the authors created the AI Taxonomy Trend Explorer, an interactive tool with hierarchical sunburst diagrams and charts showing topic changes over time across multiple sources. The authors note that selecting a fixed number of documents each month and using only titles in some parts of the analysis means giving up some detail about changes over time. However, these choices make comparisons between different types of sources fairer. Overall, the framework shows how hierarchical topic graphs can help researchers track changes in technical fields at several levels and across different information sources.

Subject of Research: A hierarchical topic framework for analyzing AI trends across multiple sources

Article Title: Multi-source, multi-level: a hierarchical taxonomy-based framework for AI trends analysis

Article References: Sørbø, S., Jiang, S., Tinn, P., Gheorghe, M., Avogadro, R., & Roman, D. (2026). Multi-source, multi-level: a hierarchical taxonomy-based framework for AI trends analysis. EPJ Data Science, 15(1), Article 82. https://doi.org/10.1140/epjds/s13688-026-00675-w

Image Credits: AI Generated

DOI: 10.1140/epjds/s13688-026-00675-w

Keywords: Artificial Intelligence, Hierarchical Taxonomies, Scientometrics, Trend Analysis, Large Language Models, Machine Learning, Natural Language Processing, Statistical Modeling, Multi-source, multi-level, hierarchical, taxonomy-based