Embracing Classical Text Classification Techniques: A Case Study with the arXiv Dataset

The Starting Point:  Pre-processing

Before diving into the complex world of keywords and classification, we had to clean up our dataset. We began by taking the raw titles and abstracts from around 40,000 arXiv papers and giving them a good scrub. This meant:

• Lowering all letters to avoid “Apple” being different from “apple.”
• Getting rid of web addresses, email IDs, and anything HTML.
• Breaking down the text into individual words or “tokens.”
• Saying goodbye to common names and non-essential words, like “the” and “is.”
• Keeping our text purely alphabetical and straightforward—no numbers or symbols allowed.
• Simplifying words to their root form, so “running” just became “run.”
• Removing Pronouns such as “Adam”, “Smith” etc
• Sticking to plain ASCII characters to maintain text uniformity‍

Initially, we stuck to single words but soon realized that pairs and triplets of words gave us a much richer context. So, we expanded our scope to include bi-grams and tri-grams, adding depth to our understanding.

Finding the Right Words: Identifying and Filtering Keywords

The heart of our project was all about finding those special words that could help us classify papers accurately. We started by building a word bank from our cleaned-up data, focusing on words that stood out in specific categories. The crux of our methodology lies in the calculation of relative frequency for each keyword, using the formula

This calculation is crucial as it helps us understand the significance of a keyword within its category compared to other categories. 

But, not all words that shine are gold. Some were too common, telling us nothing new. Others were so rare they barely made a dent. 

For example: This image contains words that are either too common in the “Mathematics” category or too rare in the “Non-Linear Sciences” category.

So, we added more strategic filtering, to ensure we get the best representative keywords for each category at the end of this exercise:

• We ignored words that were everywhere or nowhere (infinite frequency, or Words with the highest overall frequencies).
• We chose words that were at least somewhat common in their category, with a frequency of 0.5 or higher.
• We looked for a sweet spot: not too rare (appearing in more than 50 papers) but not too common (less than 8,000 appearances).

After the filtering process, the distinction was very prominent in some of the categories, such as for Astrophysics.

The Final Leap: Classification

Once we have our distilled set of distinct keywords for each category, we move to the crucial stage of classifying text. We devised a simple scoring mechanism to help us assign probabilities for each category we assign to a paper.

Scoring Formula

With our carefully curated list of keywords, we were ready to classify new papers. We calculate a score for each category. That is a weighted sum, in which each keyword’s contribution depends on its frequency in the text and its predefined weight, derived from our earlier analysis.

Using this formula, we end up with a paper and its classification into several categories along with their scores, which are then normalized to present as probabilities from 0-100%

In the following example, the actual category for the paper is “Condensed Matter” and as seen in the predicted categories, the “Condensed Matter” has the highest probability of 96%. And keywords like “atomic gas”, “impurity” and “ultracold” contributed to this classification.