Machine Learning: Working With Stop Words, Stemming, and Spam

in Development

You may have already taken a look at my previous blog post, where we talked about machine learning and training the classifier without treating the text. Today I’m going to dig deeper into machine learning, as a follow-up to my last blog post. We’ll focus on text-processing techniques to enhance the quality of the classifier output. By acting on the quality of the training data, we can change what is called the accuracy of the classifier. Let’s get started!

Stop Words

Stop words are the words we want to filter out before training the classifier. These are usually high frequency words that aren’t giving any additional information to our labeling. In fact, they actually confuse our classifier. In English, they could be the, is, at, which, and on — you can check out a list of common stop words here.

The important words we want to include when we’re training text are the words that should be connected in some way to the classification label. Let’s take a quote from a freely available random article of the Financial Times.

First, we need a list we can use in our code. A good node library we can use is stopwords — it’s just an array of English stop words.

// filter_stopwords.js

var stopwords = require('stopwords').english;

var textWithoutStopwords =
    .filter((w)=> { return stopwords.indexOf(w.toLowerCase()) < 0 })
    .join(" ")

Being rawText our unfiltered text, in the first line, we load an array of stop words using stopwords library. We then split the original text in words — also clearing out symbols and digits — and filter out the words in the stopwords array, and then convert everything back to a string.


Let’s start with what stemming is — “Stemming is the process for reducing inflected words to their word stem (base form).”

You can see above that banks and banking become bank, and investing and invested become invest. The classifier doesn’t understand that the verbs investing and invested are the same, and treats them as different words with different frequencies. By stemming them, it groups the frequencies of different inflection to just one term — in this case, invest.

Let’s stem the previous quote where we filtered out the stop words:

After filtering out the stop words, we can stem the remaining words using the natural node library, which is not only a stemming library, it’s a complete toolset for natural language text processing (bonus: it has also stop words and classifiers).

// stemming.js

var stopwords = require('stopwords').english;
var natural = require('natural');

var stemmedAndNoStopwords =
    // stopwords filtering
    .filter( (w)=> {
        return stopwords.indexOf(w.toLowerCase()) < 0
    // stemming
    .map( (word)=> {
        return natural.PorterStemmer.stem(word);
    .join(" ")

Once joined with the stemmed words, the text is ready to train the classifier or to be classified.

Accuracy Score

The accuracy score indicates how accurate the classifier is trained by our training data. This score depends on the classifier algorithm itself and by the quality of the training data.

We’ll see it better in the next blog post when we’ll use it to see how precise our spam filter is and how good the training dataset is.

Spam or Ham Project

Now we have the tools we need to build a spam filter with decent accuracy. First, the classification of the spam and non-spam (ham) needs good training data. Let’s walk through it with this training and test data you can download here. For this exercise, we just need these two files:

  • — which contains 2,500 raw, real emails.
  • — which is a text file, mapping the email id’s with the prediction: 0 is spam and 1 is ham. On each line, there’s the id of the email and its prediction.

Look at line 1 — the email with id 1 is spam

That checks out — it definitely looks like a spam email.

Now line 3 — the email with id 3 is ham.

This checks out as ham, and it’s definitely an email we’d like to keep.

To get the text, title, and address of each email, you’ll need a mail parser like the mailparser node library. Using the fs node module, we’re able to read the raw data of the email and then parse it using mailparser!

Ready to use some emails to test your spam filter? You can download test emails with the file You could also try to classify your own emails, connecting to your mailbox using inbox library and mailparser together. Now you’re ready to start trying it out on your own, and practice! And remember, if you need some help or have a tip, just post a comment below.

Code School

Code School teaches web technologies in the comfort of your browser with video lessons, coding challenges, and screencasts. We strive to help you learn by doing.


About the Author

Alvise Susmel

Alvise Susmel

Alvise Susmel is Software Architect in a London-based Hedge Fund, working on a big-data platform which analyses and processes crucial investments’ data. He loves learning new technologies, sharing the passion and building services, like

Might We Suggest