We built a simple NLP tool for counting word frequency

Giuseppe Di Vincenzo

We recently worked on a project with Zeit Online which is analyzing the frequency of words in the Bundestag's (the german parliament) speeches. For research purposes we built a tool for counting words using NLP techniques. The tool removes stop words and transforms the words to a common base (lemmatize) before actually counting the words. It is open source and you can try it here:

How does it work ?

Python was the language of choice because it is one of the most prolific languages for NLP, mostly because of the large ecosystem of stable and complete libraries, like:

NLTK: a widely adopted toolkit for natural language processing
spacy: a complete and deep-learning powered library
TextBlob: a very simple API for NLP operations

We decided to start from the ground up, that's why we choose to try out what's possible using the NLTK library.

Our word-counting tool performs the following operations:

Split the text in sentences using NLTK.sent_tokenize
Split the sentences in words using NLTK.word_tokenize
Lemmatize each word using NLTK for english and GermaLemma for german
Remove stopwords using NLTK
Find most frequent words using collections.Counter.most_common

The tool can be used over HTTP thanks to a small Flask API server.

View the full source for the frontend here and for the backend here.