What stop words are
Open any English text and count how often each word appears. Before you have read very far, a pattern emerges: a handful of words dominate. “The”, “of”, “and”, “a”, “in”, “to”, “is”, “that” — these turn up on almost every line of almost every document. They are the connective tissue of language, the words that hold sentences together grammatically without saying very much about the subject.
These are stop words. The term was coined in information retrieval research in the 1960s when computing resources were scarce and storing or searching every word in a large corpus was expensive. The practical solution was to identify the most common, least informative words and exclude them from indexes. The list of excluded words was called the stop list, and the words on it became known as stop words.
The intuition is straightforward. If you are searching a library database for documents about “the effects of climate change on ocean temperature”, the words “the”, “of”, “on” appear in virtually every document in the collection and therefore contribute nothing to distinguishing relevant from irrelevant results. Removing them and searching for “effects climate change ocean temperature” is not only computationally cheaper — it produces better results because it focuses on the content words.
The vocabulary of stop words
Stop words are almost always drawn from the same grammatical categories regardless of which language you are working in. Articles (“the”, “a”, “an”) form the smallest category but some of the highest-frequency words in English. Prepositions (“in”, “of”, “at”, “by”, “for”, “with”, “about”, “against”, “between”) describe spatial and logical relationships. Conjunctions (“and”, “but”, “or”, “nor”, “yet”, “so”) link clauses. Auxiliary and copula verbs (“is”, “are”, “was”, “were”, “be”, “been”, “have”, “has”, “had”, “do”, “does”, “did”, “will”, “would”, “could”, “should”) provide tense and modality. Pronouns (“I”, “you”, “he”, “she”, “it”, “we”, “they”, “them”, “their”) replace nouns. Determiners and quantifiers (“this”, “that”, “these”, “those”, “some”, “any”, “few”, “more”, “most”, “other”) specify quantity and identity.
In English, the standard NLTK stop word list (from the Natural Language Toolkit, the most widely used Python NLP library) contains around 179 words. The Scikit-learn default stop list has 318. The Stanford NLP group’s list has a different composition. There is no single authoritative stop word list because what counts as a stop word depends on the task.
Languages with rich inflectional morphology have different stop word structures. German has more distinct preposition and article combinations because articles change form by case (der, die, das, den, dem, des). Japanese stop word removal is more complex because the language does not use spaces between words, so tokenisation and stop word removal must happen together.
How search engines changed their approach
For most of the 1990s and 2000s, major search engines including early Google ignored stop words in queries. A search for “to be or not to be” was treated as a search for “be” — the most semantically distinctive word left after removing all the others. This was a pragmatic engineering choice: stop words appeared in billions of documents and storing posting lists for them was expensive, while they rarely helped distinguish relevant from irrelevant results.
This changed as search volume grew, hardware costs fell, and user queries became more complex. Google gradually started processing more stop words as ranking signals rather than ignoring them. The change accelerated dramatically in 2019 with the rollout of BERT (Bidirectional Encoder Representations from Transformers), which Google described as one of the most significant improvements to Search in five years.
BERT processes queries as sequences of tokens where every word, including stop words, contributes to the model’s understanding of meaning. The example Google used to illustrate BERT’s capabilities involved the query “2019 brazil traveler to usa need a visa”. The preposition “to” is critical to interpreting this query correctly — it is a Brazilian looking for information about visiting the USA, not an American seeking information about Brazil. Without understanding “to”, the query looks the same as “2019 usa traveler to brazil need a visa”. BERT handles this distinction correctly because it does not discard the preposition.
The practical takeaway for anyone doing text analysis with search applications in mind is that stop words are no longer a reliable proxy for “unimportant words”. Their importance depends entirely on context.
When stop word removal helps
Stop word removal is genuinely valuable in a specific set of situations. Topic modelling is the clearest case. Algorithms like Latent Dirichlet Allocation try to discover what topics a collection of documents covers by looking for words that co-occur across the same documents. Stop words co-occur everywhere, so they distort the topic signals. Removing them before running LDA makes the discovered topics much cleaner and more interpretable.
Word frequency analysis for content understanding follows the same logic. If you paste an article into the Word Frequency Counter without stop word removal, the top results are “the”, “and”, “of”, “a”, “to” — telling you nothing about the document’s content. With stop word removal enabled, the top results are the actual subject matter of the piece. For content auditing, SEO keyword analysis, and writing revision, this is the configuration you want.
Word cloud generation benefits enormously from stop word filtering. A word cloud from an unfiltered text is a visual representation of Zipf’s Law, not of the document’s meaning. The largest word is always “the”. Filtering stop words first produces a cloud that genuinely communicates topical emphasis, making it useful for presentations, visualisations, and quick document summaries.
Sentiment analysis can also benefit. Sentiment models try to determine whether text is positive, negative, or neutral. Most sentiment is carried by content words — “excellent”, “terrible”, “disappointing”, “wonderful” — not by prepositions and articles. Removing stop words reduces the noise that non-sentiment words introduce, though modern deep learning models often handle this implicitly.
When stop word removal hurts
Phrase matching breaks down when stop words are removed. If you are searching for exact phrases — song lyrics, legal clauses, title matches, or specific quotations — removing stop words can destroy the phrase. “To be or not to be” becomes “be be” after stop word removal, which is useless. The phrase “state of the art” loses “of the” and just becomes “state art”, which is ambiguous. Any application that requires understanding multi-word phrases in sequence needs to keep stop words.
Intent detection is another area where stop word removal causes problems. As the BERT example illustrates, prepositions like “to”, “from”, “for”, and “by” frequently carry the semantic weight that distinguishes one user intent from another. “Flights to London” and “flights from London” become identical after stop word removal, but they represent opposite travel directions. Question-answering systems, conversational AI, and any application concerned with the precise meaning of queries need to preserve stop words.
Negation is perhaps the most consequential case. “Not good” and “good” are antonyms, but if “not” is on your stop list, they look identical after filtering. Sentiment analysis systems that naively remove stop words can flip the polarity of a sentence. Most modern sentiment tools handle negation as a special case rather than including “not” in the stop list, but off-the-shelf stop lists often include it.
Stop words and word clouds
The effect of stop word choice on word cloud output is dramatic enough to be worth illustrating separately. Take a 1,500-word news article about an election. With all stop words included, the cloud’s largest words will be “the”, “and”, “of”, “said”, “a”. With a basic English stop list applied, you start to see “election”, “candidate”, “vote”, “party”. With a domain-specific political text stop list that also removes “said”, “according”, “told”, “described” (common journalistic filler), the cloud reveals the actual political concepts discussed.
The Word Frequency Counter lets you toggle stop word removal and adjust settings so you can compare the outputs and find the configuration that works for your specific document.
Building a custom stop list
Generic stop word lists are designed for general English text. When you are working with specialised material, you often need to supplement them. A financial document analysis might need to add “company”, “business”, “market”, “annual”, “report” — words that appear constantly in financial filings without identifying anything specific about the company being discussed. A support ticket analysis might add “please”, “hello”, “thanks”, “ticket”, “issue”, “customer” — the standard politeness and ticket-vocabulary that dominates every entry.
The practical approach is to start with a standard stop list, run your frequency analysis, and look at the top 50 content words. Words that appear very frequently but contribute nothing to distinguishing one document from another in your collection are candidates for your custom stop list. Add them, re-run, and repeat until the top words genuinely represent the topics and concepts you care about.
Custom stop lists are also essential when working across languages. A bilingual corpus will have stop words from both languages; a standard monolingual list will only filter one of them. Domain-specific terms that function as stop words in one field are meaningful content words in another — “apply” is a stop word candidate in a job posting analysis but a meaningful content word in a chemistry text.
Stop words are a deceptively simple concept with significant practical consequences. Understanding when to remove them, when to keep them, and how to adapt a generic list to your specific task is one of the foundations of effective text analysis. For a deeper look at how frequency analysis works once stop words are out of the way, the word frequency analysis guide covers the full workflow including n-gram analysis, SEO keyword density, and content auditing steps.