Tokenization splits text into sentences and words, filtration removes unwanted tokens (punctuation, numbers, stop words), and script validation keeps only tokens written in the expected script (here, English/Latin letters).
1. Tokenization: breaking a text into units. Sentence tokenization splits at sentence boundaries (. ! ?); word tokenization splits sentences into words.
2. Filtration: cleaning the tokens: converting to lower case, removing punctuation, special symbols and numbers, and removing stop words (very common words such as "is", "the", "in" that carry little meaning).
3. Script validation: checking that each token is in the valid script/character set for the language being processed, and rejecting tokens in other scripts (e.g. Devanagari in an English pipeline), accented or garbled characters, emojis, etc.
Program (Python, no external library)
import re, string
text = "NLP is fun!! I scored 95 marks in the NLP test. नमस्ते friends, let's learn at the café"
# 1. Tokenization
sentences = re.split(r"(?<=[.!?])\s+", text)
tokens = [w.strip(string.punctuation) for w in text.split()]
tokens = [t for t in tokens if t] # drop empty strings
# 2. Filtration: lower-case, remove numbers and stop words
stop_words = {"is", "i", "in", "the", "a", "an", "and", "of", "to", "at"}
filtered = [t.lower() for t in tokens
if not t.isdigit() and t.lower() not in stop_words]
# 3. Script validation: accept only English (Latin-script) words
valid = [t for t in filtered if re.fullmatch(r"[a-z]+('[a-z]+)?", t)]
rejected = [t for t in filtered if t not in valid]
print("Sentences:", sentences)
print("Tokens: ", tokens)
print("Filtered: ", filtered)
print("Valid: ", valid)
print("Rejected: ", rejected)Output
Sentences: ['NLP is fun!!', 'I scored 95 marks in the NLP test.', "नमस्ते friends, let's learn at the café"]
Tokens: ['NLP', 'is', 'fun', 'I', 'scored', '95', 'marks', 'in', 'the', 'NLP', 'test', 'नमस्ते', 'friends', "let's", 'learn', 'at', 'the', 'café']
Filtered: ['nlp', 'fun', 'scored', 'marks', 'nlp', 'test', 'नमस्ते', 'friends', "let's", 'learn', 'café']
Valid: ['nlp', 'fun', 'scored', 'marks', 'nlp', 'test', 'friends', "let's", 'learn']
Rejected: ['नमस्ते', 'café']The text is split into 3 sentences and 18 word tokens; filtration removes the number 95 and stop words (is, I, in, the, at); script validation rejects the Devanagari word and the accented word "café". With NLTK, the same steps use sent_tokenize, word_tokenize and stopwords.words('english').