preprocessing.tex

\subsection{Preprocessing}
The dataset from SemEval contains a thousands of articles. These articles cannot be fed into a form of artificial intelligence immediately. Several approaches are using the following preprocessing:

\subsubsection{Tokenization}
Tokenization is the process of dividing text into sentences and sentences into words and punctuation marks.
From tokenized text, further preprocessing can be applied.

\subsubsection{Stemming}
Stemming is the process of removing inflection from words.
Usually only the stem is necessary information to abstractly understand context.
An example is reducing ``stemming'' and ``stems'' to ``stem''.

\subsubsection{Lemmazation}
An alternative method to stemming.
In exchange for speed, the accuracy of removing inflection improves.
This greatly improves accuracy for texts with multiple instances of words with similar inflections.
An example is ``leaf/leaves'' and ``leave/leaves''.

\subsubsection{Stopwords}
Stopwords are words that do not influence the context or meaning of a sentence.
For example, words such as ``the'', ``and'', and ``or'' can usually be considered stopwords.
Filtering out stopwords improves accuracy and performance due to removing unnecessary words.