The main objective of this project is to develop and compare systems capable of determining whether an opinion expressed in a film review is positive or negative. Two complementary approaches have been implemented and evaluated: supervised and unsupervised models.
The supervised approach involved several classification algorithms, of which Random Forest (RF) was selected as the best model.
-
Preprocessing: Contraction normalization, symbol cleaning (preserving numerical fractions like
$4/5$ ), tokenization, Part-of-Speech (POS) tagging, and contextual lemmatization. - Stopwords: Application of a manually refined stopwords list to ensure the preservation of negations and intensifiers.
- Vectorization: Using CountVectorizer with unigrams (min_df=2).
The Random Forest model was optimized using GridSearchCV.
| Metric | Value (Test Set) |
|---|---|
| Accuracy | 87.75% |
| Weighted F1-score | 87.75% |
| Accuracy (Train) | 93.63% |
| AUC (ROC Curve) | 0.94 |
Error Analysis: The model demonstrated good generalization ability (no overfitting). Primary errors (49 total) occur in reviews with subtle irony, sarcasm, or very brief/ambiguous positive opinions (False Negatives).
Two strategies were compared: a system based on SentiWordNet (with Lesk word sense disambiguation and negation handling) and the VADER model.
The VADER model was chosen as the best performer, mainly due to its internal handling of negations and intensifiers. An optimized compound threshold of 0.9 was used.
| Metric | Value (Test Set) |
|---|---|
| Accuracy | 69.75% |
| F1-score | 69.46% |
The VADER model tends to overestimate positive polarity (80 False Positives vs. 41 False Negatives). The main limitations involve detecting nuance, irony, and failing to capture the critical context when reviews start with positive language.
The results confirm that, when labeled data is available, supervised models are clearly superior.
| Model | Accuracy (Test) | F1-score (Test) | Primary Use Case |
|---|---|---|---|
| Random Forest (Supervised) | 0.8775 | 0.8775 | Scenarios with annotated data |
| VADER (Unsupervised) | 0.6975 | 0.6946 | Scenarios with little or no labeled data |
Error analysis showed the supervised model is more robust, failing exclusively in 45 cases, while the unsupervised model failed exclusively in 87 cases.