Data Science · Natural Language Measurement
Beyond Keyword Counting: Sentiment & Theme Classification in Lusophone Social Research
Research Finding: Off-the-shelf English-translated sentiment dictionaries misclassify up to `38%` of informal Angolan Portuguese (`pt-AO`) survey responses due to regional idioms, double negations, and ironic phrasing. Accurate public-opinion measurement requires aspect-level transformer classification calibrated against locally annotated ground truth.
Contents
- Why dictionary-based sentiment fails on Portuguese responses
- Aspect-Based Sentiment Analysis (ABSA) vs. document-level polarity
- Handling `pt-AO` idioms, phonetic spelling and code-switching
- Unsupervised theme discovery with density-based clustering
- Post-stratification and confidence intervals for decision-makers
1. Why dictionary-based sentiment fails on Portuguese responses
Legacy social-listening and survey analysis tools rely on bag-of-words polarity lexicons: they count positive words, subtract negative words, and output a single score. When applied to open-ended responses in Portuguese — and particularly to everyday Angolan Portuguese (`pt-AO`) collected through digital channels — lexicon counting collapses for three reasons:
- Contextual negation and contrastive clauses: A respondent writing “O atendimento na agência até foi rápido, mas o aplicativo trava sempre que tento pagar um serviço” expresses positive sentiment toward branch staff and negative sentiment toward mobile banking reliability. A single document-level score (`0.0 Neutral`) erases both actionable signals.
- Regional idioms and colloquialisms: Expressions common in Luanda or Benguela (such as “está mesmo na bala” for excellence, or ironic uses of “muito bonito mesmo” following a service outage) are either absent from European/Brazilian lexicons or assigned the opposite polarity.
- Orthographic variation on mobile keyboards: Respondents typing on mobile devices frequently omit diacritics (`nao`, `servico`, ` tambem`), use phonetic abbreviations (`vc`, `pq`, `tb`), or blend Portuguese with Kimbundu, Umbundu, or English loanwords.
2. Aspect-Based Sentiment Analysis (ABSA) vs. document-level polarity
In the Q-Mvutu research engine, we never assign a single scalar sentiment to a multi-clause response. Instead, every open-ended text response passes through a two-stage Aspect-Based Sentiment Analysis (ABSA) pipeline:
- Span & Aspect Extraction: The model segments the response into distinct thematic propositions (for example: `["branch waiting time", "mobile app stability", "transaction fees"]`).
- Aspect-Conditioned Polarity & Intensity Scoring: For each extracted aspect, a fine-tuned multilingual transformer classifies polarity (`positive`, `neutral`, `negative`, `mixed`) and urgency (`low`, `operational_blocker`, `critical`) along with the supporting text span.
| Analysis Method | Macro-F1 on `pt-AO` Test Set | Mixed-Clause Accuracy | Inference Latency (Batch) |
|---|---|---|---|
| Lexicon / Keyword Dictionary | `0.61` | `34.2%` | `< 2ms` |
| Zero-Shot Generic Multilingual LLM | `0.82` | `78.5%` | `~950ms` |
| TICNES Fine-Tuned Distilled ABSA Pipeline | `0.93` | `91.8%` | `< 65ms` |
3. Handling `pt-AO` idioms, phonetic spelling and code-switching
To achieve a `0.93` Macro-F1 score on local corpora while keeping inference latency under `65ms`, we train a distilled encoder student model using a curated domain corpus of Lusophone survey responses annotated by native Angolan and Portuguese linguists. During pre-processing, we deliberately avoid destructive stemming; instead, we use subword tokenization paired with synthetic data augmentation that randomly strips accents and introduces common mobile keyboard abbreviations during training. As a result, the encoder maps accented formal text and unaccented mobile shorthand to nearly identical representations.
4. Unsupervised theme discovery with density-based clustering
Pre-defined category taxonomies only capture issues the survey author anticipated in advance. When a new operational failure emerges — for example, a specific mobile money USSD gateway failing after a telecom update — fixed taxonomies lump those complaints into an unhelpful “Other” bucket.
Alongside supervised aspect classification, our data science pipeline computes dense sentence embeddings for all “Other” and open-ended responses, projects them using dimensionality reduction (UMAP), and runs HDBSCAN (Hierarchical Density-Based Spatial Clustering) to isolate emerging high-density topic clusters automatically. Each discovered cluster is summarised into a human-readable label with representative verbatim quotes and surfaced on the analyst’s telemetry dashboard.
5. Post-stratification and confidence intervals for decision-makers
Raw digital survey counts never represent a population census; younger urban respondents with smartphones are almost always over-represented relative to older or rural cohorts. Reporting raw percentages without demographic weighting misleads executives and policy-makers.
Every research report generated by TICNES AI data science systems includes:
- Unweighted sample counts (`n`) alongside post-stratified weighted estimates calibrated against known demographic or customer-base distributions.
- 95% credible/confidence intervals and design-effect adjustments so decision-makers know whether a `3%` shift between monthly waves is a genuine trend or sampling noise.
- Auditability down to the row level: Clicking any bar in a sentiment distribution chart displays the anonymised verbatim responses that contributed to that bucket.