Political Analysis· 2018Q1
Text Preprocessing For Unsupervised Learning: Why It Matters, When It Misleads, And What To Do About It
- 411citations
- Q1SCImago
- 2018year
Short summary
Preprocessing choices in unsupervised text analysis profoundly impact results, yet this sensitivity is often overlooked in political science.
AI-generated from the title and abstract; the full text is not read.
Key points
- Preprocessing choices in unsupervised text analysis have profound effects on model results.
- Substantive theory and supervised learning literature offer limited guidance for unsupervised feature selection.
- A new statistical procedure and software are introduced to examine preprocessing sensitivity.
- This approach helps researchers understand the variability induced by preprocessing choices and aids replication.
AI-generated from the title and abstract; the full text is not read.
Abstract
Despite the popularity of unsupervised techniques for political science text-as-data research, the importance and implications of preprocessing decisions in this domain have received scant systematic attention. Yet, as we show, such decisions have profound effects on the results of real models for real data. We argue that substantive theory is typically too vague to be of use for feature selection, and that the supervised literature is not necessarily a helpful source of advice. To aid researchers working in unsupervised settings, we introduce a statistical procedure and software that examines the sensitivity of findings under alternate preprocessing regimes. This approach complements a researcher’s substantive understanding of a problem by providing a characterization of the variability changes in preprocessing choices may induce when analyzing a particular dataset. In making scholars aware of the degree to which their results are likely to be sensitive to their preprocessing decisions, it aids replication efforts.
The authors' abstract, as published at the source. Political Analysis, 2018 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: General Social Sciences
General Social SciencesSocial Sciences