Words begin inside complete speaking turns, not in a ready-made numeric table. To compare speakers, we must turn text into analysable rows and standardize the counts. We must then return to the transcript because frequency alone cannot explain what a speaker meant.
By the end of this case, you should be able to:
The original transcript is a text file rather than a rectangular table. A preparation script separates it into speaker turns and saves a teaching CSV. This lets the lesson focus on tidy text analysis instead of loops and custom parsing functions.
The reproducible preparation is available in scripts/prepare_debate.R for readers who want to study it later.
If needed, install tidytext once in the Console with install.packages("tidytext"). tidytext is a companion to the tidyverse: it turns language into tidy rows that can be handled with familiar verbs such as filter(), count(), and joins. Then load the packages:
Open djr.Rproj and create 12-text-analysis.Rmd. Download debate_turns.csv and save it in data/. The preparation script is optional; the lesson starts from this ready-to-import CSV.
library(tidyverse)
library(tidytext)
turns <- read_csv("data/debate_turns.csv")
glimpse(turns)Rows: 230
Columns: 2
$ speaker <chr> "PARTICIPANTS", "MODERATORS", "MUIR", "DAVIS", "MUIR", "DAVIS"…
$ text <chr> "Vice President Kamala Harris (D) and Former President Donald …One row represents one speaking turn.
We will solve the text challenge in six steps:
filter() keeps the candidate speakers, and recode() replaces their all-capital labels with names that are easier to display:
candidate_turns <- turns |>
filter(speaker %in% c("HARRIS", "TRUMP")) |>
mutate(
speaker = recode(
speaker,
"HARRIS" = "Harris",
"TRUMP" = "Trump"
)
)
candidate_turns |>
count(speaker)The row counts describe speaking turns, not speaking time or number of words.
unnest_tokens() from tidytext turns each speaking turn into separate word rows. It needs a name for the new word column and the name of the original text column:
tokens <- candidate_turns |>
unnest_tokens(word, text)
tokens |>
slice_head(n = 10)One row now represents one word token associated with a candidate.
The tidytext package provides a table called stop_words, containing common English words such as “the” and “of.” anti_join() keeps tokens that do not match that reference table:
content_words <- tokens |>
anti_join(stop_words, by = "word")Stop-word removal is an analytical choice. Words such as “not” may be important in political speech, so inspect the list and describe the rule in the methods.
First count all tokens spoken by each candidate:
token_totals <- tokens |>
count(speaker) |>
rename(all_tokens = n)Then count each content word and join the denominator:
word_rates <- content_words |>
count(speaker, word) |>
rename(uses = n) |>
left_join(token_totals, by = "speaker") |>
mutate(uses_per_1000_tokens = uses / all_tokens * 1000)A rate is more comparable than a raw count when the candidates spoke different numbers of words.
slice_max() keeps rows with the largest value. Group first so it selects the largest rates separately for each candidate:
top_words <- word_rates |>
group_by(speaker) |>
slice_max(uses_per_1000_tokens, n = 10)
top_wordsCreate a simple comparison chart:
top_words |>
ggplot(
aes(
x = uses_per_1000_tokens,
y = fct_reorder(word, uses_per_1000_tokens),
fill = speaker
)
) +
geom_col() +
facet_wrap(~ speaker) +
labs(
title = "Frequently used content words differed between the candidates",
x = "Uses per 1,000 candidate tokens",
y = NULL,
caption = "Source: supplied debate transcript"
) +
theme_minimal()
Frequency does not measure importance, sincerity, truthfulness, or audience effect.
Single words lose context. unnest_tokens() can also create adjacent two-word phrases, called bigrams. Here token = "ngrams" selects word groups and n = 2 sets their length:
bigrams <- candidate_turns |>
unnest_tokens(bigram, text, token = "ngrams", n = 2) |>
count(speaker, bigram) |>
arrange(desc(n))
bigrams |>
slice_head(n = 20)Bigrams preserve a little more context but still do not capture full sentence meaning, sarcasm, quotation, or who a pronoun refers to.
Before using a word in a story, find it in the original speaking turns:
candidate_turns |>
filter(str_detect(str_to_lower(text), "immigration")) |>
select(speaker, text)Read several examples. Check whether the candidate is making a claim, answering a moderator, quoting an opponent, or denying something.
Choose one topic and make a short list of related words. Then:
| Function | What it does in this case |
|---|---|
unnest_tokens() |
Turns speaking turns into word or phrase rows |
anti_join() |
Removes words found in the stop-word table |
count() |
Counts tokens by candidate and word |
left_join() |
Adds the token denominator to each word count |
slice_max() |
Keeps the most frequent words within each candidate |
str_detect() |
Returns counted words to their original context |
This case uses only words and bigrams. The official unnest_tokens() reference explains other token types and controls for later text projects.
The four cases began with the familiar tidyverse workflow but concentrated on different problems that journalists encounter in real datasets:
| Dataset | Unique challenge | Main strategy |
|---|---|---|
| Billboard Hot 100 | Titles alone do not identify songs, and “success” has several meanings | Use artist-title combinations and define the metric before ranking |
| Olympics | Athlete-medal rows and unique medallists are not official medals | Identify the unit, use stable athlete identifiers, and create event-level records |
| Graduate employment | The table mixes detailed and aggregated rows | Split, aggregate, recombine, reshape, and construct the denominator |
| Debate transcript | Word counts remove language from its context | Tokenize, standardize rates, and return to the original transcript |
Across all four cases, the most important question comes before the code:
What does one row represent, and does that unit match the claim we want to make?
Functions make the calculation reproducible. Defining the observation, metric, and denominator makes the result meaningful.