Text Mining & NLP

42.1 tidytext

library(tidytext); library(dplyr); library(janeaustenr)

words <- austen_books() |>
  unnest_tokens(word, text) |>                # one token per row
  anti_join(stop_words, by = "word")

words |> count(word, sort = TRUE) |> head(10)

# Sentiment analysis
words |> inner_join(get_sentiments("bing"), by = "word") |>
  count(book, sentiment) |> tidyr::pivot_wider(names_from = sentiment, values_from = n)

# TF-IDF: words characteristic of each book
words |> count(book, word) |> bind_tf_idf(word, book, n) |> arrange(desc(tf_idf))

# n-grams
austen_books() |> unnest_tokens(bigram, text, token = "ngrams", n = 2)

42.2 Other text tools

quanteda — fast corpus/document-feature matrices, keyness, readability.

topicmodels / stm — LDA and structural topic models.

text2vec, word2vec, text (transformers via Python) — embeddings.

udpipe, spacyr — POS tagging, lemmatization, dependency parsing.

wordcloud2, ggwordcloud — word clouds.

stringi — full Unicode text processing (stringr is built on it).

LLM access from R: ellmer (chat with OpenAI/Anthropic/Gemini/Ollama models, tool calling, structured output), chores, mall, gander.