我正在尝试使用tidytext使用双字母组和三字母组合。我可以为令牌使用什么代码查找2个和3个单词。
这是仅使用bigrams的代码：
library(tidytext)
library(janeaustenr)
austen_bigrams <- austen_books() %>%
unnest_tokens(bigram, text, token = "ngrams", n = 2)
austen_bigrams
如果您查看?unnest_tokens，它会告诉您...要传递给令牌生成器的参数。对于ngram，即为tokenizers::tokenize_ngrams，如果您查看其帮助页面，则它具有一个n_min参数，因此您可以执行
library(magrittr)
library(tidytext)
library(janeaustenr)
austen_bigrams <- austen_books() %>%
head(1000) %>% # otherwise this will get very large
unnest_tokens(bigram, text, token = "ngrams", n = 3, n_min = 2)
austen_bigrams
#> # A tibble: 19,801 x 2
#> book bigram
#> <fctr> <chr>
#> 1 Sense & Sensibility sense and
#> 2 Sense & Sensibility sense and sensibility
#> 3 Sense & Sensibility and sensibility
#> 4 Sense & Sensibility and sensibility by
#> 5 Sense & Sensibility sensibility by
#> 6 Sense & Sensibility sensibility by jane
#> 7 Sense & Sensibility by jane
#> 8 Sense & Sensibility by jane austen
#> 9 Sense & Sensibility jane austen
#> 10 Sense & Sensibility jane austen 1811
#> # ... with 19,791 more rows
本文收集自互联网，转载请注明来源。
如有侵权，请联系 [email protected] 删除。
我来说两句