Tutorials Token Frequency
Token Frequency counts how often each word appears in your text data. It is one of the quickest ways to spot themes and jargon in a corpus. The tool offers two views: Cloud view for a visual impression of the most frequent terms, and List view for a ranked frequency list with precise counts. When two data blocks are selected, the tool also produces a comparative keyword analysis — the Juxtorpus cloud — and a statistical measures table that highlight the terms most distinctive to each side.
Use the data-block selector to choose which corpus (or corpora) to analyse. The tool is strictly pairwise — at most two data blocks at a time — because keyword analysis is defined between exactly one reference and one study corpus. If more than two blocks are selected at the workspace level, the tool only shows the two most recent picks; any older selections are silently dropped from the panel until you deselect a newer block to make room.
When two are selected, the tool runs in comparison mode and produces the Juxtorpus cloud and statistical measures in addition to the per-block results.
For each selected block, choose the text column that contains the documents you want to count and a tokenizer model that defines the tokens. Only columns that hold plain text are available. Each choice is saved independently on that data block and initializes the corresponding control the next time you add it to a fresh Token Frequency or Concordance selector. Clearing one does not clear the other.
Every Token Frequency run requires a tokenizer model for every selected data block. The Analysis stores the exact model mapping it used, so reopening a historical result does not substitute a tokenizer preference that was changed later.
When two data blocks are selected, each selected-node card shows a Study Corpus / Reference Corpus switch. Switching one card to Study Corpus automatically makes the other card the reference, and switching one card to Reference Corpus makes the paired card the study corpus.
The reference block provides the baseline for the statistical keyword analysis: its frequencies appear as O1 and %1 in the statistics table, and the study block appears as O2 and %2. Swapping the corpus roles flips which side each statistic measures from, which can change the sign of directional measures like LogRatio.
Stop words are terms you want to exclude from the frequency count — commonly words like the, and, or domain-specific filler that would otherwise dominate the results.
- Type words separated by spaces into the stop words field. Matching is case-insensitive.
- Click Fill Default to populate the field with default stop words for the saved language of each selected column.
- Click Sort to sort the current stop-word list alphabetically.
- Click Apply Stop Words to apply the current list to the results. Removing stop words does not change the statistical measures of remaining tokens — they are excluded as a post-processing step.
- Right-click any word in the word cloud or frequency list to add it directly to the stop-word list. Words added this way are inserted at the start of the list so they are easy to find and remove. The list is not re-sorted until you click Sort.
Click Run to start. After the tab has an Analysis, the button changes to Re-run. A successful Analysis can be re-run after you change its inputs or settings; a failed or cancelled Analysis can be retried with the same settings.
The results panel shows controls for stop words and display limits at the top, followed by a Cloud view / List view tab to switch between the two output modes.
The Cloud display limit (range 10–100, default 50) sets the maximum number of tokens shown in the word clouds. Changing this value also updates the List display limit to the same number (capped at 100).
The List display limit (range 10 – vocabulary size) sets the maximum number of tokens shown in the ranked frequency lists. Values up to 100 stay in sync with the Cloud display limit; setting the list limit above 100 lets you see a longer tail in list view while the cloud remains capped at 100.
Cloud view shows a word cloud for each selected data block, followed by the Juxtorpus cloud when two blocks are selected.
Word size in each per-block cloud corresponds to frequency. Interaction:
- Left-click any word to jump to the Concordance tab and search for that term in context.
- Right-click any word to add it to the stop-word list (it is inserted at the start of the list).
A download button is available for each cloud (PNG, SVG, or PDF). You can optionally include the associated stop-word list in the download as a zip file.
When two data blocks are selected, the Juxtorpus cloud appears below the per-block clouds. It highlights the words that are most distinctively used by each block, using the keyword analysis method (log-ratio comparison).
- Size reflects combined frequency across both blocks.
- Colour shifts toward the block where the word has the higher proportional share, so differences in corpus size do not dominate the palette.
- Words are ranked by log₁₀(O₁ + O₂) × LogRatio; the cloud shows the highest and lowest N words by that score (up to twice the cloud display limit).
List view shows a ranked horizontal bar chart for each selected data block, with the statistics table below when two blocks are compared.
Word ranking
Tokens are listed in descending order of frequency. The bar length for each token is proportional to its count relative to the most frequent token in that block. When two data blocks are shown side by side, the blocks are scrolled synchronously — scrolling one list scrolls the other to the same position, making it easy to compare the same rank across both corpora.
A Filter tokens input appears at the bottom of the results when List view is active. Type a pattern to narrow both frequency lists and the statistics table simultaneously. Use * as a wildcard:
-
pre*— all tokens starting with pre -
*ing— all tokens ending in ing -
*ation*— all tokens containing ation
Click Clear to remove the filter. The word clouds are not affected by the token filter.
The Keyword Analysis table summarises token-level differences between the two data blocks. A caption directly under the section heading shows Reference corpus: {name}; Study corpus: {name} with each name coloured to match the chart palette colour you picked for that block — so it's always clear at a glance which side of the comparison is which. Click any column header to sort ascending or descending.
| Column | What it shows |
|---|---|
| O1 / O2 | Observed frequency in each data block (O1 = reference block) |
| %1 / %2 | Percentage of total tokens in each data block |
| LL | Log-likelihood G² statistic — higher means a more significant difference |
| %DIFF | Percentage-point difference between the two data blocks |
| Bayes | Bayes factor (BIC) |
| ELL | Effect size for log-likelihood |
| RRisk | Relative risk ratio |
| LogRatio | Log of relative frequencies — positive values skew toward the reference block |
| OddsRatio | Odds ratio between data blocks |
| Significance | **** p < 0.0001, *** p < 0.001, ** p < 0.01, * p < 0.05 |
Use the Head / Tail Rows (N) control to show the first and last N rows of the sorted table. Sorting always applies to the full dataset before trimming.
The full table can be downloaded as a CSV file. For further reading on keyword analysis methodology, see the Lancaster corpus linguistics resource.
The tab keeps its current Analysis in the backend so it can reload its lifecycle and Result. Clear Results removes that Analysis and resets the tab, including after failure or cancellation. Re-run clears the current Analysis and submits its replacement; if clearing fails, the replacement is not started.
| Symptom | Likely cause | What to try |
|---|---|---|
| Results unchanged after removing stop words | Stop words not applied | Click Apply Stop Words after editing the list |
| Word cloud dominated by common words | No stop words applied | Click Fill Default then Apply Stop Words |
| Juxtorpus or Keyword Analysis table are missing | Only one data block selected | Select a second data block to enable comparison mode |
| Keyword Analysis table shows no significant words | Corpora are very similar or one is very small | Try a larger or more distinct pair of data blocks |
| A workspace block I selected isn't showing in the panel | Token Frequency caps the panel to the 2 most-recent selections | Deselect a newer block to make room, or run the comparison on the visible pair |
| Right-clicked stop word is hard to find | List was already long when the word was added | New words are inserted at the top — scroll to the start, or click Sort to alphabetise |
| Analyze button is disabled | No data block, text column, or tokenizer model selected | Select a data block, text column, and tokenizer model |
| Setting | Default | Notes |
|---|---|---|
| Data blocks | None | Up to 2; comparison mode activates when 2 are selected. If more than 2 are selected workspace-wide, only the 2 most recent show in the panel. |
| Tokenizer model | Saved Data Block preference or none | Required for each selected block; the submitted Analysis freezes the exact mapping |
| Corpus role switches | First selected block is Study Corpus | Changes O1/O2 assignment in the statistics table |
| Stop words | Empty | Click Fill Default for language-matched default stop words |
| Cloud display limit | 50 | Range 10–100; mirrors to list limit |
| List display limit | 50 | Range 10 – vocabulary size; values > 100 diverge from cloud |
- Select a data block and click Analyze with the default settings.
- Click Fill Default to apply language-matched default stop words, then Apply Stop Words and compare the top tokens.
- Right-click one of the remaining high-frequency words in the cloud to add it as a custom stop word. Confirm it appears at the start of the stop-word list.
- Select a second data block. Use the card-level Study Corpus / Reference Corpus switches to set which block is the baseline, then re-run.
- Switch to List view and use Filter tokens with a wildcard pattern (e.g.
*ing) to find all gerund-form tokens. - In List view, scroll one frequency list and observe that the other list scrolls in sync.
- Sort the statistics table by LogRatio to find the words most distinctively associated with each data block.
- Left-click one of the top distinctive words to jump to Concordance and inspect it in context.