Text Skip-Grams Generator

The Text Skip-Grams Generator extracts skip-grams from a passage of text. A skip-gram is an ordered tuple of tokens drawn from a sequence where consecutive selected tokens may be separated by up to k intervening tokens, with k being the maximum skip distance. When k is 0, the output matches standard contiguous n-grams. Increasing k captures co-occurrence patterns across wider context windows, so two words that sit a token or two apart are counted as a pair even though a contiguous n-gram would never join them.

Tokenization splits the input on whitespace boundaries. Punctuation marks attached to tokens are stripped by default, so a word at the end of a sentence and the same word mid-sentence become one token; a setting retains punctuation as part of each token for analyses where it carries signal. Tokens are normalized to lowercase by default, which folds a sentence-initial capital into the same token as its lowercase form and keeps frequency counts from splitting on capitalization. Case handling can be changed in Settings when capitalization is meaningful to the analysis.

Two parameters define the extraction, both held in Settings. The n-gram size n sets how many tokens appear in each tuple: a size of 2 produces pairs, 3 produces triples, and so on. The maximum skip distance k sets the largest gap allowed between any two consecutively selected tokens in the tuple. For example, with n=2 and k=1 applied to the tokens [A, B, C, D], the resulting skip-grams are (A, B), (A, C), (B, C), (B, D), and (C, D): every pair whose members are adjacent or separated by exactly one token.

Frequency counting reports how many times each unique skip-gram appears, which turns the raw tuple list into a co-occurrence distribution. Results can be sorted by first occurrence, by frequency, or alphabetically. The output format is one tuple per line, comma separated, tab separated, or a JSON array; the JSON form is the one that feeds a script or preprocessing pipeline without further parsing. Longer passages produce more skip-grams and more meaningful frequency distributions, since a pair seen once in a short sentence says little about association.

The total number of skip-grams grows with text length, n-gram size, and skip distance. For a sequence of T tokens with n-gram size n and skip distance k, each starting position can generate up to C(k+1, n-1) tuples (the number of ways to choose gap sizes for each step), bounded by the remaining tokens in the sequence. Starting positions near the end of the sequence yield fewer tuples because too few tokens remain to fill the gaps.

Take the input "The cat sat on the mat." with n=2 and k=1. Whitespace splitting yields six tokens; stripping the trailing period and lowercasing gives [the, cat, sat, on, the, mat]. Each of the first four positions pairs with the next token and with the one after it: (the, cat), (the, sat), (cat, sat), (cat, on), (sat, on), (sat, the), (on, the), (on, mat). The fifth position has one token left, so it yields (the, mat) alone, and the final token starts nothing. That is 9 skip-grams against 5 contiguous bigrams at k=0, and it matches the formula: C(2, 1) = 2 tuples for each of the first four positions, plus 1 bounded tuple. All 9 are distinct, so every frequency count is 1; a longer text is needed before any pair repeats.

Skip-grams extend traditional n-gram analysis by capturing relationships between tokens that are not immediately adjacent. This flexibility reveals co-occurrence patterns that contiguous n-grams miss, making skip-grams valuable across multiple disciplines.

• Natural Language Processing: Build skip-gram feature vectors for text classification, sentiment analysis, or topic modeling. Varying the skip distance highlights different levels of syntactic and semantic association between words in a corpus.
• Corpus Linguistics: Investigate collocational patterns and word associations in large text collections. Comparing frequency distributions at different skip distances reveals which word pairs co-occur primarily in adjacent positions versus across clause boundaries.
• Search Engine Optimization: Analyze keyword proximity and co-occurrence on landing pages or competitor content. High-frequency skip-grams with moderate skip distances indicate topically related terms that search engines may use for semantic indexing.
• Plagiarism Detection: Compare skip-gram fingerprints between documents to find paraphrased passages. Rearranged sentences that defeat contiguous n-gram matching often share the same skip-grams because the underlying vocabulary and proximity are preserved.
• Machine Learning Feature Engineering: Generate skip-gram token pairs as input features for word embedding models such as Word2Vec. Adjusting the skip distance corresponds to the context window size, and exporting results in JSON format feeds directly into preprocessing pipelines.
• Authorship Attribution: Profile writing style by examining characteristic skip-gram frequency distributions. Different authors tend to favor different function-word pairings at specific distances, and these patterns persist even when topic vocabulary changes.
• Education: Introduce students to n-gram and skip-gram concepts with immediate visual feedback. Changing the n-gram size and skip distance on sample sentences demonstrates how context window parameters affect the set of extracted features.