The MEDLINE N-gram Set - 2015, by Split, Group, Filter, and Combine Algorithm
The MEDLINE n-gram set (generated by split, group, filter, and combine algorithm) is listed as bellows. For each MEDLINE record, title and abstract are used as the source of n-grams. They are combined, tokenized into sentences, and then tokenized into tokens (words use space as word boundary). Finally, n-grams are generated by filtering out terms with more than 50 characters or the total word count is less than 30. The specifications of generating these n-grams are listed as follows:
- MEDLINE: 2015 - TI and AB (from MEDLINE Baseline Repository - MBR, PmidTiAbS15nXXXX.txt: 1 ~ 779)
- Method: Split, Combine, Filter Algorithm
- Max. Character Size: 50
- Min. word count: 30
- Min. document count: 1
- Total document count: 23,343,329
- Total sentence count: 134,834,507
- Total token count: 2,786,085,158
- N-gram files
- File format - 3 fields:
Document count Word count N-gram - Sorted by document count, word count, then alphabetic order of n-grams. N-gram set is not sorted. It can be sorted by nGramUtil package.
Download:
N-grams File Zip Size Actual Size No. of n-grams Unigrams 1-gram.2015.tgz 5.7MB 14MB 843,206 Bigrams 2-gram.2015.tgz 35MB 103MB 4,845,965 Trigrams 3-gram.2015.tgz 52MB 171MB 6,702,194 Four-grams 4-gram.2015.tgz 35MB 123MB 4,082,612 Five-grams 5-gram.2015.tgz 17MB 59MB 1,674,715 N-gram Set nGramSet.2015.30.tgz 143MB 467MB 18,148,692 Distilled N-gram Set distilledNGram.2015.tgz 55MB 176MB 6,793,561 - File format - 3 fields:
