Is there any tool or ut开发者_Python百科ility or perl/python script that can find longest repeated substrings in a large text file and print those patterns and the number of times each pattern occurs?
http://en.wikipedia.org/wiki/Longest_repeated_substring_problem:
The longest repeated substring problem is finding the longest substring of a string that occurs at least twice. This problem can be solved in linear time and space by building a suffix tree for the string, and finding the deepest internal node in the tree
Suffix trees in python (a little dated, though ..): http://hkn.eecs.berkeley.edu/~dyoo/python/suffix_trees/
Javascript implementation with further explaination: http://www.allisons.org/ll/AlgDS/Tree/Suffix/
精彩评论