Source and keyword extraction code with entropy difference between internal and external mode with Visual Basic .NET

Full description of the file

Download source and keyword extraction code with entropy difference between internal and external mode with Visual Basic .NET

We are trying to propose a new criterion for evaluating and ranking the relevance of words in a text. This method uses the Shannon entropy difference between the inner and outer state, which refers to the fact that related words significantly reflect the author's intention to write, for example, their occurrence is modulated by the author's purpose, while That irrelevant words are randomly distributed in the text. . Using the origin of the species by Charles Darwin as a representative text sample, our detector performance is demonstrated and compared with previous suggestions. Since the reference text of the "profession" is all the writings, books, articles, etc. of an author, there is no need for his collected works. Our approach is especially suitable for individual documents where no prior information is available.

theme

One of the most important differences between texts written by humans and typing monkeys is the existence of general meaningful themes in human written texts. Keyword / Extraction and ranking of relevant words is the starting point of vital tasks such as subject identification and tracking in written texts. They are widely used in information extraction, selection and retrieval.

Internal mode and external mode in word type events in the text.

Here we give a brief introduction to the principle of the algorithm, it can help you better understand and use the dll and software. The idea of ​​internal-external mode is based on the general idea that very important words tend to be modified by the author's intention, while common words are essentially evenly distributed throughout the text. Thus the intrinsic state of statistical properties indicates the appearance of a related word in a subject, for example, the statistical properties of clustering in each subject. At the same time, the external state shows the statistical characteristics of the disappearance of a word clustering throughout a written text and determines the relationship between the occurrence of clustering of words in a subject and the author's writing style. As shown in the figure. 2. The distance between two words that are consecutive repetitions is defined as di = ti + 1 - ti. Ti is the place of the word in the text. If _di

Highlights

• We propose a new criterion for evaluating and ranking the relevance of words in a text.
• This metric uses the Shannon entropy difference between the inner and outer states.
• We believe this is a new result in keyword extraction and ranking.
• Our approach is especially suitable for individual documents where no prior information is available.