RoboTech

A Comparative Study on Text mining Techniques

ABSTRACT:

This research is intended to give detailed overview of basic concept of two text mining techniques namely information retrieval and information extraction.

This paper has provided a comparison table of both these techniques on the basis of characteristic and their relationship to each other.

We have also underlined many interesting research challenges which will benefit in managing the extracted valuable information.

Later this research has also been motivated to highlight the main role of information extraction in terms of retrieval context in future to be played.

KEYWORD:

Information retrieval, Information extraction, Text mining

CITE:

Ahmad, P.H., & Dang, S. (2014). A Comparative Study on Text mining Techniques.

PUBLINK:

http://www.gjar.org/articles/A-Comparative-Study-on-Text-mining-Techniques

DOCLINK:

https://www.ijsr.net/archive/v3i12/U1VCMTQ4NjU=.pdf

RGTLINK:

https://www.researchgate.net/publication/270704468_A_Comparative_Study_on_Text_mining_Techniques

While many lexica annotated with words polarity are available for sentiment analysis, very few tackle the harder task of emotion analysis and are usually quite limited in coverage.

In this paper, we present a novel approach for extracting - in a totally automated way - a high-coverage and high-precision lexicon of roughly 37 thousand terms annotated with emotion scores, called DepecheMood.

Our approach exploits in an original way 'crowd-sourced' affective annotation implicitly provided by readers of news articles from rappler.com.

By providing new state-of-the-art performances in unsupervised settings for regression and classification tasks, even using a na\"{\i}ve approach, our experiments show the beneficial impact of harvesting social media data for affective lexicon building.

approach, affective lexicon building, emotion scores, unsupervised settings, sentiment analysis

https://aclanthology.org/P14-2070/

https://www.semanticscholar.org/paper/Depeche-Mood%3A-a-Lexicon-for-Emotion-Analysis-from-Staiano-Guerini/492cefde909957fd8b0f77f75e51b066567671b6

https://aclanthology.org/P14-2070.pdf

https://app.razzi.my/findgref?gid=1_P_NOkeQQ0T5lUUS6qwW-qRUb9MFp1FI

Even though considerable attention has been given to the polarity of words (positive and negative) and the creation of large polarity lexicons, research in emotion analysis has had to rely on limited and small emotion lexicons.

In this paper, we show how the combined strength and wisdom of the crowds can be used to generate a large, high-quality, word–emotion and word–polarity association lexicon quickly and inexpensively.

We enumerate the challenges in emotion annotation in a crowdsourcing scenario and propose solutions to address them.

Most notably, in addition to questions about emotions associated with terms, we show how the inclusion of a word choice question can discourage malicious data entry, help to identify instances where the annotator may not be familiar with the target term (allowing us to reject such annotations), and help to obtain annotations at sense level (rather than at word level).

We conducted experiments on how to formulate the emotion-annotation questions, and show that asking if a term is associated with an emotion leads to markedly higher interannotator agreement than that obtained by asking if a term evokes an emotion.

Emotions, affect, polarity, semantic orientation, crowdsourcing, Mechanical Turk, emotion lexicon, polarity lexicon, word–emotion associations, sentiment analysis.

https://onlinelibrary.wiley.com/doi/10.1111/j.1467-8640.2012.00460.x

https://arxiv.org/pdf/1308.6297.pdf

https://www.semanticscholar.org/paper/CROWDSOURCING-A-WORD%E2%80%93EMOTION-ASSOCIATION-LEXICON-Mohammad-Turney/54227c063bb04489caffd65ff9fc6218788ddb25

https://www.researchgate.net/publication/256199465_Crowdsourcing_a_Word-Emotion_Association_Lexicon

https://app.razzi.my/findgref?gid=15LQdmmfNMwhK7jb-4zkPkKOjZ0apm6fz

The Google Ngram Viewer or Google Books Ngram Viewer is an online search engine that charts the frequencies of any set of search strings using a yearly count of n-grams found in sources printed between 1500 and 2019 in Google's text corpora in English, Chinese (simplified), French, German, Hebrew, Italian, Russian, or Spanish. There are also some specialized English corpora, such as American English, British English, and English Fiction.

The program can search for a word or a phrase, including misspellings or gibberish. The n-grams are matched with the text within the selected corpus, optionally using case-sensitive spelling (which compares the exact use of uppercase letters), and, if found in 40 or more books, are then displayed as a graph.

The Google Ngram Viewer supports searches for parts of speech and wildcards. It is routinely used in research.

The program was developed by Jon Orwant and Will Brockman and released in mid-December 2010. It was inspired by a prototype called "Bookworm" created by Jean-Baptiste Michel and Erez Aiden from Harvard's Cultural Observatory and Yuan Shen from MIT and Steven Pinker.

The Ngram Viewer was initially based on the 2009 edition of the Google Books Ngram Corpus. As of July 2020, the program supports 2009, 2012, and 2019 corpora.

Operation and restrictions

Commas delimit user-entered search-terms, indicating each separate word or phrase to find. The Ngram Viewer returns a plotted line chart within seconds of the user pressing the Enter key or the "Search" button on the screen.

As an adjustment for more books having been published during some years, the data are normalized, as a relative level, by the number of books published in each year.

Due to limitations on the size of the Ngram database, only matches found in at least 40 books are indexed in the database; otherwise the database could not have stored all possible combinations.

Typically, search terms cannot end with punctuation, although a separate full stop (a period) can be searched. Also, an ending question mark (as in "Why?") will cause a second search for the question mark separately.

Omitting the periods in abbreviations will allow a form of matching, such as using "R M S" to search for "R.M.S." versus "RMS".

Corpora

The corpora used for the search are composed of total_counts, 1-grams, 2-grams, 3-grams, 4-grams, and 5-grams files for each language. The file format of each of the files is tab-separated data. Each line has the following format:

total_counts file

year TAB match_count TAB page_count TAB volume_count NEWLINE

Version 1 ngram file (generated in July 2009)

ngram TAB year TAB match_count TAB page_count TAB volume_count NEWLINE

Version 2 ngram file (generated in July 2012)

ngram TAB year TAB match_count TAB volume_count NEWLINE

The Google Ngram Viewer uses match_count to plot the graph.

As an example, a word "Wikipedia" from the Version 2 file of the English 1-grams is stored as follows:

ngram year match_count volume_count

Wikipedia 1904 1 1

Wikipedia 1912 11 1

Wikipedia 1924 1 1

Wikipedia 1925 11 1

Wikipedia 1929 11 1

Wikipedia 1943 11 1

Wikipedia 1946 11 1

Wikipedia 1947 11 1

Wikipedia 1949 11 1

Wikipedia 1951 11 1

Wikipedia 1953 22 2

Wikipedia 1955 11 1

Wikipedia 1958 1 1

Wikipedia 1961 22 2

Wikipedia 1964 22 2

Wikipedia 1965 11 1

Wikipedia 1966 15 2

Wikipedia 1969 33 3

Wikipedia 1970 129 4

Wikipedia 1971 44 4

Wikipedia 1972 22 2

Wikipedia 1973 1 1

Wikipedia 1974 2 1

Wikipedia 1975 33 3

Wikipedia 1976 11 1

Wikipedia 1977 13 3

Wikipedia 1978 11 1

Wikipedia 1979 112 12

Wikipedia 1980 13 4

Wikipedia 1982 11 1

Wikipedia 1983 3 2

Wikipedia 1984 48 3

Wikipedia 1985 37 3

Wikipedia 1986 6 4

Wikipedia 1987 13 2

Wikipedia 1988 14 3

Wikipedia 1990 12 2

Wikipedia 1991 8 5

Wikipedia 1992 1 1

Wikipedia 1993 1 1

Wikipedia 1994 23 3

Wikipedia 1995 4 1

Wikipedia 1996 23 3

Wikipedia 1997 6 1

Wikipedia 1998 32 10

Wikipedia 1999 39 11

Wikipedia 2000 43 12

Wikipedia 2001 59 14

Wikipedia 2002 105 19

Wikipedia 2003 149 53

Wikipedia 2004 803 285

Wikipedia 2005 2964 911

Wikipedia 2006 9818 2655

Wikipedia 2007 20017 5400

Wikipedia 2008 33722 6825

The graph plotted by the Google Ngram Viewer using the above data is here:

https://books.google.com/ngrams/graph?content=Wikipedia&year_start=1900&year_end=2020&corpus=15&smoothing=0

Criticism

The data set has been criticized for its reliance upon inaccurate OCR, an overabundance of scientific literature, and for including large numbers of incorrectly dated and categorized texts. Because of these errors, and because it is uncontrolled for bias (such as the increasing amount of scientific literature, which causes other terms to appear to decline in popularity), it is risky to use this corpus to study language or test theories. Since the data set does not include metadata, it may not reflect general linguistic or cultural change and can only hint at such an effect.

Guidelines for doing research with data from Google Ngram have been proposed that address many of the issues discussed above.

OCR issues

Optical character recognition, or OCR, is not always reliable, and some characters may not be scanned correctly. In particular, systemic errors like the confusion of "s" and "f" in pre-19th century texts (due to the use of the long s which was similar in appearance to "f") can cause systemic bias. Although Google Ngram Viewer claims that the results are reliable from 1800 onwards, poor OCR and insufficient data mean that frequencies given for languages such as Chinese may only be accurate from 1970 onward, with earlier parts of the corpus showing no results at all for common terms, and data for some years containing more than 50% noise.

REF:

https://en.wikipedia.org/wiki/Google_Ngram_Viewer

https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-to-you.html

ABSTRACT:

With the rise of Web 2.0 paradigm new trends in information retrieval (IR) and information extraction (IE) can be observed. Significance of IR/IE as fundamental method of acquiring new and up-to-date information is crucial for efficient decision making.

Social aspects of modern information retrieval are gaining on its importance over technical aspects. The main reason for this trend is that IR and IE services are becoming more and more widely available to end users that are not information professionals but regular users. Also new methods that rely primarily on user interaction and communication show similar success in IR and IE tasks.

Web 2.0 has overall positive impact on IR and IE as it is based on a more structured data platform than the earlier web. Moreover, new tools are being developed for online IE services that make IE more accessible even to users without technical knowledge and background.

The goal of this paper is to review these trends and put them into context of what improvements and potential IR and IE have to offer to knowledge engineers, information workers, but also typical Internet users.

KEYWORD:

Information extraction, Information retrieval, Web 2.0, Social bookmarking, Mashups, Folksonomies.

PUBLINK:

https://www.bib.irb.hr/480921?rad=480921

DOCLINK:

https://www.naun.org/main/NAUN/computers/19-556.pdf

SEMLINK:

https://www.semanticscholar.org/paper/Information-Retrieval-and-Information-Extraction-in-Vlahovic/d91a84ac6f8c752b610cd9a6678c2e5057668c6e

Natural Language Processing: part 1 of lecture notes

Lecture Synopsis

Aims

This course aims to introduce the fundamental techniques of natural language processing, to develop an understanding of the limits of those techniques and of current research issues, and evaluate some current and potential applications.

• Introduction. Brief history of NLP research, current applications, generic NLP system architecture, knowledgebased versus probabilistic approaches.

• Finite state techniques. Inflectional and derivational morphology, finite-state automata in NLP, finite-state transducers.

• Prediction and part-of-speech tagging. Corpora, simple N-grams, word prediction, stochastic tagging, evaluating system performance.

• Parsing and generation I. Generative grammar, context-free grammars, parsing and generation with contextfree grammars, weights and probabilities.

• Parsing and generation II. Constraint-based grammar, unification, simple compositional semantics.

• Lexical semantics. Semantic relations, WordNet, word senses, word sense disambiguation.

• Discourse. Anaphora resolution, discourse relations.

• Applications. Machine translation, email response, spoken dialogue systems.

https://www.cl.cam.ac.uk/teaching/2002/NatLangProc/nlp1-4.pdf

This paper presents a simple unsupervised learning algorithm for classifying reviews as recommended (thumbs up) or not recommended (thumbs down).

The classification of a review is predicted by the average semantic orientation of the phrases in the review that contain adjectives or adverbs.

A phrase has a positive semantic orientation when it has good associations (e.g., "subtle nuances") and a negative semantic orientation when it has bad associations (e.g., "very cavalier").

In this paper, the semantic orientation of a phrase is calculated as the mutual information between the given phrase and the word "excellent" minus the mutual information between the given phrase and the word "poor".

A review is classified as recommended if the average semantic orientation of its phrases is positive.

The algorithm achieves an average accuracy of 74% when evaluated on 410 reviews from Epinions, sampled from four different domains (reviews of automobiles, banks, movies, and travel destinations).

The accuracy ranges from 84% for automobile reviews to 66% for movie reviews.

Turney, P.D. (2002). Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. ACL.

https://aclanthology.org/P02-1053/

https://aclanthology.org/P02-1053.pdf

https://www.semanticscholar.org/paper/Thumbs-Up-or-Thumbs-Down-Semantic-Orientation-to-of-Turney/9e7c7853a16a378cc24a082153b282257a9675b7

https://www.researchgate.net/publication/2537987_Thumbs_Up_or_Thumbs_Down_Semantic_Orientation_Applied_to

https://app.razzi.my/findgref?gid=1MSx8ggcFJ9yOAVIw7WAXkIWSUlsIMapm

A Comparative Study on Text mining Techniques

Depeche Mood: a Lexicon for Emotion Analysis from Crowd Annotated News

Crowdsourcing a Word-Emotion Association Lexicon

Google Ngram Viewer

Information retrieval and Information extraction in Web 2.0 environment

Natural Language Processing: part 1 of lecture notes

Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews