Skip to main content

Text Analytics: Creating Word Cloud

In the word cloud, words that appear most frequently in documents are plotted bolder and bigger than less frequently occurring words. The size and maybe the color of the word is determined by the frequency of the word in the documents.  R package 'wordcloud' provides functions that generate such word cloud.



A collection of documents is converted into a vector of characters. 

sentence <- c("sentence1", "sentence2", "sentenceN")

We will use gsub() function to remove whitespace, number, and control characters from these sentences.


sentence <- gsub('[[:punct:]]', ' ', sentence)

sentence <- gsub('[[:cntrl:]]', ' ', data)

sentence <- gsub('\\d+', ' ', sentence)



stopwords() function from text mining package 'tm' provides a list of stop words in English. We will use this list to remove stopwords from the sentences later.


stopwords <-  stopwords("SMART")

Let us convert our sentence into a corpus.

corpus <- Corpus(VectorSource(sentence))


tm package provides several predefined transformation that can be applied to text using tm_map() function. getTransformtions() function will list those transformation function names. tm package provides content_transformer() function to create our own text processing function.  For example, to convert the text to lower case, we can use: 

corpus <- tm_map(corpus, content_transformer(tolower))

and to remove whitespace and stopwords, we can use the predefined function removePunctuation and removeWords

corpus <- tm_map(corpus, removePunctuation)

corpus <- tm_map(corpus, removeWords, stopwords)


Next, we create a term-document matrix using function TermDocumentMatrix(). This function needs a list of options, which we set by creating a list as shown below. Here, minWordLength will discard all those words whose lenght this less than 4 characters. 

control <- list(weighting=weightTf, minWordLength=4, removeNumbers=TRUE)
tdm <- TermDocumentMatrix(corpus, control=control)

To reduce the size of tdm object for generating the word cloud, we can remove all those words that are found in less than 5% of the documents or sentences. 

tdm <- removeSparseTerms(tdm, 0.95)

If you display the tdm object as a matrix, this will print document id as column name and word as row name with the occurrence of the word in the particular document as the value.  

Now, we want to sum the count of each word in each document to get the frequency of that words. 



word_count <- sort(rowSums(as.matrix(tdm)), decreasing=TRUE)

word_name_frequency <- data.frame(word=names(word_count), freq= word_count)

Create a palette using the brewer function from RColorBrewer package. Several color options are available to create the palette. For example, we are using the OrRd palette to generate 7 different colors.  Function display.brewer.pal(7,"OrRd") will show all the colors of the palette used in drawing word cloud. Remove any color from the palette that is same as the background color.

palette <- brewer.pal(7, "OrRd")
palette <- palette[-(1)]

Open a png file and plot the word cloud. Function wordcloud takes multiple arguments, but I am using the default option for most of them. Wordcloud provides arguments to control the number of words, height or width of the words, the minimum frequency of the words required to plot.

png(filename='wordcloud.png')


wordcloud(word_name_frequency $word, 

                             word_name_frequency $freq, 

                             random.order=FALSE, 
                             colors=palette)
dev.off()



Comments

Post a Comment

Popular posts from this blog

Decision Tree

Decision tree is a multi-class classification tool allowing a data point to be classified into one of many (two or more) classes available.  A decision tree divides the sample space into a rectilinear region. This will be more clear with an example. Let us say we have this auto-insurance claim related data as shown in the following table. We want to predict what type of customer profile may more likely lead to claim payout.  The decision tree model may first divide the sample space based on age. So, now we have two regions divided based on the age. Next, one of those regions will further sub-divided based Marital_status, and then that newly divided sub-regision may further get divide based on Num_of_vehicle_owned.  A decision tree is made up of a root node followed by intermediate node and leaf node.  Each leaf node represents one of the class into which data points have been classified to. An intermediate node represents the decision rule based...
Post a Comment
Read more

Recommender System using Collaborative filtering

Recommender system using collaborative filtering approach uses the past users' behavior to predict what items the current user would like. We create a UxM matrix where U is the number of users and M is the number of different items or products. Uij is the rating expressed by the user-i for product-j. In the real world, not every user expresses an opinion about every product. For example, let us say there are five users including Bob has expressed their opinion about four movies as shown below Table 1: movie1 movie2 movie3 movie4 user1 1 3 3 5 user2 2 4 5 user3 3 2 2 user4 1 3 4 Bob 3 2 5 ?  Our goal is to predict what movies to recommend to Bob, or put it another way should we recommend movie4 to Bob, knowing the rating for four movies from other users including Bob. Traditionally, we could do item to item comparison, which means if the user has liked item1 in the past then that user may like other items similar to item1. Another way to recommend...
Post a Comment
Read more

Sentimental Analysis Using Scikit-Learn and Neural Network

Using Scikit-Learn and NLTK for Sentimental Analysis Sentimental analysis is a way of categorizing text into subgroup based on the opinion or sentiments expressed in the text. For example, we would like to categorize review or comments of people about a movie to determine how many like the movie and how many don't. In a supervised sentimental analysis, we have some training data which is already categorized or sub-grouped into different categories, for example, into 'positive' or 'negative' sentiments. We used these training data to train our model to learn what makes a text to be part of a specific group. By text I mean a sentence or a paragraph. Using this labeled sentences, we are going to build a model. So, let us say we have following training text: training_positive = list() training_positive[0] =  "bromwell high is a nice cartoon comedy perfect for family" training_positive[1] =  " homelessness or houselessness as george carlin s...
Post a Comment
Read more