Natural Language Processing

Day 3: Cleaning text, stopwords, predictions

Cleaning the text means to remove special characters, numbers and multiple empty spaces from the text.

Add a new script frame, click copy text button and paste the contents into the frame.

There is no output from this script.

The text that appears below shows numbers, special characters and extra spaces. The items to pay attention to are: review 73, 78 and 79. Originally these had extra spaces, numbers and special characters.

Add a new script frame, click copy text button and paste the contents into the frame.

The output shows the dataset before it is cleaned.

Add a new script frame, click copy text button and paste the contents into the frame.

The output shows that numbers, special characters and extra space have been removed.

The stopwords are a list of words that are very very common but don’t provide useful information for most text analysis procedures. While it is helpful to understand the structure of sentence, it does not help you understand the semantics of the sentences themselves.¹

Here’s the code to generate a list of most commonly used stop words.

Add a new script frame, click copy text button and paste the contents into the frame.

Your output should look like:

The following script shows you how this function works. I made a new Python project and put the file into a dictionary format. # Import stopwords with nltk. from nltk.corpus import stopwords import pandas as pd stop = stopwords.words('english') comments =[('Executive summary was incomplete. lacking financial information.','negative'), ('Marketing plan was excellent in that it addressed product, price, place and promotion','positive'), ('They made no mention of the management structure of the company.','negative'), ('The SWOT analysis was excellent.','positive'), ('The mission statement was dreadfull.','negative'), ('There was no section to explain the reason for seeking a business loan.','negative'), ('The plan sucessfully included a section on three months worth of operating income.','positive'), ('There was no mention on how to concentrate the company marketing efforts.','negative'), ('A plan on how to expand the business was well done.','positive'), ('There was no mention on how social media could be used.','negative'), ('It was not clear what type of business was.','negative'), ('The plan authors missed the fact that the business should be designated as an S corporation.','negative'), ('The organizational chart was well done.','positive'), ('Since it was an existing company, a history of the company should have been included.','negative'), ('There was no backround on key personnel.','negative'), ('The report did not contain the location of the virtual busines.','negative'), ('The section on the channels of distrubution fell short.','negative'), ('The report lacked a section on how the products would be stored.','negative'), ('The delivery options: UPS and FedEx were explained well.','positive'), ('A list of job responsibilities for each person in the organization needed to accomplish goals was excellent.','positive'), ('Explanation of the pricing strategies was pathetic.','negative'), ('The plan did not address the fact that the Human Resources Department was responsible for the news letter.','negative'), ('There was no mention of the employee month award.','negative'), ('The business plan lacked instructions on how to finish assignments on time.','negative'), ('The plan did a good job of looking at its competitors.','positive'), ('Identifying strengths and threats in short and long term planning were inadequate.','negative'), ('There was no mention on how to interest potential customers and clients in their product.','negative'), ('Mission statement ws horrible.','negative'), ('Mission statment did not include clear objectives.','negative'), ('There was not mention on why the company needed a business loan','negative'), ('There was an excellent section on how they were planning on expanding the business.','positive'), ('No profit goals were set.','negative'), ('Breakeven analysis was well done.','positive'), ('Market share section was lacking detail.','negative'), ('The business plan was missing a good description of the company.','negative'), ('Organizational chart was excellent.','positive'), ('Decrription of the market place was very good.','positive'), ('Plan lacked details about the number of employees and their job titles.','negative'), ('The business plan had an excellent section on internal operations.','positive'), ('The plan included a compehensive list of additional assets need to run the company.','positive'), ('Channels of distribution was not very weel explained.','negative'), ('Organization chart was poorly done.','negative'), ('The plan had an excellent section on how to keep all employees on track.','positive'), ('Plan lacked consequences for employees who did not perform their jobs.','negative'), ('The market analysis section was incomplete.','negative'), ('The plan did an excellent job of identifying strengths and threats.','positive'), ('The products section seemed unclear to me.','negative'), ('The placement section on marketing seemed vague.','negative'), ('The pricing element in the marketing mix did not address profit margins.','negative'), ('No discounts were included in the pricing element of he marketing mix.','negative'), ('The plan did not address calculation of profit for each item.','negative'), ('It would appear that the only mechanisim in place for promoting the company was their website.','negative'), ('The target market was articulated well.','positive'), ('Market segments were clearly delineated.','positive'), ('Demographics were poorly defined.','negative'), ('There was an excellent section on psycho-graphics.','positive'), ('Market segments were not fully explained.','negative'), ('The breakeven analysis was well done by using a spreadsheet program.','positive'), ('The plan confused fixed and variable expenses.','negative'), ('The cost of goods sold section was way off.','negative'), ('The plan beginning balance sheet was inaccurate.','negative'), ('The trial balance did not balance.','negative'), ('The business loan was correctly listed under liabilities on the balance sheet.','positive'), ('The business loan amount contained ten months of operating expenses instead of three.','negative'), ('The amount of rent expense was way too high.','negative'), ('The loan was for ten years at ten percent. That is incorrect.','negative'), ('The authors of the plan should have used bizstats.','negative'), ('The utilities were corrctly calculated based on the number of employees in the company.','positive'), ('The interest rate for the business loan was incorrect.','negative'), ('They did a good job on the accounting section of the business plan.','positive'), ('The amount of the business contract was well done.','positive'), ('There was no sample payroll included to show how the businees contract was calculated.','negative'), ('Monthly expenses were clearly calculated.','positive'), ('The salary expense of was incorectly calculated.','negative'), ('The plan lacked any mention of payroll taxes.','negative'), ('There was no projected income statement.','negative'), ('Cashflow analysis was excellent.','positive'), ('The plan lacked an adequate section on cash flow.','negative'), ('The strengths section of the SWOT analysis was well done.','positive'), ('The plan was missing supporting documentation.','negative')] test = pd.DataFrame(comments) test.columns =['Impression','class'] # Exclude stopwords with Python's list comprehension and pandas.DataFrame.apply. test['Impression_without_stopwords'] = test['Impression'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)])) print(test)

Start a new Python project then click copy text button and paste the contents into the frame. Save it and call it "stopWords" and run it.

Your output should look like:

Python gives us with a number of data structures such as:

• lists []
• tuples ()
• sets {}
• dictionaries { :}
  
  

  They all are used to store and organize the data in an efficient manner.

  Here are the differences between them:

  geeksforgeeks web site
  Adding stop words to list

  Let's modify our list with tuples. Make the first sentence read "The Executive summary was incomplete. It was lacking financial information."

  Modify line 5, the tuple about the mission statement. Make it read "In my opinion, The mission statement was dreadfull."

  Save it and run it. You should see that the stop words were removed.

  If you want to see more than the first lines of the print(test), change the line to the following: print (test.head(20))
  

  Day 4: Sentiment Analysis with natural language tool kit (NLTK)

  Sentiment Analysis is a technique to extract emotions from textual data.

  We are trying to capture the sentiment of the judges as they evaluate our students' business plans.

  Studies shows that VADER performs as good as individual human raters at matching ground truth. Further inspecting the F1 scores (classification accuracy), we see that VADER (0.96) outperforms individual human raters (0.84) at correctly labelling the sentiment of tweets into positive, neutral, or negative classes.³

  Now go back to your initial Python project, "businessPlan" for this section.

  Add the following script to begin our sentiment analysis.

  import nltk from nltk.sentiment.vader import SentimentIntensityAnalyzer nltk.download('vader_lexicon') sia = SentimentIntensityAnalyzer()
  

  Start a new frame then click copy text button and paste the contents into the frame.

  vader stands for "Valence Aware Dictionary and Sentiment Analysis." It can be used on unsupervised data. Our current data set is labled with the rating 0 or 1 given by each judge for each comment.

  the variable "sia" is created by making and instance of the SentimentIntensity Analyzer

  There is no output from this script.

  from sklearn.feature_extraction.text import TfidfVectorizer #converting rfeviews to text via TFDIF vectorization vectorizer = TfidfVectorizer (max_features = 2000, min_df=5, max_df=0.7, stop_words=stopwords.words('english')) X= vectorizer.fit_transform(X_sentences).toarray()
  

  Start a new frame then click copy text button and paste the contents into the frame.

  Here we are converting the text to numbers

  The max features specifies that a maximum of 2000 most occurring words should be use to create the feature dictionary.

  The min_df attribute specifies to only include words that appear a minimum of five times.

  max_df defines not to include words that occur in more than 70 percent of the document.

  There is no output from this script.

  #dividing the data into training and test sets from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, random_state=42)
  

  Start a new frame then click copy text button and paste the contents into the frame.

  Here we are training the dataset on 80% of the data. Twenty percent will make up the X_test file.

  There is no output from this script.

  #training Random Forest Classifier on the training data from sklearn.ensemble import RandomForestClassifier clf = RandomForestClassifier(n_estimators = 250, random_state =0) clf.fit(X_train, y_train)
  

  Start a new frame then click copy text button and paste the contents into the frame.

  Our model is a supervised text classifier model since the dataset contains the judges reviews and their rating 0 for a poor business plan and 1 if they thought it was a good plan.

  Using the Random Forest classfier, the training data will learn the relationship between the text review and the rating 0 or 1

  There is no output from this script.

  #making predictions on the test set y_pred = clf.predict(X_test) #evaluating model performance from sklearn.metrics import classification_report, confusion_matrix, accuracy_score print(confusion_matrix(y_test, y_pred)) print(classification_report(y_test, y_pred)) print(accuracy_score(y_test, y_pred))
  

  Start a new frame then click copy text button and paste the contents into the frame.

  The output is a confusion matrix, classification report and accuracy score.

  
  

  The confusion matrix, shows 9 true positive and 5 true negative.

  The confusion matrix,shows that our model predicted good values and actual good plans were 9.

  The confusion matrix show that 5 were actual poor plans and 5 were predicted to be poor plans.

  There were 2 false negatives. It is the total counts having prediction is Not a good plan while actually, it was a good plan.

  There were no false positives.

  Our sample test consisted of 16 reviews (80 times .20). Our model was pretty accurate 14 with correct predictions out of 16, .875.

  Making predictions with unseen plan reviews.

  Add the following line in a new frame. Here we are going to see if our model can predict a bad review.

  print(clf.predict(vectorizer.transform(['The whole plan was poorly done.'])))
  
  

  Save and run the code. Did you get a [0] for your output indicating that the above text is a negative review?

  Now type some positive sounding text, save and run. Did our model predict that this was a postive review?