One of the many ways in which humanists now employ computers is in analyzing texts. That is, they use programs to identify characteristics of texts as a starting point for deeper reflections on those texts. These initial analyses can be complex—such as identifying sets of words that regularly occur together, which raises issues not only of efficiency but how you segment sets—while others can be more straightforward, such as looking at the frequency of adverbs or adjectives in a text.
Throughout this project, you should use good programming techniques that we have discussed so far including
let and let*)For this assignment, create one single file titled textual-analysis.rkt. Include your answers to all parts in this single file. Your file should contain a header (see example below), and should clearly label the various parts and subproblems using comments. Make sure to organize the file so that it’s easy for another human (e.g. your professor, mentors, graders) to read it. Turn in your file on Gradescope, and attempt to fix any errors that appear from the autograder.
# lang racket
(require rackunit)
(require csc151)
(require csc151/rex)
(require 2htdp/image)
; textual-analysis.rkt
; Author: Stu Dent
; Class: 151-02 Spring 02 2021
; Mini-Project 3, Parts 1-4
; Date: April 29, 2021
; Citations:
; XXX
; YYY
; Code here
Make sure to also include all text files that your code relies on in your submission.
As you saw in a recent lab, the tools you already know permit you to
count appearances of words in a longer text.
Document and write a procedure, (count-words list-of-words filename)
that takes as input a list of words and produces as output a list
of lists, each with a word and its frequency. Note that you should
do case-insensitive counting; “Sam”, “sam”, “SAM”, and even “saM”
should match “Sam”.
> (count-words (list "Sam" "Amazing" "Evil" "Funny") "sams-course-reviews.txt")
'(("Sam" 132) ("Amazing" 2) ("Evil" 666) ("Funny" 0))
You will likely need procedures which can be found in the strings reading, files, more list operations reading, and perhaps recursion.
You can test your procedure on sams-course-reviews.txt. We would recommend that you also create some simple sample files so that you can run more tests on them. However, if you include these tests in your main definitions pane and do not comment them out before submission, you must include your text files in your Gradescope submission.
There are a number of algorithms for computing the readability of prose. One that we will implement now is the Dale-Chall readability formula:
https://en.wikipedia.org/wiki/Dale%E2%80%93Chall_readability_formula
The formula given in the year 1948 is:
score = 0.1579 × (PDW × 100) + 0.0496 × ASL
ASL is the “average sentence length”, the total number of words in the text divided by the number of sentences in the text.
PDW is the “percentage of difficult words”, the number of words in the
text not found in Dale-Chall’s list of about 3000 familiar words:
http://countwordsworth.com/download/DaleChallEasyWordList.txt
Furthermore, if the percentage of difficult words is above 5%, then add 3.6365 to the score. Otherwise, keep the score as it is.
a. Document and write a procedure, compute-dale-chall-score, that takes three parameters:
num-difficult-words, the number of difficult words in the text;total-words, the total number of words in the text;num-sentences, the number of sentences in the text;and computes the Dale-Chall score from these statistics. Here is an example execution of the procedure:
> (compute-dale-chall-score 35 200 40)
6.64775
> (compute-dale-chall-score 25 100 18)
7.859555555555556
Experiment with your procedure using a variety of inputs to gain confidence that your procedure is implemented correctly.
b. This score can be then translated into a grade level for which the text is appropriate. The score-to-grade table from Wikipedia is given below:
4.9 or lower: "4th grade or lower"
Greater than 4.9 and less than or equal to 5.9: "5th-6th grade"
Greater than 5.9 and less than or equal to 6.9: "7th-8th grade"
Greater than 6.9 and less than or equal to 7.9: "9th-10th grade"
Greater than 7.9 and less than or equal to 8.9: "11th-12th grade"
Greater than 8.9: "13th-15th grade"
Document and write a procedure, (score->grade score), that
takes a Dale-Chall score and returns the grade level corresponding
to the range that score falls under. Your function should return
the grade level as the strings given in the above table.
c. Come up with a set of short texts that will be useful for exploring Dale-Chall scores. Feel free to use outside sources (with citation) such as novels, articles, etc. or try to write sensible sentences that target these grade levels.
In this part of your submission, write a list of the files that you have (at least 3) a short description, and the name you gave to each of the text files.
d. Write a procedure (extract-words str) which uses regular expressions to return a list of words
from a string. Note that a word is a sequence of letters and appostrophes.
> (extract-words "")
'()
> (extract-words "this and that and the other thing")
'("this" "and" "that" "and" "the" "other" "thing"))
> (extract-words "this; and!that and the?, other thing")
'("this" "and" "that" "and" "the" "other" "thing"))
> (extract-words "I'm going places")
'("I'm" "going" "places")
e. Write a procedure (dale-chall-score str) that takes a string
that represents a text, computes the various aspects of the string
(number of words, number of difficult words, number of sentences),
applies the computations above, and gives the Dale-Chall score for
the word. You can assume that every period represents a sentence
break and that only periods represent sentence breaks. You can
assume that every space represents a word break and that only spaces
represent word breaks.
In this part, you will need to use the file of easy words linked in part a. You should use
file->string and the procedure you wrote in part d to format the file as a list. (In this case, file->words will not work because some of the words have apostrophes, and file->words does not include apostrophes.)
You can verify that your code runs by taking one of the files you created in part c and running
(dale-chall-score (file->string example-file.txt)).
Another common approach to text analysis is called “sentiment analysis”. Broadly, sentiment analysis is intended to determine the writer’s overall sentiment in a piece of writing. Are they happy? Sad? Angry? Enthusiastic? I believe Amazon uses sentiment analysis in selecting reviews to prioritize and for other things, too.
The most straightforward form of sentiment analysis involves looking at word frequencies. A document with more positive than negative words is likely to be interepreted as positivve. A document with more negative than positive words is likely to be interpreted as negative.
Conveniently, there are two long lists of positive and negative words available to you.
a. Save these two files to your computer, in the same folder as your racket program. If you were to open these files manually, you would see that each has a long header at the top. Create lists named pos-words and neg-words respectively, using file->lines, take, and reverse to remove the header within your code. pos-words and neg-words should define lists, not procedures.
Note that the first positive word is "a+", and the first negative word is "2-faced". Also note that you will need to upload these files in your gradescope submission.
b. Document and write a procedure, (posneg str), that takes a
string as an input and returns "positive" if the number of
positive words is greater than the number of negative words,
"negative" if the number of negative words is greater than the
positive words, and "neutral" if the two are the same.
Although the tools we have created to analyze texts produce results, the results are only in the form of numbers and words. However, some people better understand information in visual form.
a. Document and write a procedure (visualize-text filename) that reads in an arbitrary text file (e.g. one of the texts from Project Gutenberg) and produces a useful visualization of some characteristics of that text. You should use the images library we learned at the beginning of this course to create your visualization. It is completely your choice what you want to visualize and how you want to visualize it - be creative!
Your goal is that the visualization provide useful information about texts in a way that others might easily understand, and it should work with any reasonable text file. You may use any of the preceding code you have written up to this point to analyze text, however you should also develop one or more new ways to analyze the text.
I will likely share examples of your images in class, and ask you to explain what we’re looking at.
b. Give a description (written in comments) of the type of visualization that your function visualize-text produces. For example: “My visualization is of a smiley face and a frowny face, the sizes of the faces are in proportion to the number of positive and negative words respectively”. (Note that this submission would not meet the requirement of part a of analyzing the text in one or more new ways.)
Submissions that lack any of these charactersitics will get an I.
[ ] File correctly named textual-analysis.rkt
[ ] File includes a header as instructed
[ ] All text files are included in submission (sample files from part 1, EasyWordList from part 2, short texts from part 2c, positive and negative words from part 3)
[ ] If the file references other files, it does so with the base file name, rather than a complex path
[ ] File runs without error
[ ] In part 1, the output is formatted corretly (as a list of length 2 lists)
[ ] In part 1, word counts are mostly correct
[ ] In part 2b, output strings match instructions exactly
[ ] In part 2c, at least three files are listed and described
[ ] In part 2d, extract-words works in some cases
[ ] In part 2e, values produced by dale-chall-score are close to correct
[ ] In part 3b, output strings match instructions exactly
Submissions that lack any of these characterstics will get an R.
[ ] Code was reformatted using Ctrl-I before submitting
[ ] All text files include a citation for where they were obtained
[ ] All procedures are documented and is mostly correct
[ ] Variable names are clear
[ ] Procedures avoid reading in files more than once
[ ] In part 2a, compute-dale-chall-score works on 100% of tests
[ ] In part 2b, score->grade works on 100% of tests
[ ] In part 2d, extract-words works on 100% of tests
[ ] In part 2e, dale-chall-score avoid repeated computations
[ ] In part 3a, file->lines, take, and reverse are used to correctly remove the header
[ ] In part 3a, pos-words and neg-words define lists
[ ] In part 3b, posneg works on 90% of tests
[ ] In part 4a, visualize-text works on arbitrary file inputs
[ ] In part 4a, visualize-text returns an image
[ ] In part 4b, an accurate and helpful description is given
Submissions that lack any of these characteristics will get an M.
[ ] In part 1, word counts works on 100% of tests
[ ] In part 1, avoids repeated work and decomposes the problem into sub problems
[ ] In part 2e, dale-chall-score works on 100% of tests
[ ] In part 3b, posneg works on 100% of tests
[ ] In part 4a, at least one new innovative text analysis technique is used