Assignment 5 – Processing words and phrases

Assigned

Friday, Feb 28, 2020

Due

Thursday, Mar 5, 2020 by 10:30pm

Summary

For this assignment, you will put your data science skills to work by processing string data and files.

Collaboration

You must work with your assigned partner(s) on this assignment. You may discuss this assignment with anyone, provided you credit such discussions when you submit the assignment.

Submitting

Email your answers to csc151-02-grader@grinnell.edu. The subject of your email should be [CSC151 02] Assignment 5 and should contain your answers to all parts of the assignment. Please send your scheme code in an attached .rkt file.

Problem 1: Parsing words

Topics: Strings

When processing textual data, it is often useful to extract out the individual words used from a sentence, paragraph, or even an entire transcribed speech. Notice that the file->words procedure does something like this, but it only works on an entire file. If we want to perform a similar operation on a single string, we are stuck.

Write and document a procedure (string->words str) that takes a single string and returns a list of individual words used in str. The procedure should return the words in lowercase as well as remove any punctuation and numbers. For example,

> (string->words "The quick brown fox jumps over the lazy dog.")
'("the" "quick" "brown" "fox" "jumps" "over" "the" "lazy" "dog")
> (string->words "That's funny. I wasn't aware of that!")
'("thats" "funny" "i" "wasnt" "aware" "of" "that")
> (string->words "Hey! The number 5 times 4 isn't equal to 30!")
'("hey" "the" "number" "times" "isnt" "equal" "to")

Hint: you might find the string-split, string-trim, char-alphabetic?, and string-downcase procedures helpful in this problem. You may find it useful to write a predicate for checking if a string is equal to the empty string.

Note: You need only remove spaces, normal punctuation (periods, commas, colons, semicolons, exclamation points, and question marks), and digits. As the examples suggest, apostrophes are treated slightly differently.

Disclaimer: Subsequent problems depend on your solution to this problem. If you do not finish this problem to your satisfaction, you may use (string-split str) as a satisfactory means of extracting words from strings in the following problems.

Problem 2: Comparing word usage in presidential debates

Topics: Files, Heterogeneous lists and tables, Strings

All presidential debates are transcribed and recorded. For example, Kaggle has a dataset of the presidential debates in 2016. We’ve downloaded the CSV dataset for you and named it us-debates.csv. Make a copy of the file (right-click and “Save As”) and familiarize yourself with the data. You will likely find that the us-debates.csv has some oddities to it. For example, the first row of the file contains header information. Furthermore, the candidates column documents the names of the speaking candidates, but it contains candidates named "QUESTION", "Audience", and "CANDIDATES" as well.

a. Define a variable us-debate-table that contains the parsed table from us-debates.csv, but with the header row removed and rows associated with the "QUESTION", "Audience", and "CANDIDATES" candidates filtered out.

b. Document and write a procedure, (total-words-of-candidate table), that takes a table in the format of us-debate-table and a string of the name of a speaker and returns the total number of words spoken by that speaker in the given table. For example:

> (total-words-of-candidate us-debate-table "Clinton")
19372
> (total-words-of-candidate us-debate-table "Pence")
5026

By “total number”, we mean all words, from all responses, from all debates, in the entire table.

c. Document and write a procedure, (average-word-length-of-candidate table speaker), that takes a table in the format of us-debate-table and a string of the name of a speaker and returns the mean length of words spoken by that speaker. For example:

> (average-word-length-of-candidate us-debate-table "Clinton")
4.315919884369193

Hint: another way of thinking about the mean length of word is that it is the total number of characters divided by the total number of words.

Problem 3: Frequently used words and phrases in debates

Topics: Files, Heterogeneous lists and tables, Strings, Verifying preconditions

a. Write a procedure, (frequently-used-words-kernel table speaker), that takes a table in the format of us-debate-table and a string name of a candidate or moderator and returns a list of word tallies sorted in descending order.

Run frequently-used-words-kernel to find the top 5 words used by several candidates and moderators in the us-debate-table file.

b. Document and write a procedure (frequently-used-words table speaker), that first verifies preconditions and then calls frequently-used-words-kernel. You should give helpful error messages for all possible inputs.

Problem 4: Categorizing words used in debates

Topics: files, strings, conditionals, tallying

We’ve seen a number of ways to categorize words. They may be short or long. They may start with or contain certain letters. They may contain repeated letters. They may be near other words. They may be common. They may be uncommon.

First, pick and describe between six and ten categories. Categories like common and uncommon may be more challenging to describe and implement than others, so it is up to you to choose how challenging of categories you want to use. Then, document and write a procedure, (categorize-word word), that gives the category for a word as a string. You should use “uncategorized” for words that do not fit into your categories. For example,

> (categorize-word "aardvark")
"starts with vowel"
> (categorize-word "jabberwocky")
"Carrollian"
> (categorize-word "defenestrate")
"uncommon"
> (categorize-word "madam")
"palindrome"
> (categorize-word "Grinnell")
"proper-name"
> (categorize-word "elephant")
"starts-with-vowel"
> (categorize-word "me")
"short"
> (categorize-word "hello")
"uncategorized"

If a word falls into multiple categories, you will pick only one. You may include lists of words to use in your categorizing and remember that index-of searches a list for a value matching the one provided. You may find string-ref and char-ci=? useful as well.

Next, document and write a procedure, (categorize-debate-words table speaker), that takes a table in the format of us-debate-table and a string name of a candidate or moderator and returns a list of tallies of how many words in each category the speaker used in the table.

Finally, categorize several candidates from the us-debate-table and see whether the categorization tells you anything about how they speak.

Note: You will almost certainly use a conditional to write categorize-word.

Problem 5: Map and Filter

Topics: basic recursion

A. As you know, the (map proc lst) procedure applies proc to each element of lst and creates a new list with the resulting values.

We can use recursion to implement our own map and better understand how it works.

Document and write a recursive procedure, (my-map proc lst), that works just like map with two arguments.

Note: You may not use map to implement my-map.

B. As you also know, the (filter pred? lst) procedure applies pred? to each element of lst and creates a new list with only the elements from lst that pred? returns #t for.

Document and write a recursive procedure, (my-filter pred? lst), that works just like filter.

Note: You may not use filter to implement my-filter.

Evaluation

We will primarily evaluate your work on correctness (does your code compute what it’s supposed to and are your procedure descriptions accurate); clarity (is it easy to tell what your code does and how it achieves its results; is your writing clear and free of jargon); and concision (have you kept your work short and clean, rather than long and rambly).