How to answer a Regular Expression

Question

Run the regexp tokenizer with the regular pattern on the sentence “Mr. Black and Mrs. Brown attended the lecture by Dr. Gray, but Gov. White wasn’t there.”

a. Design and add a line to the pattern of this tokenizer so that titles like “Mr.” are tokenized as having the dot inside the token. Test and add some other titles to your list of titles.

b. Design and add the pattern of this tokenizer so that words with a single apostrophe, such as “wasn’t” are taken as a single token

Answer 1

To begin you can run the following simple regexp to get most words

sent = "Mr. Black and Mrs. Brown attended the lecture by Dr. Gray, but Gov. White wasn't there."

print(regexp_tokenize(sent,pattern="[A-Za-z]+"))

['Mr', 'Black', 'and', 'Mrs', 'Brown', 'attended', 'the', 'lecture', 'by', 'Dr', 'Gray', 'but', 'Gov', 'White', 'wasn', 't', 'there']

For part a, they want titles to include the periods. Looking at the sentence we can define titles as words that start with a capital and end with a period. We don't want to include the period at the end of the sentence unless the last word is a title. First search for exactly one capital: [A-Z] Then for 0 or more lowercase [a-z]* then exactly one period \. . We can then use and or operator to include the rest of the words, which looks like this:

print(regexp_tokenize(sent,pattern="[A-Z][a-z]*\.|[A-Za-z]+"))

['Mr.', 'Black', 'and', 'Mrs.', 'Brown', 'attended', 'the', 'lecture', 'by', 'Dr.', 'Gray', 'but', 'Gov.', 'White', 'wasn', 't', 'end']

For part B, we can see the wasn't is split into wasn and t and skips the apostrophe. If we include an apostrophe as a character to search for along with the rest of the letters, the apostrophe would be included in the token. That looks like this:

print(regexp_tokenize(sent,pattern="[A-Z][a-z]*\.|[A-Za-z']+"))

['Mr.', 'Black', 'and', 'Mrs.', 'Brown', 'attended', 'the', 'lecture', 'by', 'Dr.', 'Gray', 'but', 'Gov.', 'White', "wasn't", 'end']

Note: I copied and pasted the sentence here, and the apostrophe was not the common apostrophe (U+0027), but a right single quotation mark (U+2019). I replaced it with the regular apostrophe you type on the keyboard because I assume that is more in the spirit of the question

' vs ’

Note 2: ['A-Za-z]+\.*(?!$) is slightly simpler alternative solution without the OR(|) where we include the period only if we lookahead and it is not the end of the sentence. It would fail though if we end the sentence with a title. and it does not check if the first letter of a title is capitalized, so it would include the period in "st." for example. It works for that specific sentence though.