Python Archives - Page 3 of 7 - Michael's Programming Bytes

Python Lesson 46: Image Blurring (AI pt. 12)

Hello everybody,

Michael here, and in this post, we’ll explore image blurring! Image blurring is a pretty self-explanatory process since the whole point of image blurring is to make the image, well, blurry. This process has many uses, such as blurring the background on your work video calls (and yes, I do that all the time during work video calls).

Intro to image blurring

Now that we know a little bit about image blurring, let’s explore it with code. Here’s the image that we’ll be using:

The photo above is of Stafford Park, a lovely municipal park in Miami Springs, FL.

Unlike image eroding, image blurring has a pretty self-explanatory description since the aim of this process is to, well, blur images. How can we accomplish this through OpenCV?

Before we get into the fun image-blurring code, let’s discuss the three main types of image blurring that are possible with OpenCV:

Gaussian blur-this process softens out any sharp edges in the image
Median blur-this process helps remove image noise* by changing pixel colors wherever necessary
Bilateral blur-this process makes the central part of the image clearer while making any non-central part of the image fuzzier

*For those unfamiliar with image processing, image noise is (usually unwanted) random brightness or color deviations that appear in an image. Median blurring assists you with removing image noise.

Now that we know the three different types of image blurring, let’s see them in action with the code

Gaussian blur

Before we start to blur the image, let’s read in the image in RGB colorscale:

import cv2
import matplotlib.pyplot as plt

park=cv2.imread(r'C:\Users\mof39\OneDrive\Documents\20230629_142648.jpg', cv2.IMREAD_COLOR)
park=cv2.cvtColor(park, cv2.COLOR_BGR2RGB)
plt.figure(figsize=(10, 10))
plt.imshow(park)

Next, let’s perform a Gaussian blur of the image:

gaussianPark = cv2.GaussianBlur(park, (7, 7), 10, 10)
plt.figure(figsize=(10,10))
plt.imshow(gaussianPark)

Notice anything different about this image? The sharp corners in this photo (such as the sign lettering) have been smoothed out, which is the point of Gaussian blurring (to smooth out rough edges in an image).

Now, what parameters does the cv2.GaussianBlur() method take?

The image you wish to blur (park in this case)
A 2-integer tuple indicating the size of the kernel you wish to use for the blurring process-yes, this is similar to the kernels we used for image erosion in the previous post Python Lesson 45: Image Resizing and Eroding (AI pt. 11) (we’re using a 7-by-7 kernel here).
Two integers that represent the sigmaX and sigmaY of the Gaussian blur that you wish to perform. What are sigmaX and sigmaY? Both integers represent the numerical factors you wish to use for the image blurring-sigmaX being the factor for horizontal blurring and sigmaY being the factor for vertical blurring.

A few things to keep in mind regarding the Gaussian blurring process:

Just as you did with image erosion, ensure that both dimension of the blurring kernel are positive and odd-numbered integers (like the 7-by-7 kernel we used above).
sigmaX and sigmaY are optional parameters, but keep in mind if you don’t include a value for either of them, both will default to a 0 value, which might not blur your picture the way you intended. Likewise, if you use a very high value for both sigmas, you’ll end up with a very, very blurry picture.

Median blur

Since median blurring helps remove image noise, we’re going to be using this altered park image with a bunch of noise for our demo:

Next up, let’s explore the median blur with our noisyPark image:

medianPark = cv2.medianBlur(noisyPark, 5)
plt.figure(figsize=(10,10))
plt.imshow(medianPark)

As you can see, median blurring the noisyPark image cleared out a significant chunk of the image noise! But how does this function work? Let’s explore some of its parameters:

The image you wish to blur (noisyPark in this case)
A single integer indicating the size of the kernel you wish to use for the blurring process-yes, this is similar to the kernels we used for Gaussian blurring, but you only need a single integer instead of a 2-integer tuple (we’re using a 5-by-5 kernel here). The integer must be a positive and odd number since the kernel must be an odd number (same rules as the Gaussian blur apply here for kernel creation).

Bilateral blur

Last but not least, let’s explore bilateral blurring! This time, let’s use the non-noise altered park image.

bilateralPark = cv2.bilateralFilter(park, 12, 120, 120) 
plt.figure(figsize=(10,10))
plt.imshow(bilateralPark)

Wow! As I mentioned earlier, the purpose of bilteral blurring is to make a central part of the image clearer while make other, non-central elements of the image blurrier. And boy, does that seem to be the case here since the central element of the image (the park sign and all its lettering) really pops out while everything in the background seems a bit blurrier.

How does the cv2.bilateralFilter() function work its magic? Here’s how:

The image you wish to blur (park in this case)
The diameter (in pixels) of the region you wish to iterate through to blur-in this case, I chose a 12-pixel diameter as my “blurring region”. It works in a similar fashion to the kernels we used for our “erosion region” in the previous lesson.
The next two integers-both 120-are the sigmaColor and sigmaSpace variables, respectively. The sigmaColor variable is a factor that considers how much color should be considered in the blurring process while the sigmaSpace variable is a factor that considers the proximity of several elements in the image (such as the runners in the background). The higher both of these values are, the blurrier the background will be.

Thanks for reading,

Michael

Python Lesson 45: Image Resizing and Eroding (AI pt. 11)

Hello everybody,

Michael here, and today’s lesson will be our first foray into image manipulation with OpenCV. We’ll learn two new techniques for image manipulation-resizing and eroding.

Let’s begin!

Resizing images

First off, let’s start this post by exploring how to resize images in OpenCV. Here is the image we’ll be working with throughout this post

This is an image of a hawk on a soccer goal at Sevier Park (Nashville, TN), taken in August 2021.

Now, how could we possible resize this image? Take a look at the code below (and yes, we’ll work with the RGB colorscale version of the image) to first upload and display the image:

import cv2
import matplotlib.pyplot as plt

hawk=cv2.imread(r'C:\Users\mof39\Downloads\20210807_172420.jpg', cv2.IMREAD_COLOR)
hawk=cv2.cvtColor(hawk, cv2.COLOR_BGR2RGB)
plt.figure(figsize=(9, 9))
plt.imshow(hawk)

Before we start with resizing the image, let’s first get the image’s size (I’ll explain why this information will be helpful later):

print(hawk.shape)

(3000, 4000, 3)

To get the image’s size, use the print([image variable].shape) method. This method returns a 3-integer tuple that indicates height, width and dimensions; in the case of the hawk image, the image is 3000 px tall by 4000 px wide and 3-dimensional overall (px stands for pixels-recall that computer image dimensions are measured in pixels).

Now, how can we resize this image? Take a look at the code below:

smallerHawk = cv2.resize(hawk, (2000, 1500))
plt.imshow(smallerHawk)

As you can see here, we reduced the size of the hawk image in half without cropping out any of the image’s elements. How did we do that? We used the cv2.resize() method and passed in not only the hawk image but also a 2-integer tuple-(2000, 1500)-to indicate that I wanted to reduced the size of the hawk image in half.

Now, there’s something interesting about the (2000, 1500) tuple I want to point out. See, when we listed the shape of the image, the 3-inter tuple that was returned (3000, 4000, 3) listed the image’s height before its width. However, in the tuple we passed to the cv2.resize() method, the image’s width (well, half of the image’s width) was listed before the image’s height (rather, half the height). Listing the width before the height allows you to properly resize the image the way you intended.

Now, what happens when we make this image bigger? Take a look at the following code:

largerHawk = cv2.resize(hawk, (6000, 8000))
plt.figure(figsize=(9, 9))
plt.imshow(largerHawk)

Granted, the image may not appear larger at first, but that’s mostly due to how we’re plotting it on MATPLOTLIB. If you look closely at the tick marks on each axis of the plot, you will see that the image size has indeed doubled to 6000 by 8000 px.

Image erosion

The next image manipulation technique I want to discuss is image erosion. What does image erosion do?

The simple answer is that image erosion, well, erodes away the boundaries on an image’s foreground object, whatever that may be (if it helps, think of the OpenCV image erosion process like geological erosion, only for images). How the image erosion is acoomplished is more complicated than a simple method like cv2.resize(), however let’s explore the image erosion process in the code below:

import numpy as np
kernel = np.ones((5,5), np.uint8)
erodedHawk = cv2.erode(hawk, kernel)
plt.figure(figsize=(10,10))
plt.imshow(erodedHawk)

OK, so aside from the cv2.erode() method, we’re also creating a numpy array. Why is that?

Well, the numpy array kernel (aptly called kernel) is essentially a matrix of 1s like so:

Since we specified that our matrix is of size (5, 5), we get a 5-by-5 matrix of ones. Pretty simple right? Here are some other things to keep in mind when creating the kernel:

Make sure the kernel’s dimensions are both odd numbers to ensure the presence of a central point in the kernel.
Theoretically, you could create a kernel of 0s, but a kernel of 1s is better suited for image erosion.
Ideally, you should also include np.uint8 as the second parameter in the kernel creation. For those who don’t know, np.unit8 stands for numpy unsigned 8-bit integer. The reason I suggest using this parameter is because doing so will store the matrix as 8-bit integers, which is beneficial for memory optimization in computer programs.

Now, how does this kernel help with image erosion? See, the 5-by-5 kernel that we just created iterates through the image we wish to erode (hawk in this case) by checking if each pixel that borders the kernel’s central pixel is set to 0 or 1. If all pixels that border the central pixel in the image are set to 1, then the central pixel is also set to 1. Otherwise, the central pixel is set to 0?

What do the 0s and 1s all mean here? Notice how the leaves on the tree in this eroded image look slightly darker than the tree leaves in the original image. That’s because image erosion manipulates an image’s foreground (in this case, OpenCV percieves the tree as the foreground) by removing pixels from the foreground’s boundaries, thus making certain parts of the image appear slightly darker after erosion. The slightly darker tree leaves make the image of the hawk stand out more than it did in the original image.

Thanks for reading,

Michael

Python Lesson 44: Image Color Spaces (AI pt. 10)

Hello everybody,

Michael here, and today’s post will cover how to understand color spaces in images.

Granted, I’ve previously discussed various colorscales you can find in computer programming in this post-Colors in Programming-but in this post, we’ll take a deeper dive into the use of colors in images.

But first, what is a color space?

Well, as the header above asks, what is a color space? In the context of images, a color space is a way to represent a certain color channel in an image.

Still confused? Let’s take the image we used in our first computer vision lesson (it can be found here Python Lesson 42: Intro To Computer Vision Part One-Reading Images (AI pt. 8)). Assuming we’re analyzing the RGB image of Orange Boy, the color spaces simply represent the intensities (or spaces) of red, blue and green light in the image.

And now let’s analyze colorspaces in OpenCV

As the header says, let’s examine color spaces in Open CV! Here’s the image we’ll be using for this tutorial:

This is a photo of autumn at Bicentennial Capitol Mall State Park in Nashville, TN, taken in October 2022.

Before we start exploring colorspaces, let’s read in this image to our IDE using the RGB colorscale (which means you should remember to convert the image’s default colorscale):

import cv2
import matplotlib.pyplot as plt
park=cv2.imread(r'C:\Users\mof39\Downloads\20221022_101648.jpg', cv2.IMREAD_COLOR)
park=cv2.cvtColor(park, cv2.COLOR_BGR2RGB)
plt.figure(figsize=(18, 18))
plt.imshow(park)

Great! Now that we have our RGB image, let’s explore the different color channels!

First off, let’s examine this image’s red colorspace! How can we do that? Take a look at the code below:

B, G, R = cv2.split(park)
plt.figure(figsize=(18, 18))
plt.imshow(R, cmap='Reds')

plt.show()

In this example, I used the first line of code (the one with B, G, R) to split the image into three distinct colorspaces-blue, green and red.

Aside from the standard plt.figure() functions, I did make a slight modification to the plt.imshow() function. Instead of simply passing in the park image, I passed in the R variable so that we see the image’s red colorspace AND passed in the cmap parameter with a value of Reds to display the red colorspace in, well, red.

Now, how can we show the green and blue colorspaces? We’d use the same logic as we did for the red colorspace, except swap the R in the plt.imshow() function for G and B for the green and blue colorspaces and change the cmap values to Greens and Blues, respectively.

Here’s the image’s blue colorspace:

plt.figure(figsize=(18, 18))
plt.imshow(B, cmap='Blues')
plt.show()

And here’s the image’s green colorspace:

plt.figure(figsize=(18, 18))
plt.imshow(G, cmap='Greens')
plt.show()

As you can see from all three of these color-altered images, the sky, park lawn, and buildings in the background are ceratinly more coloed than the trees, which look bright-white in all three color-altered images.

A little more on colorspace

Now that we’ve examined image colorspaces a bit, let’s see how we can find the most dominant color in an image! Take a look at the code below (which uses the park image):

from colorthief import ColorThief
colorthief =
ColorThief(r'C:\Users\mof39\Downloads\20221022_101648.jpg')
dominantColor = colorthief.get_color(quality=1)
print(dominantColor)

(120, 94, 72)

Granted, you could realistically use a package like numpy to find the most dominant color in an image, but the colortheif module is a much more efficient (and more fun) approach.

In case you didn’t know, you’ll need to pip install the colortheif module.

After creating a ColorTheif object (and passing in the image’s filepath on your computer), you’ll then need to use the get_color() method and pass in quality=1 as this method’s parameter. Using the quality=1 parameter will extract the most dominant color in an image.

You can certainly use a variable to store the most dominant color like I did here (I used the dominantColor variable) but that’s completely optional.

Once you print the dominant color, you’ll notice you don’t get a color name, but rather a 3-integer tuple that represents the frequency of red, blue and green in the image (the tuple is based off of the RGB colorscale). In this case, our most dominant color is RGB(120, 94, 72). What does that translate to?

In plain English, the most dominant color in this image is a very desaturated dark orange. If you take a look at the original RGB image, it makes sense not only because of the color of the park lawn but also due to all the trees and buildings in the image.

What if you want to know not only the most dominant color in an image, but also its color palette? The colortheif module can help you there too! Here’s how:

palette = colorthief.get_palette(color_count=5)
print(palette)

[(120, 94, 72), (179, 192, 208), (130, 160, 197), (28, 31, 32), (182, 141, 108)]

Just as colortheif did with the most dominant color in an image, all colors are represented as RGB 3-integer tuples. The get_palette() function helps returns the top X colors used in the image-the X is represented by the value of the color_count parameter. In plain English, five colors used in this image include:

very desaturated dark orange (the most dominant color)
grayish blue
slightly desaturated blue
very dark almost black blue
slightly desaturated orange.

This feature is like imagining a painter’s palette in Python form-pretty neat right! As you can see, our painter’s paletter for the park image has a lot of blues and oranges.

Thanks for reading!

Michael

Python Lesson 43: Intro to Computer Vision Part Two-Writing & Saving Images (AI pt. 9)

Hello everybody,

Michael here, and in today’s post, we’ll continue our introduction to computer vision, but this time we’ll explore how to write images to a certain place on your computer using OpenCV.

Let’s begin!

Let’s write an image!

Before we begin, here’s the image we will be working with:

This is an image of Simba/Orange Boy and his sister Marbles (on Christmas Day 2017 excited to get their presents), both of whom got an acknowledgement in The Glorious Five-Year Plan Part Two.

Now, here’s the code to read in the image to the IDE:

cats=cv2.imread(r'C:\Users\mof39\Downloads\IMG_4778 (1).jpg', cv2.IMREAD_COLOR)
cats=cv2.cvtColor(cats, cv2.COLOR_BGR2RGB)

Once this image is read onto the IDE, here’s the code we’d use to not only write this image but also save it to a certain directory on your computer:

import os

imagePath = r'C:\Users\mof39\Downloads\IMG_4778 (1).jpg'
imageDestination = r'C:\Users\mof39\OneDrive\Documents'

cats = cv2.imread(imagePath)
os.chdir(imageDestination)

savedImage = 'simbaandmarbles.jpg'
cv2.imwrite(savedImage, cats)

What does all of this code mean? Let me explain.

You’ll first need to import the os module (or pip install it if you haven’t already done so)-this will help you write and save the image to a specific directory.

The two variables that follow-imagePath and imageDestination-represent the current location of the image on my computer and the location on my computer where I wish to write and save the image, respectively. In this case, my image is currently located in my Downloads folder and I wish to send it to my Documents folder.

The cats variable is the result of reading in the image of the cats to the IDE. The os.chdir() function takes in one parameter-the string containing the image destination path. This function will allow you to set the destination of the image to ensure that your image is written and saved to the location you set in the imageDestination variable.

The savedImage variable allows you to set both the image name and the image extension to the image you wish to save and write-in this case, my image will be named simbaandmarbles and it will have a jpg extension.

Last but not least, use the cv2.imwrite() function to write and save the image to your desired directory (represented by the imageDestination variable). You’ll notice that this function takes two parameters-savedImage and cats in this example-but why might that be? Take a look at the code above and you’ll see why!

See, savedImage is the name we’d like to use for the saved image-this is a necessary paramater because we want OpenCV to save the image using the name/extension we specified. cats saves the image itself to the desired location (or imageDestination).

You should certainly change the values of imagePath, imageDestination and savedImage to reflect accurate image locations/destinations/names/extensions on your computer!

But wait! How do we know if our code worked? Take a look at the output below:

True

Since the output of this code returned True, the image was succesfully written and saved to the desired destination on our computer! Want another way to verify if our code worked? Take a look at my Documents folder (which was my imageDestination):

As you can see, my image was succesfully written to my Documents folder with the name/extension I specified (simbaandmarbles/JPG).

Now we know the image was succesfully written and saved to the Documents folder, but how do we know if the rendering worked? In other words, did OpenCV zoom in or crop too much of the image (or change the colorscale during the writing/saving process)? Click on the image to find out:

As you can see, not only did OpenCV correctly write and save the image to the correct location, but it also wrote and saved the image without changing the zoom-in/zoom-out view or the image’s colorscale!

And that, dear readers, is how you can write and save an image anywhere on your computer using eight simple lines of code!

Thanks for reading.

Michael

Python Lesson 42: Intro To Computer Vision Part One-Reading Images (AI pt. 8)

Hello everybody,

Michael here, and today’s Python lesson looks to be quite a bit of fun! Wonder why?

We’re introducing a new topic today-computer vision!

Now, one of the Python concepts I’ve covered over the course of this blog’s run is NLP-or natural language processing, which is a form of AI. Computer vision (or CV for short) is also a form of AI, but instead of using text language, computer vision deals with images.

An intro to OpenCV

To further explore computer vision in Python, we’ll introduce a package called OpenCV. If you don’t already have this package installed, here’s the line of code to run (either on your IDE or command prompt) to install it:

pip install opencv-python

Yes, you’ll need to install the opencv-python package. Using the line pip install opencv won’t work.
If you pip installing this or any other package on your IDE, include the ! before the pip install line.

Once we get our package installed, let’s start exploring the fun stuff it can do!

And now, time to explore OpenCV

The OpenCV concepts we’ll explore in this post are two of its simpler functions-reading an image onto the IDE and displaying that image onto the IDE!

Before we begin, here’s the image I’ll be using for this section in case you want to follow along with this tutorial.

Regular readers of Michael’s Programming Byte’s will likely recognize this cat as Simba/The Orange Boy who (along with his sister Marbles) got a well-deserved recognition on the second part of my fifth anniversary post (The Glorious Five-Year Plan Part Two).

Now to read the image into Python, here’s the code we’ll use for this tutorial:

import cv2

cat = cv2.imread(r'C:\Users\mof39\Downloads\IMG_5427.jpg', cv2.IMREAD_COLOR)
cv2.imshow("image", cat)
cv2.waitKey(60000)
cv2.destroyAllWindows()

Not sure what all this code means? Let’s break it down:

To use Python’s OpenCV package, you’d need to run the line import cv2, not import opencv.
The cv2.imread() function takes two parameters-the path to your image and the mode you want to use to read the image into the IDE. As of this writing, OpenCV has 13 different modes to read an image into the IDE! The IMREAD_COLOR mode allows you to display the image as a standard color image
The cv2.imshow() function also takes two parameters-the word image and the image variable (cat in this case) and unlike the cv2.imread() function, this function displays the image on your IDE (in this case, a window will pop up)
The cv2.waitKey() method takes an integer as a parameter. The point of this function is to close the window with the image after a specified number of milliseconds-I used 60000 milliseconds in this case (equal to 1 minute).
The cv2.destroyAllWindows() function takes no parameters since all it does is, well, destroy all open windows in the IDE after the specified number of millseconds (specified by the cv2.waitKey() function).

The cv2.waitKey() and cv2.destoryAllWindows() functions are optional to include, but if you don’t include them, the window with the image will simply stay open unless you close it.

Troubleshooting and the magic of OpenCV colorscales

Now here’s what the image looks like after it’s read into the IDE with OpenCV:

I’ll be honest, even though OpenCV did succesfully read and display the image, it didn’t do a good job of processing the image. How can we fix this? Take a look at the code below:


import matplotlib.pyplot as plt
cat=cv2.imread(r'C:\Users\mof39\Downloads\IMG_5427.jpg', cv2.IMREAD_COLOR)
plt.figure(figsize=(8,8))
plt.imshow(cat)

Now, let’s see what kind of output we get:

OK, so what did we do differently? Well, we used a combination of the MATPLOTLIB* and OpenCV packages to read our image onto the IDE and display it as well. While that combination of packages did the trick when it came to displaying the entire image on the IDE, you’ll notice that the cat (along with most other things in the image) looks rather blue.

*The plt.imshow() function is a MATPLOTLIB function.

Why might that be? After all, we did use the IMREAD_COLOR mode to read the image into the IDE, so how did we get this result? Take a look at the revised code below for a solution to this issue:

cat=cv2.imread(r'C:\Users\mof39\Downloads\IMG_5427.jpg', cv2.IMREAD_COLOR)
cat=cv2.cvtColor(cat, cv2.COLOR_BGR2RGB)
plt.figure(figsize=(8,8))
plt.imshow(cat)

Here’s the output we get:

I used most of the same code as I did for the previous example, with one additional line-the line that uses the cv2.cvtColor() function.

Why did I need to use this function? Well, if you’re wondering why Orange Boy looked a little blue in the first image, that’s because OpenCV read the first image in the BGR colorscale, which is the OpenCV default method of reading images.

The BGR colorscale stands for blue, green, red colorscale and it reads in images based off of the intensity of blue, green, and red light in the image. Since the blue light in the image was the most intense, the image looked blue upon display.

Now, here’s where the cv2.cvtColor() function comes in! This function takes two parameters-the image whose colorscale you want to convert and the conversion mode you want to use for the image. There are over a dozen conversion modes you can use for the image, but in this example, we’ll use the COLOR_BGR2RGB conversion mode, which changes the image’s colorscale from BGR to RGB (or red, green, blue). After running this function, then run the plt.imshow() function on the color-converted Orange Boy image so that you can see the normal-looking image, not the Blue Boy image.

Now, why does OpenCV read in images with a BGR colorscale by default? The reason for this is because back when OpenCV was first developed (in the summer of 2000), the BGR colorscale was the more popular colorscale to use for computer graphics. However, the RGB colorscale has since become the more widely-adopted colorscale for computer graphics (including Python packages like MATPLOTLIB)-and in all my honesty, I think it makes images display a lot better.

Another colorscale conversion, why not?

Now that I’ve taught you the basics of reading images into the Python IDE with the OpenCV package, let’s have a little fun with the colorscale conversions, shall we? Take a look at the code below:

cat=cv2.imread(r'C:\Users\mof39\Downloads\IMG_5427.jpg', cv2.IMREAD_COLOR)
cat=cv2.cvtColor(cat, cv2.COLOR_BGR2RGB)
cat=cv2.cvtColor(cat, cv2.COLOR_RGB2HSV)
plt.figure(figsize=(8,8))
plt.imshow(cat)

In this example, I converted the image of Orange Boy first to the RGB colorscale and then to the HSV (hue, saturation, value) colorscale. As you can see, the cat looks like something out of a thermal camera.

If you want an image like the one I got here, then you’ll need to first convert the image to RGB colorscale BEFORE converting to another colorscale!
The cv2.cvtColor() function contains about 150 different color conversion modes!
Just in case you’re wondering what the HSV colorscale is, here’s a simple three-bullet-point explanation:
H stands for hue, which represents the type of color the image contains. The value for H is represented on the color wheelusing a value from 0-360 degrees (as in angle degrees).
S stands for saturation, which represents the intensity (or well, saturation) of the color. The value for S is represented on a scale between 0% (fully desaturated pure grey)-100% (fully saturated pure color)
V stands for value (or brightness), which represents how bright or dark the color will appear in the image. The value for V, just as with S, is represented on a scale between 0% (fully black)-100% (fully illuminated color).

Thanks for reading and be sure to stay tuned for Part Two of Intro to Computer Vision!

Michael

The Glorious Five-Year Plan Part Two

Hello everybody!

Michael here, and as I previously promised, here’s part 2 of my five-year blog anniversary post (after all, what fun it is to split the 5-year anniversary letter into two posts).

My Answer To Michael’s Five-Year Coding Challenge

Also, as I previously promised on the last post, I will discuss my solution for Michael’s Five-Year Coding Challenge. I know I said you can use any programming language you like for this challenge (as long as your code follows certain criteria outlined in my previous post https://michaelsprogrammingbytes.com/2023/06/13/the-glorious-five-year-plan-part-one/), but here’s my approach using Python!

secretCode = {'Z': 'A', 'Y': 'B', 'X': 'C', 'W': 'D', 'V': 'E', 'U': 'F', 'T': 'G', 'S': 'H', 'R': 'I', 'Q': 'J', 'P': 'K', 
              'O': 'L', 'N': 'M', 'M': 'N', 'L': 'O', 'K': 'P', 'J': 'Q', 'I': 'R', 'H': 'S', 'G': 'T', 'F': 'U', 'E': 'V', 
              'D': 'W', 'C': 'X', 'B': 'Y', 'A': 'Z'}
encodedString = """Gszmp blf gl zoo nb ivzwvih uli urev dlmwviufo bvzih. R dlfowm'g szev pvkg gsrh yolt ifmmrmt uli zh olmt zh R wrw drgslfg blfi dlmwviufo hfkklig.  slkv blf ovzimvw hlnvgsrmt zolmt gsv dzb gsvhv ozhg urev bvzih-zmw, svb, nzbyv blf tzrmvw z olev lu kiltiznnrmt (zmw kviszkh mvd qly hprooh) rm gsv kilxvhh. Gszg dlfow xvigzrmob nzpv nv z evib szkkb dirgvi. Sviv'h gl nzmb, nzmb nliv bvzih lu xlwrmt gltvgsvi! Nrxszvo"""
punctuation = ['\'', '.', '!', '(', ')', '-']
decodedString = ''

for e in encodedString:
    for k, v in secretCode.items():
        if e == k:
            decodedString += v
        elif e.lower() == k.lower():
            decodedString += v.lower()
                   
    if e == ' ':
        decodedString += ' '
    elif e in punctuation:
        decodedString += e

print(decodedString)

As you can see, I solved this coding challenge with 20 simple lines of Python code (not including the output). How did I accomplish this? Here are 10 things I kept in mind when solving my own puzzle:

I used the secretCode dictionary to set my reverse substitution cipher (HINT: even though there are both uppercase and lowercase letters, you won’t need two separate dictionaries for the cipher-I’ll explain why).
I created a list containing all of the punctuation in this message-you’ll see why it’s important later. The list is aptly named punctuation.
I used the decodedString variable to store the decoded message-I personally thought it was more convinient than figuring out how to replace all the characters in encodedString one by one.
I created a nested for-loop to iterate both through the characters in encodedString and the items (that is, both keys and values) in the secretCode dictionary.
In the nested loop where I iterate through the items of the secretCode dictionary (for k, v in secretCode.items()), I check if each character in the encodedString string equals the current corresponding key in the secretCode dictionary and if so, I append the corresponding value to the decodedString string.
Remember how I said that even though my secret message contains both upper and lower-case letters you won’t need to create two dictionaries? Well, the statement elif e.lower() == k.lower() handles this scenario in just two lines of code by checking to see if the lowercase version of the character in encodedString equals the lowercase version of the corresponding key and if so, I append the lowercase corresponding value to the decodedString string.
How would I handle spaces and punctuation in the encodedString? Well, in the case of spaces, if a character in encodedString is a space, I append it to the decodedString just as I did with the letters.
The same logic applies for any punctuation in encodedString, but this time, I check to see if a character equals any element in the punctuation list that I mentioned in item #2.
One thing I kept in mind when checking for spaces and punctuation in encodedString-I kept the if-elif statement pair OUTSIDE of the dictionary loop because if I placed those statements inside that loop, I would’ve ended up with a ton of spaces and punctuation. However, placing this code outside of the dictionary loop ensures that I end up with the correct amount of spaces and punctuation.
Last but not least, I printed the decodedString message. Without further ado, here it is:

Thank you to all my readers for five wonderful years. I wouldn't have kept this blog running for as long as I did without your wonderful support.  hope you learned something along the way these last five years-and hey maybe you gained a love of programming (and perhaps new job skills) in the process. That would certainly make me a very happy writer. Here's to many many more years of coding together! Michael

Trust me, dear readers, I mean every word of this. I wouldn’t kept this blog running as long as I did without you!

One More Note

I know I’ve emphasized this over these last two posts, but thank you, thank you, thank you loyal readers for reading my blog for the last five years. Hopefully you’ll keep coming back for more, because boy do I have several more years of great coding content I can provide (and you bet it’ll still be good when I hit year 10 and beyond)!

However, there is someone I wanted to acknowledge on this five-year anniversary. Rather, a furry friend I’d like to mention.

His name is Simba (or Orange Boy) and he is my fluffy orange cat who certainly helped me during the blog’s early days (and yes, he was there when I wrote the welcome post). Here he is on the night of September 17, 2018, likely looking over me as I wrote another post (or searching for a post college job, as I was in the thick of job hunting in fall 2018). He is certainly a good fluffy boy!

Also, here’s Simba with Marbles, his sister (the brown cat aka Pretty Girl), who accompanied me during writing every now and then:

The kitties on Christmas morning 2017-pre blog days.

To many more years of developing together,

Michael

Python Lesson 41: Word2Vec (NLP pt.7/AI pt.7)

Hello everybody,

Michael here, and in today’s lesson, we’ll discover another AI NLP algorithm-word2vec (recall that a few posts back in Python Lesson 40: The NLP Bag-Of-Words (NLP pt. 6/AI pt.5) we discussed the bag-of-words algorithm).

What is the word2vec algorithm?

So, how does the word2vec algorithm work? Let’s say we were to process a text document using this algorithm. The word2vec algorithm would analyze each word in this document along with all other words commonly found near a certain word. For instance, if the word “family” was found in a document and the words “daughter”, “parents”, and “generations” were found near it-the word “family” would be lumped in with these words, as they are all related to each other.

If you were to ask your program for the three most similar words to “family” (according to our hypothetical document), it would say “daughter”, “parents” and “generations”. How would the word2vec algorithm find the most common words? By using a mathematical metric called cosine similarity-more on that later.

Preparing the data

So, before we dive into the magic of word2vec, let’s gather the data we’re going to use.

In this example, we’ll utilize this ChatGPT-generated essay below that contains five paragraphs on a very relevant topic in the year 2023-returning to the office-including both sides of the debate.

return-to-office-1 Download

So, let’s open up our Python IDEs and start coding!

First, let’s open and read this file into our IDE:

with open(r'C:\Users\mof39\OneDrive\Documents\return to office.txt', 'r', encoding='utf-8') as file:
    word2VecFile = file.read()
    print(word2VecFile)

In Favor of Returning to Office:

Many argue that returning to the office is crucial for maintaining the productivity and collaboration necessary for businesses to thrive. Working in the same space allows for easier communication, quicker decision-making, and better team building. It also provides opportunities for social interactions that can improve employee morale and job satisfaction.

Another benefit of returning to the office is the separation of work and home life. With many employees working remotely during the pandemic, the lines between work and personal time have become blurred. This can lead to burnout and other negative consequences. By returning to the office, employees can more easily maintain a work-life balance and avoid the mental exhaustion that comes with always being “on.”

Against Returning to Office:

On the other hand, many people argue that remote work has proven to be effective, and there is no need to return to the office. In fact, studies have shown that remote workers are often more productive than those who work in the office. Additionally, remote work allows for more flexibility in scheduling and reduces the amount of time and money spent commuting.

Another important consideration is the health and safety of employees. With the ongoing threat of COVID-19 and the emergence of new variants, returning to the office could put employees at risk. Some may not feel comfortable being in close proximity to their colleagues or may have concerns about the effectiveness of safety protocols. In these cases, continuing to work remotely may be the best option.

Ultimately, the decision to return to the office or continue remote work will depend on a variety of factors, including the type of work being done, the needs of the business, and the preferences of employees. It is important to consider all perspectives and prioritize the health and safety of everyone involved.

You can technically open a .txt file with the pd.read_csv method, but I prefer using the with...open method of reading .txt files since the pd.read_csv method outputs the text file in tabluar form, which isn’t going to work for this tutorial.

And now let’s create the word2vec model

Now that we’ve read our text file into the system, the next thing we’ll do is create our word2vec model. Here’s the code to do so:

First, lets gather our list of tokens for the analysis:

import nltk
import gensim

modelData = []
punctuation = [',', '.', '”', '“', ':']

for w in nltk.sent_tokenize(word2VecFile):
    tokens = []
    
    for t in nltk.word_tokenize(w):
        if t not in punctuation:
            tokens.append(t.lower())
        
    modelData.append(tokens)
    
print(modelData)

[['in', 'favor', 'of', 'returning', 'to', 'office', 'many', 'argue', 'that', 'returning', 'to', 'the', 'office', 'is', 'crucial', 'for', 'maintaining', 'the', 'productivity', 'and', 'collaboration', 'necessary', 'for', 'businesses', 'to', 'thrive'], ['working', 'in', 'the', 'same', 'space', 'allows', 'for', 'easier', 'communication', 'quicker', 'decision-making', 'and', 'better', 'team', 'building'], ['it', 'also', 'provides', 'opportunities', 'for', 'social', 'interactions', 'that', 'can', 'improve', 'employee', 'morale', 'and', 'job', 'satisfaction'], ['another', 'benefit', 'of', 'returning', 'to', 'the', 'office', 'is', 'the', 'separation', 'of', 'work', 'and', 'home', 'life'], ['with', 'many', 'employees', 'working', 'remotely', 'during', 'the', 'pandemic', 'the', 'lines', 'between', 'work', 'and', 'personal', 'time', 'have', 'become', 'blurred'], ['this', 'can', 'lead', 'to', 'burnout', 'and', 'other', 'negative', 'consequences'], ['by', 'returning', 'to', 'the', 'office', 'employees', 'can', 'more', 'easily', 'maintain', 'a', 'work-life', 'balance', 'and', 'avoid', 'the', 'mental', 'exhaustion', 'that', 'comes', 'with', 'always', 'being', 'on.', 'against', 'returning', 'to', 'office', 'on', 'the', 'other', 'hand', 'many', 'people', 'argue', 'that', 'remote', 'work', 'has', 'proven', 'to', 'be', 'effective', 'and', 'there', 'is', 'no', 'need', 'to', 'return', 'to', 'the', 'office'], ['in', 'fact', 'studies', 'have', 'shown', 'that', 'remote', 'workers', 'are', 'often', 'more', 'productive', 'than', 'those', 'who', 'work', 'in', 'the', 'office'], ['additionally', 'remote', 'work', 'allows', 'for', 'more', 'flexibility', 'in', 'scheduling', 'and', 'reduces', 'the', 'amount', 'of', 'time', 'and', 'money', 'spent', 'commuting'], ['another', 'important', 'consideration', 'is', 'the', 'health', 'and', 'safety', 'of', 'employees'], ['with', 'the', 'ongoing', 'threat', 'of', 'covid-19', 'and', 'the', 'emergence', 'of', 'new', 'variants', 'returning', 'to', 'the', 'office', 'could', 'put', 'employees', 'at', 'risk'], ['some', 'may', 'not', 'feel', 'comfortable', 'being', 'in', 'close', 'proximity', 'to', 'their', 'colleagues', 'or', 'may', 'have', 'concerns', 'about', 'the', 'effectiveness', 'of', 'safety', 'protocols'], ['in', 'these', 'cases', 'continuing', 'to', 'work', 'remotely', 'may', 'be', 'the', 'best', 'option'], ['ultimately', 'the', 'decision', 'to', 'return', 'to', 'the', 'office', 'or', 'continue', 'remote', 'work', 'will', 'depend', 'on', 'a', 'variety', 'of', 'factors', 'including', 'the', 'type', 'of', 'work', 'being', 'done', 'the', 'needs', 'of', 'the', 'business', 'and', 'the', 'preferences', 'of', 'employees'], ['it', 'is', 'important', 'to', 'consider', 'all', 'perspectives', 'and', 'prioritize', 'the', 'health', 'and', 'safety', 'of', 'everyone', 'involved']]

Before creating our tokens lists to use for our model, I first imported the nltk and gensim packages (as always, if there’s a required package you don’t have, pip install it!).

I then created two lists, modelData and tokens. Then I sentence-tokenized this file before word-tokenizing it, appending MOST of the tokens to the tokens list before appending each tokens list to the modelData list. I say appending MOST (not all) of the tokens as I excluded any tokens that were part of the punctuation list.

Now, you may be wondering if we should remove stopwords here like we’ve done for our previous NLP tutorials. Normally, I’d say yes, but since the text is so short and we’re trying to figure out word connections, I’d say keep the stopwords here to get more accurate word2vec results.

Now that we’ve got our tokens list-or should I say lists-within-a-list-it’s time to build our model!

Rather, I should say model(s), since there are two approaches we can take to build our word2vec analysis-skipgrams and continous-bag-of-words.

Skip-gram

The first word2vec approach we’ll explore is the skip-gram model.

What exactly is the skip-gram model, though? Using our text file as an example, let’s say we were trying to guess which words we’d commonly see before and after the word “working”. The skip-gram model would analyze our text file to guess which words we’d likely see before and after the word “working” such as “remotely” or “hardly”.

Now, how do we implement this model in Python? Take a look at the code below:

skipGram = gensim.models.Word2Vec(modelData, min_count = 1, vector_size = 2, window = 5, sg = 1)

What do each of these parameters mean. Let me explain:

min_count-sets the minimum amount of times a word must appear in a document to be factored into the skip-gram analysis; in this example, a word must appear in the document at least once to be factored into the skip-gram analysis
vector_size-sets the number of dimensions that each word vector will contain; in this example, each word vector will contain 2 dimensions
window-sets the maximum distance between current and predicted words in the document; this skip-gram analysis will look at the five words following and preceding any given word to determine cosine similarity (more on that later)
sg-if this value is 1, use the skip-gram analysis; if this value is 0, use the continous bag-of-words analysis (more on that later)

Now that we have our skip-gram word2vec model set up, let’s test it on three word pairs:

print("Cosine similarity between 'returning' and 'office' - Skip Gram : ", skipGram.wv.similarity('returning', 'office'))     
print("Cosine similarity between 'remote' and 'covid-19' - Skip Gram : ", skipGram.wv.similarity('remote', 'covid-19'))
print("Cosine similarity between 'reduces' and 'commuting' - Skip Gram : ", skipGram.wv.similarity('reduces', 'commuting'))

Cosine similarity between 'returning' and 'office' - Skip Gram :  0.9542125
Cosine similarity between 'remote' and 'covid-19' - Skip Gram :  -0.8962952
Cosine similarity between 'reduces' and 'commuting' - Skip Gram :  0.9316714

In this example, we are analyzing the cosine similiarty between three word pairs-returning & office, remote & covid-19 and reduces & commuting. As you can see from the output, two of the word pairs have positive cosine similarity while the other word pair has negative cosine similarity.

What does positive and negative cosine similarity mean? Well, the higher the positive cosine similarity, the more semantically similar the two words are to each other (in the context of the document being analyzed). The lower the negative cosine similarity, the more dissimilar the two words are to each other (again, in the context of the document being analyzed).

Just a tip, but using a vector_size of 2 is often not ideal, especially because most NLP analyses work with larger documents. I used a vector_size of 2 here because the document we’re working with is rather small.
Using a vector size of 0 or 1 won’t work for either the skip-gram or continuous bag-of-words model, as it will return cosine similarities of 1, -1, or 0, which aren’t ideal for your analysis.

Continuous bag-of-words

Now that we’ve explored the skip-gram model, let’s now analyze the continous bag-of-words model.

What is the continuous bag-of-words model? Well, like the skip-gram model, the continuous bag-of-words model is like a word2vec guessing game. However, while the skip-gram model attempts to predict words that will come before and after a given word, the continuous bag-of-words model will analyze a sentence in a document and try to predict what any given word in a sentence would be based on the surrounding words in the sentence.

Here’s a simple example:

The happy couple decided to close on their first _____ yesterday, eagerly anticipating all the memories they would make there.

Just from this sentence alone, what do you think the missing word would be? If you guessed house, you’d be right! In this example, the continuous bag-of-words model would look at each word in the sentence and based on the given words, guess the missing word.

Now let’s see how to implement the CBOW (continuous bag-of-words) model in Python.

CBOW = gensim.models.Word2Vec(modelData, min_count = 1, vector_size = 2, window = 5, sg = 0)

Now, here’s the best part about the CBOW model-it’s same up in exactly the same way, with the same parameters, as the skip-gram model. The only difference between the two models is that you’d need to set the sg parameter to 0 to indicate that you’d like to use the CBOW model.

Now let’s test our CBOW word2vec model on the same three word pairs we used for the skip-gram model.

print("Cosine similarity between 'returning' and 'office' - Skip Gram : ", CBOW.wv.similarity('returning', 'office'))     
print("Cosine similarity between 'remote' and 'covid-19' - Skip Gram : ", CBOW.wv.similarity('remote', 'covid-19'))
print("Cosine similarity between 'reduces' and 'commuting' - Skip Gram : ", CBOW.wv.similarity('reduces', 'commuting'))

Cosine similarity between 'returning' and 'office' - Skip Gram :  0.95738184
Cosine similarity between 'remote' and 'covid-19' - Skip Gram :  -0.9320767
Cosine similarity between 'reduces' and 'commuting' - Skip Gram :  0.94031644

Assuming a change in word2vec model but leaving all other parameters unchanged, we can see that the cosine similarity scores between the three word pairs show small differences from the skip-gram cosine similarity scores for the same three word pairs.

Cosine similarity explained

So, now that I’ve shown you all what cosine similarity looks like, it’s time to explain it further.

In the word2vec algorithm (for both the skip-gram and CBOW models), each word in the document is treated like a vector. Still unsure about the whole vector thing? Here’s an illustration that might help:

Imagine a simple right triangle, where the three words returning, office, and working are the angles. A and B are the sides while C is the triangle’s hypotenuse.

Let’s say you wanted to find the cosine similarity between the words returning and office. If you know even basic trigonometry, you’ll know that the cosine of a right triangle is the ratio of the length of the side adjacent to a given angle to the length of the triangle’s hypotenuse. The cosine would represent the cosine similarity between these two words.

In word2vec, cosine similarity scores can range from -1 to 1. Here’s an illustration on cosine similarity scores:

In simple terms, a word pair with a cosine similarity score of -1 or 0 indicates completely different, not-at-all semantic similar words (in the context of the document being analyzed). A word pair with a cosine similarity score greater than -1 but less than 0 indicates that the two words are somewhat, but not entirely, dissimilar. Finally, a word pair with a cosine similarity score of 1 indicates that the two words are identical or very, very similar semantically.

In the example above, the words returning and office have cosine similarity scores of roughly 0.95 (for both the skip-gram and CBOW models), indicating that these two words are quite semantically similar in the context of the document. Interestingly, the words remote and covid-19 have cosine similarity scores lower than -0.85 (for both the skip-gram and CBOW models)-personally I find this interesting as the word remote seems semantically similar to the word covid-19 in the context of this document (to me at least).

How is negative cosine similarity possible? Well, think of an obtuse traingle. In an obtuse triangle, one angle has to be greater than 90 degrees, which means the cosine of that angle can possibly be negative. Think of NLP cosine similarity the same way.

Thanks for reading,

Michael

Python Lesson 40: The NLP Bag-Of-Words (NLP pt. 6/AI pt.5)

Hello everybody,

Michael here, and in today’s post, we’re going to explore a Python NLP machine learning/AI technique known as the bag-of-words.

What is the bag-of-words?

Good question-what is Python’s bag-of-words technique? The bag-of-words is a simple NLP algorithm that turns text into fixed-length vectors by counting the number of times a word occurs in a text string or document. The information that the bag-of-words algorithm provides is useful for various NLP tasks such as topic modelling (along the lines of categorizing a news article based on its content) and sentiment analysis, among other things.

Do you wonder why this algorithm is called the bag-of-words? The bag-of-words algorithm represents a text string or document as a, well, “bag” of words. All this algorithm does is count how many times a word appears in a text string or document-the string/document’s syntax and semantics aren’t taken into account here. By that I mean if we have a word like free in a sentence that’s used as both a noun and a verb, the bag-of-words algorithm won’t take the different tenses of the word into account.

It’s data preparation time!

Now that you know the gist of the bag-of-words algorithm, let’s implement it in Python!

However, before we get to the fun part (implementing the algorithm), let’s first import the three packages and download the two NLTK modules we’ll be using in this lesson:

import pandas
import nltk
nltk.download('punkt')
nltk.download('stopwords')
from nltk.corpus import stopwords

Next up, let’s add a list of strings that we will be analyzing:

reviews = ["Wow! You’ll say that over and over again as this mind-blowing, superhero epic unfolds. Wow!",
          "The tribute here is heartfelt, but the spirit of the man and the character sometimes get lost in all the bric-a-brac of the Marvel machine... the film lands on a triumphant note of succession, as it must– the gods inside and above the narrative demand it.",
          "An exercise in superhero mourning done right.",
          "The MCU’s mechanics are too oppressive to allow for true mournful meditation.",
          "This soulful sequel teams an emotional tribute to late star Chadwick Boseman with some spectacular visual action. A maturity milestone for the Marvel Cinematic Universe, starring Angela Bassett and Winston Duke.",
          "The opening and closing sequences of Wakanda Forever will make your heart ache. But at 2hrs 41mins, this is also one of the longest films in the MCU. And there are long stretches in it which border on boredom. I was weepy but also weary.",
          "Coogler pulls off an incredible feat, despite some story stumbles, creating a superhero film that is emotionally affecting, politically and culturally urgent, and that pays loving tribute not just to T’Challa but Chadwick Boseman too.",
          "“Wakanda Forever” is the first blockbuster wake, and it’s powered not by vibranium but by its vibrant and fully felt emotions.",
          "For all its comic-book violence, over-the-top villainy, and too dark CGI, at its core this is a film about dealing with loss.",
          "It’s both a tribute to the late Chadwick Boseman and a problem for the movie that “Black Panther: Wakanda Forever” feels his loss so keenly.",
          "Presented the daunting task of bidding farewell to a star tragically taken in his prime in sober but stirring fashion, Coogler has given audiences, and the studio, a solidly and gracefully executed dive into a “Wakanda” for right now."]

In this example, we’re going to analyze 11 randomly selected critic reviews from the most recently released MCU (Marvel Cinematic Universe) film Black Panther: Wakanda Forever-which, by the way, is one of the MCU’s best entries since Avengers: Endgame.

Well, now that Ant-Man and the Wasp: Quantumania is out, Black Panther: Wakanda Forever is no longer the most recently released MCU film.

Now that we have the strings that we are going to analyze, let’s start analyzing! The first step in our analysis will be data preparation-which should be the first step in any data analysis you do. Here’s one way to approach the data preparation process:

stopwordsList = set(stopwords.words('english'))
tokensList = []

for r in reviews:
    tokens = nltk.word_tokenize(r)                
    tokens = list(filter(lambda word: word not in '.!,’“”:...', tokens))
    tokens = list(filter(lambda word: word.casefold() not in stopwordsList, tokens))
        
    if 'must-' in tokens:
        tokens.remove('must-')

    tokensList.append(tokens)
               
    print(tokens)  

['Wow', 'say', 'mind-blowing', 'superhero', 'epic', 'unfolds', 'Wow']
['tribute', 'heartfelt', 'spirit', 'man', 'character', 'sometimes', 'get', 'lost', 'bric-a-brac', 'Marvel', 'machine', 'film', 'lands', 'triumphant', 'note', 'succession', 'must–', 'gods', 'inside', 'narrative', 'demand']
['exercise', 'superhero', 'mourning', 'done', 'right']
['MCU', 'mechanics', 'oppressive', 'allow', 'true', 'mournful', 'meditation']
['soulful', 'sequel', 'teams', 'emotional', 'tribute', 'late', 'star', 'Chadwick', 'Boseman', 'spectacular', 'visual', 'action', 'maturity', 'milestone', 'Marvel', 'Cinematic', 'Universe', 'starring', 'Angela', 'Bassett', 'Winston', 'Duke']
['opening', 'closing', 'sequences', 'Wakanda', 'Forever', 'make', 'heart', 'ache', '2hrs', '41mins', 'also', 'one', 'longest', 'films', 'MCU', 'long', 'stretches', 'border', 'boredom', 'weepy', 'also', 'weary']
['Coogler', 'pulls', 'incredible', 'feat', 'despite', 'story', 'stumbles', 'creating', 'superhero', 'film', 'emotionally', 'affecting', 'politically', 'culturally', 'urgent', 'pays', 'loving', 'tribute', 'Challa', 'Chadwick', 'Boseman']
['Wakanda', 'Forever', 'first', 'blockbuster', 'wake', 'powered', 'vibranium', 'vibrant', 'fully', 'felt', 'emotions']
['comic-book', 'violence', 'over-the-top', 'villainy', 'dark', 'CGI', 'core', 'film', 'dealing', 'loss']
['tribute', 'late', 'Chadwick', 'Boseman', 'problem', 'movie', 'Black', 'Panther', 'Wakanda', 'Forever', 'feels', 'loss', 'keenly']
['Presented', 'daunting', 'task', 'bidding', 'farewell', 'star', 'tragically', 'taken', 'prime', 'sober', 'stirring', 'fashion', 'Coogler', 'given', 'audiences', 'studio', 'solidly', 'gracefully', 'executed', 'dive', 'Wakanda', 'right']

So, how exactly did I preprocess the data? Well, I first created a stopwordsList, which will allow us to filter out all of the [English] stopwords from the text. I also created a tokensList that I will append each of the processed tokens to-I’ll explain this more further in the post.

When running NLP analyses, you don’t necessarily have to remove the stopwords from the text you’re analyzing-it’s more of a best practice thing to do!

After creating my stopwords list, I then ran a for loop through all of the elements in the reviews list and word-tokenized each element using NLTK’s sent_tokenize method. I also stored the outputs of the word-tokenization in the tokens variable.

The two following lines are where the data preparation magic really happens, as I utilize a combination of filter and lambda functions to remove both commonly occuring punctuation and stopwords from each tokens list.

In case you’re wondering why I chose to remove punctuation and stopwords on separate lines of code, I tried running this line-tokens: list(filter(lambda word: word not in '.!,’“”:...', tokens) | filter(lambda word: word.casefold() not in stopwordsList, tokens)) and it didn’t remove the punctuation or stopwords.
Yes, you’ll need to include the list wrapper in your code. Otherwise, the filter function will return a bunch of Filter class objects rather than the processed word-tokenized list (tokens).

After removing the punctuation and stopwords from the list, I noticed that there was a must- token (yes, with a dash) among the filtered tokens, so I added in a few lines of code to check for this token and remove it. Lastly, I then printed out all of the processed tokens (after the punctuation and stopwords have been removed).

Now for the fun part…the bag-of-words implementation!

Now that the data has been processed, it’s time for the fun part…implementing the bag-of-words algorithm! The first step in implementing the bag of words would be to create a vocab list of all the tokens (words) found in each of the reviews (and pay attention to the underlined lines of code):

stopwordsList = set(stopwords.words('english'))
vocab = []

for r in reviews:
    tokens = nltk.word_tokenize(r)                
    tokens = list(filter(lambda word: word not in '.!,’“”:...', tokens))
    tokens = list(filter(lambda word: word.casefold() not in stopwordsList, tokens))
        
    if 'must–' in tokens:
        tokens.remove('must–')
               
    for t in tokens:
        vocab.append(t)
        
vocab = list(set(vocab))

print(vocab)

['loss', 'milestone', 'powered', 'Challa', 'bidding', 'mechanics', 'triumphant', 'task', 'violence', 'spectacular', 'CGI', 'feat', 'lands', 'creating', 'fashion', 'allow', 'feels', 'stretches', 'starring', 'villainy', 'gods', 'movie', 'sober', 'Cinematic', 'felt', 'incredible', 'action', 'Chadwick', 'opening', 'affecting', 'get', '41mins', 'border', 'sequel', 'problem', 'Bassett', 'wake', 'note', 'spirit', 'done', 'succession', 'machine', 'Angela', 'loving', 'comic-book', 'dive', 'pulls', 'star', 'stirring', 'man', 'boredom', 'pays', 'first', 'prime', 'ache', 'taken', 'late', 'demand', 'Presented', 'fully', 'exercise', 'one', 'film', 'Panther', 'despite', 'sometimes', 'farewell', 'mind-blowing', 'Winston', 'blockbuster', 'weary', 'character', 'Marvel', 'meditation', 'Black', 'mourning', 'emotions', 'heartfelt', 'Coogler', 'MCU', 'emotionally', 'studio', 'closing', 'superhero', 'lost', 'Universe', '2hrs', 'Wakanda', 'inside', 'keenly', 'long', 'executed', 'also', 'films', 'sequences', 'core', 'vibrant', 'tribute', 'tragically', 'culturally', 'epic', 'Wow', 'audiences', 'urgent', 'emotional', 'soulful', 'over-the-top', 'vibranium', 'visual', 'teams', 'Duke', 'bric-a-brac', 'true', 'maturity', 'gracefully', 'Forever', 'right', 'mournful', 'oppressive', 'make', 'unfolds', 'weepy', 'given', 'Boseman', 'dark', 'story', 'dealing', 'say', 'heart', 'solidly', 'narrative', 'stumbles', 'politically', 'daunting', 'longest']

The new lines of code I added include initializing an empty vocab list that I will add the vocabulary list that I create from the tokens in each string in the reviews list.

I also added another for loop within the main for loop that iterates through each token in the tokens list and appends it to the vocab list. Once all the tokens from each review have been iterated through, I then run the list(set(...)) nested function to turn the vocab list into a set and back into a list before printing the vocab list.

Why do I turn the vocab list into a set? I wanted to remove all duplicate elements from the vocab list but still wanted to keep vocab as a list, so changing the vocab list to a set then back to a list was the easiest thing to do. Recall that sets in Python are like lists but with no duplicate elements.

It’s vectorization time!

Now that we have a vocab list ready, it’s time for vectorization!

What is vectorization though? In the context of the bag-of-words algorithm, vectorization utilizes a common vocabulary list-like the vocab list we just created-and based off of that common vocabulary list, creates a frequency count for each word (or combined phrase like we did here) by assigned a number to that word that indicates how many times that word appears in a document/string.

How would we implement vectorization? First off, and this part is completely optional, let’s sort the vocab list alphabetically:

vocabSorted = sorted(vocab)
print(vocabSorted)

['2hrs', '41mins', 'Angela Bassett', 'Black Panther', 'CGI', 'Chadwick Boseman', 'Coogler', 'MCU', 'Marvel Cinematic Universe', 'Presented', 'T`Challa', 'Wakanda Forever', 'Winston Duke', 'Wow', 'ache', 'action', 'affecting', 'allow', 'also', 'audiences', 'bidding', 'blockbuster', 'border', 'boredom', 'bric-a-brac', 'character', 'closing', 'comic-book', 'core', 'creating', 'culturally', 'dark', 'daunting', 'dealing', 'demand', 'despite', 'dive', 'done', 'emotional', 'emotionally', 'emotions', 'epic', 'executed', 'exercise', 'farewell', 'fashion', 'feat', 'feels', 'felt', 'film', 'films', 'first', 'fully', 'get', 'given', 'gods', 'gracefully', 'heart', 'heartfelt', 'incredible', 'inside', 'keenly', 'lands', 'late', 'long', 'longest', 'loss', 'lost', 'loving', 'machine', 'make', 'man', 'maturity', 'mechanics', 'meditation', 'milestone', 'mind-blowing', 'mournful', 'mourning', 'movie', 'narrative', 'note', 'one', 'opening', 'oppressive', 'over-the-top', 'pays', 'politically', 'powered', 'prime', 'problem', 'pulls', 'right', 'say', 'sequel', 'sequences', 'sober', 'solidly', 'sometimes', 'soulful', 'spectacular', 'spirit', 'star', 'starring', 'stirring', 'story', 'stretches', 'studio', 'stumbles', 'succession', 'superhero', 'taken', 'task', 'teams', 'tragically', 'tribute', 'triumphant', 'true', 'unfolds', 'urgent', 'vibranium', 'vibrant', 'villainy', 'violence', 'visual', 'wake', 'weary', 'weepy']

In order to sort the vocabulary list alphabetically, I used the sorted() function and passed in the vocab list as the function’s parameter. I also saved the sorted vocabulary list to the vocabSorted variable.

As you can see from the output above, the sorted() function will sort all of the uppercase strings in the list alphabetically before doing the same with the lowercase strings. That’s why the capitalized Wow is listed before the lowercase ache.

As I just said, it’s not required to sort the vocabulary list, but I just wanted to do it in order to make the vectorization process easier.

Now, how would we create the bag-of-words vectors for each string? Take a look at the code below:

wordVectorDict = {}

for t in tokensList:
    for v in vocabSorted:
        if v in t:
            wordVectorDict[v] = t.count(v)
        else:
            wordVectorDict[v] = 0
            
    print(wordVectorDict)
    print()

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 0, 'CGI': 0, 'Chadwick': 0, 'Challa': 0, 'Cinematic': 0, 'Coogler': 0, 'Duke': 0, 'Forever': 0, 'MCU': 0, 'Marvel': 0, 'Panther': 0, 'Presented': 0, 'Universe': 0, 'Wakanda': 0, 'Winston': 0, 'Wow': 2, 'ache': 0, 'action': 0, 'affecting': 0, 'allow': 0, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 0, 'despite': 0, 'dive': 0, 'done': 0, 'emotional': 0, 'emotionally': 0, 'emotions': 0, 'epic': 1, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 0, 'felt': 0, 'film': 0, 'films': 0, 'first': 0, 'fully': 0, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 0, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 0, 'long': 0, 'longest': 0, 'loss': 0, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 1, 'mournful': 0, 'mourning': 0, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 0, 'problem': 0, 'pulls': 0, 'right': 0, 'say': 1, 'sequel': 0, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 0, 'stretches': 0, 'studio': 0, 'stumbles': 0, 'succession': 0, 'superhero': 1, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 0, 'triumphant': 0, 'true': 0, 'unfolds': 1, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 0, 'weary': 0, 'weepy': 0}

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 0, 'CGI': 0, 'Chadwick': 0, 'Challa': 0, 'Cinematic': 0, 'Coogler': 0, 'Duke': 0, 'Forever': 0, 'MCU': 0, 'Marvel': 1, 'Panther': 0, 'Presented': 0, 'Universe': 0, 'Wakanda': 0, 'Winston': 0, 'Wow': 0, 'ache': 0, 'action': 0, 'affecting': 0, 'allow': 0, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 1, 'character': 1, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 1, 'despite': 0, 'dive': 0, 'done': 0, 'emotional': 0, 'emotionally': 0, 'emotions': 0, 'epic': 0, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 0, 'felt': 0, 'film': 1, 'films': 0, 'first': 0, 'fully': 0, 'get': 1, 'given': 0, 'gods': 1, 'gracefully': 0, 'heart': 0, 'heartfelt': 1, 'incredible': 0, 'inside': 1, 'keenly': 0, 'lands': 1, 'late': 0, 'long': 0, 'longest': 0, 'loss': 0, 'lost': 1, 'loving': 0, 'machine': 1, 'make': 0, 'man': 1, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 0, 'mournful': 0, 'mourning': 0, 'movie': 0, 'narrative': 1, 'note': 1, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 0, 'problem': 0, 'pulls': 0, 'right': 0, 'say': 0, 'sequel': 0, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 1, 'soulful': 0, 'spectacular': 0, 'spirit': 1, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 0, 'stretches': 0, 'studio': 0, 'stumbles': 0, 'succession': 1, 'superhero': 0, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 1, 'triumphant': 1, 'true': 0, 'unfolds': 0, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 0, 'weary': 0, 'weepy': 0}

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 0, 'CGI': 0, 'Chadwick': 0, 'Challa': 0, 'Cinematic': 0, 'Coogler': 0, 'Duke': 0, 'Forever': 0, 'MCU': 0, 'Marvel': 0, 'Panther': 0, 'Presented': 0, 'Universe': 0, 'Wakanda': 0, 'Winston': 0, 'Wow': 0, 'ache': 0, 'action': 0, 'affecting': 0, 'allow': 0, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 0, 'despite': 0, 'dive': 0, 'done': 1, 'emotional': 0, 'emotionally': 0, 'emotions': 0, 'epic': 0, 'executed': 0, 'exercise': 1, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 0, 'felt': 0, 'film': 0, 'films': 0, 'first': 0, 'fully': 0, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 0, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 0, 'long': 0, 'longest': 0, 'loss': 0, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 0, 'mournful': 0, 'mourning': 1, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 0, 'problem': 0, 'pulls': 0, 'right': 1, 'say': 0, 'sequel': 0, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 0, 'stretches': 0, 'studio': 0, 'stumbles': 0, 'succession': 0, 'superhero': 1, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 0, 'triumphant': 0, 'true': 0, 'unfolds': 0, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 0, 'weary': 0, 'weepy': 0}

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 0, 'CGI': 0, 'Chadwick': 0, 'Challa': 0, 'Cinematic': 0, 'Coogler': 0, 'Duke': 0, 'Forever': 0, 'MCU': 1, 'Marvel': 0, 'Panther': 0, 'Presented': 0, 'Universe': 0, 'Wakanda': 0, 'Winston': 0, 'Wow': 0, 'ache': 0, 'action': 0, 'affecting': 0, 'allow': 1, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 0, 'despite': 0, 'dive': 0, 'done': 0, 'emotional': 0, 'emotionally': 0, 'emotions': 0, 'epic': 0, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 0, 'felt': 0, 'film': 0, 'films': 0, 'first': 0, 'fully': 0, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 0, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 0, 'long': 0, 'longest': 0, 'loss': 0, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 0, 'mechanics': 1, 'meditation': 1, 'milestone': 0, 'mind-blowing': 0, 'mournful': 1, 'mourning': 0, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 1, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 0, 'problem': 0, 'pulls': 0, 'right': 0, 'say': 0, 'sequel': 0, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 0, 'stretches': 0, 'studio': 0, 'stumbles': 0, 'succession': 0, 'superhero': 0, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 0, 'triumphant': 0, 'true': 1, 'unfolds': 0, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 0, 'weary': 0, 'weepy': 0}

{'2hrs': 0, '41mins': 0, 'Angela': 1, 'Bassett': 1, 'Black': 0, 'Boseman': 1, 'CGI': 0, 'Chadwick': 1, 'Challa': 0, 'Cinematic': 1, 'Coogler': 0, 'Duke': 1, 'Forever': 0, 'MCU': 0, 'Marvel': 1, 'Panther': 0, 'Presented': 0, 'Universe': 1, 'Wakanda': 0, 'Winston': 1, 'Wow': 0, 'ache': 0, 'action': 1, 'affecting': 0, 'allow': 0, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 0, 'despite': 0, 'dive': 0, 'done': 0, 'emotional': 1, 'emotionally': 0, 'emotions': 0, 'epic': 0, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 0, 'felt': 0, 'film': 0, 'films': 0, 'first': 0, 'fully': 0, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 0, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 1, 'long': 0, 'longest': 0, 'loss': 0, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 1, 'mechanics': 0, 'meditation': 0, 'milestone': 1, 'mind-blowing': 0, 'mournful': 0, 'mourning': 0, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 0, 'problem': 0, 'pulls': 0, 'right': 0, 'say': 0, 'sequel': 1, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 1, 'spectacular': 1, 'spirit': 0, 'star': 1, 'starring': 1, 'stirring': 0, 'story': 0, 'stretches': 0, 'studio': 0, 'stumbles': 0, 'succession': 0, 'superhero': 0, 'taken': 0, 'task': 0, 'teams': 1, 'tragically': 0, 'tribute': 1, 'triumphant': 0, 'true': 0, 'unfolds': 0, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 1, 'wake': 0, 'weary': 0, 'weepy': 0}

{'2hrs': 1, '41mins': 1, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 0, 'CGI': 0, 'Chadwick': 0, 'Challa': 0, 'Cinematic': 0, 'Coogler': 0, 'Duke': 0, 'Forever': 1, 'MCU': 1, 'Marvel': 0, 'Panther': 0, 'Presented': 0, 'Universe': 0, 'Wakanda': 1, 'Winston': 0, 'Wow': 0, 'ache': 1, 'action': 0, 'affecting': 0, 'allow': 0, 'also': 2, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 1, 'boredom': 1, 'bric-a-brac': 0, 'character': 0, 'closing': 1, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 0, 'despite': 0, 'dive': 0, 'done': 0, 'emotional': 0, 'emotionally': 0, 'emotions': 0, 'epic': 0, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 0, 'felt': 0, 'film': 0, 'films': 1, 'first': 0, 'fully': 0, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 1, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 0, 'long': 1, 'longest': 1, 'loss': 0, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 1, 'man': 0, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 0, 'mournful': 0, 'mourning': 0, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 1, 'opening': 1, 'oppressive': 0, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 0, 'problem': 0, 'pulls': 0, 'right': 0, 'say': 0, 'sequel': 0, 'sequences': 1, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 0, 'stretches': 1, 'studio': 0, 'stumbles': 0, 'succession': 0, 'superhero': 0, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 0, 'triumphant': 0, 'true': 0, 'unfolds': 0, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 0, 'weary': 1, 'weepy': 1}

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 1, 'CGI': 0, 'Chadwick': 1, 'Challa': 1, 'Cinematic': 0, 'Coogler': 1, 'Duke': 0, 'Forever': 0, 'MCU': 0, 'Marvel': 0, 'Panther': 0, 'Presented': 0, 'Universe': 0, 'Wakanda': 0, 'Winston': 0, 'Wow': 0, 'ache': 0, 'action': 0, 'affecting': 1, 'allow': 0, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 1, 'culturally': 1, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 0, 'despite': 1, 'dive': 0, 'done': 0, 'emotional': 0, 'emotionally': 1, 'emotions': 0, 'epic': 0, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 1, 'feels': 0, 'felt': 0, 'film': 1, 'films': 0, 'first': 0, 'fully': 0, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 0, 'heartfelt': 0, 'incredible': 1, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 0, 'long': 0, 'longest': 0, 'loss': 0, 'lost': 0, 'loving': 1, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 0, 'mournful': 0, 'mourning': 0, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 0, 'pays': 1, 'politically': 1, 'powered': 0, 'prime': 0, 'problem': 0, 'pulls': 1, 'right': 0, 'say': 0, 'sequel': 0, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 1, 'stretches': 0, 'studio': 0, 'stumbles': 1, 'succession': 0, 'superhero': 1, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 1, 'triumphant': 0, 'true': 0, 'unfolds': 0, 'urgent': 1, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 0, 'weary': 0, 'weepy': 0}

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 0, 'CGI': 0, 'Chadwick': 0, 'Challa': 0, 'Cinematic': 0, 'Coogler': 0, 'Duke': 0, 'Forever': 1, 'MCU': 0, 'Marvel': 0, 'Panther': 0, 'Presented': 0, 'Universe': 0, 'Wakanda': 1, 'Winston': 0, 'Wow': 0, 'ache': 0, 'action': 0, 'affecting': 0, 'allow': 0, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 1, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 0, 'despite': 0, 'dive': 0, 'done': 0, 'emotional': 0, 'emotionally': 0, 'emotions': 1, 'epic': 0, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 0, 'felt': 1, 'film': 0, 'films': 0, 'first': 1, 'fully': 1, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 0, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 0, 'long': 0, 'longest': 0, 'loss': 0, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 0, 'mournful': 0, 'mourning': 0, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 1, 'prime': 0, 'problem': 0, 'pulls': 0, 'right': 0, 'say': 0, 'sequel': 0, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 0, 'stretches': 0, 'studio': 0, 'stumbles': 0, 'succession': 0, 'superhero': 0, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 0, 'triumphant': 0, 'true': 0, 'unfolds': 0, 'urgent': 0, 'vibranium': 1, 'vibrant': 1, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 1, 'weary': 0, 'weepy': 0}

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 0, 'CGI': 1, 'Chadwick': 0, 'Challa': 0, 'Cinematic': 0, 'Coogler': 0, 'Duke': 0, 'Forever': 0, 'MCU': 0, 'Marvel': 0, 'Panther': 0, 'Presented': 0, 'Universe': 0, 'Wakanda': 0, 'Winston': 0, 'Wow': 0, 'ache': 0, 'action': 0, 'affecting': 0, 'allow': 0, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 1, 'core': 1, 'creating': 0, 'culturally': 0, 'dark': 1, 'daunting': 0, 'dealing': 1, 'demand': 0, 'despite': 0, 'dive': 0, 'done': 0, 'emotional': 0, 'emotionally': 0, 'emotions': 0, 'epic': 0, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 0, 'felt': 0, 'film': 1, 'films': 0, 'first': 0, 'fully': 0, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 0, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 0, 'long': 0, 'longest': 0, 'loss': 1, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 0, 'mournful': 0, 'mourning': 0, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 1, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 0, 'problem': 0, 'pulls': 0, 'right': 0, 'say': 0, 'sequel': 0, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 0, 'stretches': 0, 'studio': 0, 'stumbles': 0, 'succession': 0, 'superhero': 0, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 0, 'triumphant': 0, 'true': 0, 'unfolds': 0, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 1, 'violence': 1, 'visual': 0, 'wake': 0, 'weary': 0, 'weepy': 0}

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 1, 'Boseman': 1, 'CGI': 0, 'Chadwick': 1, 'Challa': 0, 'Cinematic': 0, 'Coogler': 0, 'Duke': 0, 'Forever': 1, 'MCU': 0, 'Marvel': 0, 'Panther': 1, 'Presented': 0, 'Universe': 0, 'Wakanda': 1, 'Winston': 0, 'Wow': 0, 'ache': 0, 'action': 0, 'affecting': 0, 'allow': 0, 'also': 0, 'audiences': 0, 'bidding': 0, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 0, 'dealing': 0, 'demand': 0, 'despite': 0, 'dive': 0, 'done': 0, 'emotional': 0, 'emotionally': 0, 'emotions': 0, 'epic': 0, 'executed': 0, 'exercise': 0, 'farewell': 0, 'fashion': 0, 'feat': 0, 'feels': 1, 'felt': 0, 'film': 0, 'films': 0, 'first': 0, 'fully': 0, 'get': 0, 'given': 0, 'gods': 0, 'gracefully': 0, 'heart': 0, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 1, 'lands': 0, 'late': 1, 'long': 0, 'longest': 0, 'loss': 1, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 0, 'mournful': 0, 'mourning': 0, 'movie': 1, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 0, 'problem': 1, 'pulls': 0, 'right': 0, 'say': 0, 'sequel': 0, 'sequences': 0, 'sober': 0, 'solidly': 0, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 0, 'starring': 0, 'stirring': 0, 'story': 0, 'stretches': 0, 'studio': 0, 'stumbles': 0, 'succession': 0, 'superhero': 0, 'taken': 0, 'task': 0, 'teams': 0, 'tragically': 0, 'tribute': 1, 'triumphant': 0, 'true': 0, 'unfolds': 0, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 0, 'weary': 0, 'weepy': 0}

{'2hrs': 0, '41mins': 0, 'Angela': 0, 'Bassett': 0, 'Black': 0, 'Boseman': 0, 'CGI': 0, 'Chadwick': 0, 'Challa': 0, 'Cinematic': 0, 'Coogler': 1, 'Duke': 0, 'Forever': 0, 'MCU': 0, 'Marvel': 0, 'Panther': 0, 'Presented': 1, 'Universe': 0, 'Wakanda': 1, 'Winston': 0, 'Wow': 0, 'ache': 0, 'action': 0, 'affecting': 0, 'allow': 0, 'also': 0, 'audiences': 1, 'bidding': 1, 'blockbuster': 0, 'border': 0, 'boredom': 0, 'bric-a-brac': 0, 'character': 0, 'closing': 0, 'comic-book': 0, 'core': 0, 'creating': 0, 'culturally': 0, 'dark': 0, 'daunting': 1, 'dealing': 0, 'demand': 0, 'despite': 0, 'dive': 1, 'done': 0, 'emotional': 0, 'emotionally': 0, 'emotions': 0, 'epic': 0, 'executed': 1, 'exercise': 0, 'farewell': 1, 'fashion': 1, 'feat': 0, 'feels': 0, 'felt': 0, 'film': 0, 'films': 0, 'first': 0, 'fully': 0, 'get': 0, 'given': 1, 'gods': 0, 'gracefully': 1, 'heart': 0, 'heartfelt': 0, 'incredible': 0, 'inside': 0, 'keenly': 0, 'lands': 0, 'late': 0, 'long': 0, 'longest': 0, 'loss': 0, 'lost': 0, 'loving': 0, 'machine': 0, 'make': 0, 'man': 0, 'maturity': 0, 'mechanics': 0, 'meditation': 0, 'milestone': 0, 'mind-blowing': 0, 'mournful': 0, 'mourning': 0, 'movie': 0, 'narrative': 0, 'note': 0, 'one': 0, 'opening': 0, 'oppressive': 0, 'over-the-top': 0, 'pays': 0, 'politically': 0, 'powered': 0, 'prime': 1, 'problem': 0, 'pulls': 0, 'right': 1, 'say': 0, 'sequel': 0, 'sequences': 0, 'sober': 1, 'solidly': 1, 'sometimes': 0, 'soulful': 0, 'spectacular': 0, 'spirit': 0, 'star': 1, 'starring': 0, 'stirring': 1, 'story': 0, 'stretches': 0, 'studio': 1, 'stumbles': 0, 'succession': 0, 'superhero': 0, 'taken': 1, 'task': 1, 'teams': 0, 'tragically': 1, 'tribute': 0, 'triumphant': 0, 'true': 0, 'unfolds': 0, 'urgent': 0, 'vibranium': 0, 'vibrant': 0, 'villainy': 0, 'violence': 0, 'visual': 0, 'wake': 0, 'weary': 0, 'weepy': 0}

In this example, I created a wordVectorDict dictionary, which I’ll use to create the word vectors for each element in the tokensList.

After creating the wordVectorDict dictionary, I then run a for loop through the tokensList and run a nested for loop through the vocabSorted list (you can simply use the vocab list if you chose not to sort the vocabulary). As for the wordVectorDict dictionary, each of the elements in the vocabSorted list serve as keys while the count of each element in a processed review string serves as the corresponding values. For instance, in the first review, the word Wow is used twice, so the key-value pair for the word Wow in the first wordVectorDict would be Wow: 2. If an element in the sortedVocab list doesn’t appear in a processed review string, the corresponding value to the vocabulary key would be 0. For instance, since the word farewell doesn’t appear in the first review, its key-value pair would be farewell: 0.

As you could probably guess from my code, I created 11 wordVectorDict dictionaries, one for each element in the tokensList, and printed them all out so you can see what each word vector will eventually look like (more on that later).

Creating the word vectors

Now that we’ve got an idea as to the token count for each processed review, it’s time to create the word vectors! How would we do so? Take a look at the underlined lines of code to see one approach to creating the word vectors:

import numpy as np

wordVectorDict = {}
wordVector = []

for t in tokensList:
    for v in vocabSorted:
        if v in t:
            wordVectorDict[v] = t.count(v)
        else:
            wordVectorDict[v] = 0
        
    wordVector = np.array(list(wordVectorDict.values()))
    print(wordVector)

[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0
 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 1 0 0 1 0 1 0 1 0 0 0 0
 1 0 1 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0
 0 0 0 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0]
[0 0 1 1 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0
 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 1 1
 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 0]
[1 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0 2 0 0 0 1 1 0 0 1 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0
 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1]
[0 0 0 0 0 1 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1
 1 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0
 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
 0 1 0 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0]
[0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0
 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0]
[0 0 0 0 1 1 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1
 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0
 0 0 1 0 0 0 1 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 1 0 0 0 0 1 0
 1 0 0 1 0 0 0 1 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0]

To create the word vector lists, I simply grabbed the values from all of the wordVectorDict elements, placed them into a numpy array, and printed each array.

Yes, you will need to install and import numpy for this example.

As you can see from the output, most of the elements in each numpy array are zeroes and ones with a handful of twos-indicating that many of the tokens in the vocabSorted list only appear in each string once or not at all.

Presenting our bag-of-words

Now that we’ve created our word vector for each processed element in the reviews list, it’s time to figure out how to best present the data. Take a look at the code below (and pay attention to the underlined lines of code):

import numpy as np
import pandas as pd

wordVectorDict = {}
wordVector = []
bagOfWords = pd.DataFrame()
wordVectorList = []

for t in tokensList:
    for v in vocabSorted:
        if v in t:
            wordVectorDict[v] = t.count(v)
        else:
            wordVectorDict[v] = 0
        
    wordVector = np.array(list(wordVectorDict.values()))
    
    wordVectorList.append(wordVector)
    
    bagOfWords = pd.DataFrame(wordVectorList)
    
bagOfWords

In this example, I created a pandas data-frame (appropriately called bagOfWords) for all 11 wordVectors that shows you how many times a token appears in a particular string. I used the wordVectorList variable to gather all 11 wordVector elements into a single list; creating the wordVectorList made it easier to create the data-frame.

The 0-row corresponds to the first element in reviews whereas the 10-row would correspond to the 11th and final element in reviews.

So, data-frame is looking pretty good, right? There’s just one issue-you can’t tell which tokens are which just by the headers (though granted, this data-frame does utilize the vocabSorted list, so it’ll take you some time to figure out which token corresponds to which index).

How can we fix this issue? There’s just one tiny change in the code above that you’ll need to make. Can you guess what that would be?

import numpy as np
import pandas as pd

wordVectorDict = {}
wordVector = []
bagOfWords = pd.DataFrame()
wordVectorList = []

for t in tokensList:
    for v in vocabSorted:
        if v in t:
            wordVectorDict[v] = t.count(v)
        else:
            wordVectorDict[v] = 0
        
    wordVector = np.array(list(wordVectorDict.values()))
    
    wordVectorList.append(wordVectorDict)
    
    bagOfWords = pd.DataFrame(wordVectorList, columns=vocabSorted)
    
bagOfWords

The small change I made in the code is to add the columns = vocabSorted line to the pd.DataFrame() function and just like that, the indeces of each element in the vocabSorted list is replaced with the token itself, making it much easier to tell where all the ones and zeroes connect to.

Thanks for reading!

Michael

Python Lesson 39: One Simple Way To Improve Your Neural Network’s Accuracy (AI pt. 3)

Hello everybody,

Michael here, and hope you all had a wonderful holiday season. I’ve got lots of exciting content planned for 2023-including something special for the blog’s 5th anniversary (yup, this blog turns 5 on June 13)-and I hope you all will follow along on this programming journey.

To start the year, I thought I’d pick up where I left off in 2022. If you recall, the last post I wrote in 2022 involved creating a basic neural network in Python using the famous MNIST dataset-Python Lesson 38: Building Your First Neural Network (AI pt. 2). In that post, you’ll also likely recall that the neural network we built had an accuracy of less than 20%. In this post, we’ll explore a simple way to improve that neural network’s accuracy. Let’s get coding!

A little refresher on our previous project

In case you’d like to see it again, here’s our code for the neural network project we made in the previous post:

import tensorflow as tf
import keras as kr
import tensorflow_datasets as tfds

(trainX, trainY), (testX, testY) = mnist.load_data()

trainX.shape
testX.shape
trainY.shape
testY.shape

import matplotlib.pyplot as plt
imageNum = 1500
plt.imshow(trainX[imageNum], cmap='magma')

import matplotlib.pyplot as plt
imageNum = 3332
plt.imshow(testX[imageNum], cmap='magma')

firstNeuralNetwork = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28,28)),
    tf.keras.layers.Dense(150, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10)
])

firstNeuralNetwork.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
firstNeuralNetwork.fit(x=trainX,y=trainY, epochs=25)

firstNeuralNetwork.evaluate(testX, testY)

To recap, in this code, we built a basic neural network in Python to classify handwritten digits in the MNIST dataset and as I mentioned earlier, this model wasn’t very accurate. In fact, we didn’t acheive accuracy higher than 20% through any of the iterations. Let’s explore some ways we can change that.

One simple way to improve the neural network’s accuracy

Pay attention to this line of code-it creates the second Dense layer in our neural network (the layer that must have ten neurons in this example):

tf.keras.layers.Dense(10)

Similar to what we did for the first Dense layer, add an activation parameter when creating this dense layer (after the number 10). However, this time, set the value of the activation parameter to softmax, like so:

tf.keras.layers.Dense(10, activation='softmax')

You’re likely wondering, what is the softmax function? Here’s an easy way to explain it. Imagine you’re arranging a summertime trip and have four choices of departure dates-June 30, July 1, July 3, and July 5. Let’s say you wanted to use the softmax function to decide a departure date.

The way the softmax function works is that it takes the four aforementioned dates and assigns random probabilities to each of them-the sum of these four probabilites will equal 1 (essentially, we’re dividing the group of possible departure dates into four parts). In this example, let’s say the four probabilities assigned were 46% (for June 30), 20% (for July 1), 19% (for July 3), and 15% (for July 5). All of these probabilites add up to 1-or 100%.

Now that we’ve explained the softmax function, let’s see how it helps improve our neural networks accuracy without changing anything else in the code.

First, let’s see how the accuracy for each epoch is affected:

firstNeuralNetwork.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
firstNeuralNetwork.fit(x=trainX,y=trainY, epochs=25)

Epoch 1/25
1875/1875 [==============================] - 5s 2ms/step - loss: 2.4216 - accuracy: 0.7781
Epoch 2/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.5508 - accuracy: 0.8612
Epoch 3/25
1875/1875 [==============================] - 5s 3ms/step - loss: 0.4453 - accuracy: 0.8877
Epoch 4/25
1875/1875 [==============================] - 5s 3ms/step - loss: 0.3841 - accuracy: 0.9004
Epoch 5/25
1875/1875 [==============================] - 5s 2ms/step - loss: 0.3730 - accuracy: 0.9069
Epoch 6/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3482 - accuracy: 0.9123
Epoch 7/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3343 - accuracy: 0.9167
Epoch 8/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3250 - accuracy: 0.9178
Epoch 9/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3182 - accuracy: 0.9224
Epoch 10/25
1875/1875 [==============================] - 5s 2ms/step - loss: 0.3103 - accuracy: 0.9238
Epoch 11/25
1875/1875 [==============================] - 5s 2ms/step - loss: 0.3041 - accuracy: 0.9251
Epoch 12/25
1875/1875 [==============================] - 5s 2ms/step - loss: 0.3022 - accuracy: 0.9258
Epoch 13/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2983 - accuracy: 0.9280
Epoch 14/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2962 - accuracy: 0.9288
Epoch 15/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2832 - accuracy: 0.9320
Epoch 16/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2904 - accuracy: 0.9321
Epoch 17/25
1875/1875 [==============================] - 5s 3ms/step - loss: 0.2861 - accuracy: 0.9308
Epoch 18/25
1875/1875 [==============================] - 5s 3ms/step - loss: 0.2805 - accuracy: 0.9337
Epoch 19/25
1875/1875 [==============================] - 5s 2ms/step - loss: 0.2859 - accuracy: 0.9334
Epoch 20/25
1875/1875 [==============================] - 5s 3ms/step - loss: 0.2775 - accuracy: 0.9365
Epoch 21/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2800 - accuracy: 0.9346
Epoch 22/25
1875/1875 [==============================] - 4s 2ms/step - loss: 0.2825 - accuracy: 0.9371
Epoch 23/25
1875/1875 [==============================] - 5s 3ms/step - loss: 0.2743 - accuracy: 0.9370
Epoch 24/25
1875/1875 [==============================] - 6s 3ms/step - loss: 0.2749 - accuracy: 0.9383
Epoch 25/25
1875/1875 [==============================] - 5s 3ms/step - loss: 0.2703 - accuracy: 0.9372

Well, that’s a significant improvement from the per-epoch accuracy from the previous post! I mean, 77.8% accuracy on just the first epoch is quite impressive-and by the 25th and last epoch-the model achieves 93.7% accuracy.

Now, let’s check out the overall accuracy of the model:

firstNeuralNetwork.evaluate(testX, testY)

313/313 [==============================] - 1s 2ms/step - loss: 0.4758 - accuracy: 0.9471

94.7% overall accruacy-all by adding a simple line of code! If you recall from the previous post, our model’s overall accuracy was just 10.5%.

Thanks for reading, and I can’t wait to share all of the exiciting programming content I have planned for you all in 2023!

Also, if there’s anything you can take away from this lesson, it’s that sometimes the smallest code changes can make a big difference in your program.

Python Lesson 38: Building Your First Neural Network (AI pt. 2)

Hello everybody,

Michael here, and in today’s post-my last post of 2022-I will be showing you how to create your first neural network in Python. I know you haven’t seen stuff like this on my blog before, but I thought I’d end the year teaching you all something new.

Now, there are two possible ways you can create a neural network in Python-one of which involves creating the framework for your neural network by scratch with a combination of classes and functions (which, if you readers want, I’ll cover how to do this). The other way involves using two of Python’s built-in packages-Keras and TensorFlow-which I will discuss more in this post.

A little bit about Keras and Tensorflow

Tensorflow and Keras are two prominent Python neural network machine learning packages. However, Tensorflow is an entire open-source end-to-end neural network package while Keras is more like an interface within Tensorflow. If it helps, think of Keras like a package-within-a-package in Tensorflow; whenever you use Keras, you’re actually using the Tensorflow library. However, Keras is a more intutive version of the Tensorflow libary, albeit with some trade-offs (such as the lack of ability to access more complex functionalities).

Package installation

Before we get started with our neural network creation, let’s first install our packages. You’re going to need both Tensorflow and Keras for this tutorial, but you only need to run the pip install tensorflow command on the command prompt, as installing Tensorflow will usually install Keras too. However, on the off chance that Keras doesn’t get installed with Tensorflow, you could run the pip install keras command on the command prompt.

Just in case you forgot, if you want to check if you’ve already pip-installed a certain package, run the pip list command and run through the list of installed packages to find the package you’re looking for (all packages are listed in alphabetical order).

Setting up the neural network

For this lesson, we’re going to start off by building a simple neural network-one that works with the MNIST Keras dataset. For those who don’t know, the MNIST (Modified National Institute of Standards and Technology) dataset is a very, very large dataset of images containing the handwritten digits 0-9-the MNIST dataset is commonly used for training image processing systems (or if you’re just starting out with neural network machine learning). This dataset contains 70,000 28×28 pixel images-60,000 images for the training dataset and 10,000 images for the testing dataset.

The MNIST dataset is certainly larger than most of the other datasets we’ve worked with in earlier posts (if you recall, the datasets from my earlier machine learning posts had about a few thousand elements tops). The reason for this is because, unlike the other machine learning I’ve taught you (k-means clustering, Naive Bayes classifications), neural networks are really well-suited for large datasets-and by large, I mean at least 10,000 records.

To start creating our neural network, first include these three lines of code in your Jupyter notebook:

import tensorflow as tf
import keras as kr
import tensorflow_datasets as tfds

Pay attention to the highlighted import line-in addition to the Tensorflow and Keras packages, you’ll also need the tensorflow_datasets package for this lesson. The tensorflow_datasets package contains several Tensorflow datasets you can work with when developing neural networks (such as the MNIST dataset, which we will be working with in this lesson).

If you haven’t installed the tensorflow_datasets package yet, run the line pip install tensorflow_datasets on your command prompt or run the line !pip install tensorflow_datasets on your Jupyter notebook (or whichever IDE you’re using).

Loading the MNIST dataset (and a word of advice)

Now that we’ve imported the necessary packages into our Python IDE, the next thing we need to do is import the MNIST dataset into our IDE. Here’s the code to do so:

from keras.datasets import mnist

Unlike most of my other machine learning/data analytics posts, I won’t be attaching a dataset to this post because we’ll be using a built-in Python dataset for this post. If you’re familiar with some popular data analytics/machine learning datasets such as titanic (detailing survivors and victims of the Titanic disaster), iris (detailing petal and sepal widths of a sample of 50 irises), and mtcars (detailing various features about a bunch of old cars), you’ve probably seen them on A LOT of data analytics/machine learning tutorials. There’s a good reason for that-they’re freely available and built-in datasets on several programs (Python and R to name just two).

For those who’ve been following my blog for a while, you’ll notice that I try to stay away from overly cliche datasets (I mean, if you’re a data science/data anayltics machine learning student, you’re probably quite sick of the iris dataset). However, even though MNIST is a very commonly used (and a little cliche) dataset, I think it will be the most appropriate first dataset to introduce you all to neural network creation.

Also, final word of advice for you all-if you’re trying to build a data science/data analytics/machine learning portfolio to land yourself a tech job (as I did when I launched this blog in summer 2018), try to stay away from cliche datasets. Find datasets that stand out (and ideally interest you)-you’ll be sure to impress the recruiters!

Now back to the lesson! After importing the MNIST dataset into your IDE, run this line of code to split the MNIST dataset into training and testing datasets:

(trainX, trainY), (testX, testY) = mnist.load_data()

When loading the MNIST dataset into your IDE (or any large dataset for that matter), remember to split your dataset into training and testing datasets, each denoted by their own variables.

I know it’s been a while since I’ve done any machine learning posts, so as a refresher, when building a machine learning model, the training dataset trains the model to work while the testing dataset is used to test if the model works as intended. When working with machine learning datasets, don’t split the main dataset 50-50 into training and testing datasets. The training dataset should be the larger dataset; a split like 70% training/30% testing should work fine-though the MNIST dataset has a split of ~85% training/~15% testing, which will work for this dataset.

Why do we need X and Y training and testing datasets? The X datasets encompass the whole dimensions of the training and testing datasets-the size (60,000 for training and 10,000 for testing) along with the dimensions of each image (28×28 pixels). The Y datasets on the other hand just encompass the sizes of each dataset.

In case you’re wondering about the size of each X and Y dataset, run the .shape command for each like so-remember not to include a pair of parentheses after each .shape command, as you can’t call tuple objects:

trainX.shape
(60000, 28, 28)

testX.shape
(10000, 28, 28)

trainY.shape
(60000,)

testY.shape
(10000,)

And now…time to build the model!

Now that we’ve loaded our MNIST dataset into Python, split the data into training and testing datasets, and obtained the shapes of each dataset, it’s time to get our feet wet and build our first neural network!

However, before we dive into the neural network nitty-gritty, there’s something I want to show you. Take a look at the code and output below:

import matplotlib.pyplot as plt
imageNum = 1500
plt.imshow(trainX[imageNum], cmap='magma')

In this example, I imported the matplotlib.pyplot package (which you may recall from my MATPLOTLIB lessons) to plot the 1501st image in the MNIST training dataset in MATPLOTLIB’s magma color scheme (the cmap parameter refers to MATPLOTLIB’s color schemes). As you can see, this image of a handwritten 9 is displayed as a 28×28 pixel image-which makes sense, as all images in the MNIST dataset (both training and testing) have a 28×28 pixel size.

MATPLOTLIB has several different color schemes to choose from. For a list of all available color scheme choices, check out this link-https://matplotlib.org/stable/tutorials/colors/colormaps.html.
In order to plot any of the images in the MNIST dataset, you’ll need to use either of the X datasets (in this example, trainX and testX) since they encompass the image sizes and in turn, contain the actual images. The Y datasets simply encompass the images themselves, so you would be able to retrieve any element from the MNIST dataset from either of the Y datasets, but you won’t be able to plot the image itself.
Just like many of the other Python projects I’ve done throughout this blog involving lists, the MNIST dataset is basically a giant zero-indexed list of images. So for a parameter like imageNum, you can choose any value between 0 and 59,999 if you’re analyzing the 60,000 image training dataset. If you’re analyzing the 10,000 image testing dataset, you can choose any value between 0 and 9,999. In the example above, I chose the 1,501st image in the testing dataset (as the imageNum I chose was 1,500, which represents the element at index 1,500).

Just for fun, let’s also plot a random image from the testing dataset:

import matplotlib.pyplot as plt
imageNum = 3332
plt.imshow(testX[imageNum], cmap='magma')

In this example, I did the same thing as I did in the previous example, except I decided to plot the 3,333rd image from the MNIST testing dataset-which happens to be the number 4.

Now that we know how to plot each element in the MNIST dataset (for both the testing and training datasets) it’s time to create our model! Take a look at the code below to see how we can create our first Python neural network model:

firstNeuralNetwork = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28,28)),
    tf.keras.layers.Dense(150, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10)
])

Now, if you’ve never seen a Python neural network before, you’re probably wondering what all of this code means. But don’t worry-your friendly neighborhood coding blogger is here to break it all down for you!

First off, let’s start with the Sequential sub-module. We use this sub-module in order to create the outer part of the neural network; in this sub-module, we wrap all the functions for the neural network inside of a list wrapped inside of the Sequential object constructor (referrring to the pair of parentheses that enclose the list). Why do we need a sequential model for the neural network? In this example, using a sequential model for the neural network allows us to add the other four layers in this neural network-Flatten, Dense, Dropout, and Dense-in sequential order, which is important for neural networks.

Now what about the four layers wrapped in our sequential model-Flatten, Dropout and the two Dense layers? The Flatten layer, well, flattens the input from 2-dimensional to 1-dimensional-which is important as we’re dealing with thousands of 2-D images for this dataset. How does Flatten flatten the input data? The Flatten layer’s input_shape parameter takes in the dimensions of the object to flatten-in this case each 28×28 image in the MNIST dataset-and takes in the (28, 28) tuple as the value of the input_shape argument.

The Dropout layer removes some of the data from the model in order to prevent overfitting. In the context of machine learning, what is overfitting? Overfitting in machine learning is what happens when your model has excellent accuracy with training data but not with new and unfamiliar data.

Let me give you an example. Let’s say you want to create a model that predicts whether an employee at a very, very, very large company is going to get a promotion based off of their resume. Let’s also assume that you train a model containing 5,000 resumes and it predicts outcomes with 96% accuracy-pretty awesome, right! Now let’s say you feed the model a new set of 2,500 resumes and it predicts outcomes with only a 44% accuracy-what happened here? The model experienced overfitting, as it was able to predict outcomes with great accuracy for the training dataset but with less-than-stellar accuracy for the new and unfamiliar dataset.

In our neural network, the Dropout layer will ignore 10% of the data in the training dataset to avoid overfitting.

Last but not least, we have two Dense layers for our neural network. The first Dense layer activates the neural network using the ReLU, or rectified linear unit activation, function. For more on the algebra behind ReLU, check out this article-https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/ (if you’re into linear algebra and/or trigonometry, I think you’ll enjoy this article). In the most basic sense, ReLU is a linear activation function that is used in a lot of neural networks due to its easy-to-train and well-performance.

In the first Dense layer, you’ll notice a number right before the activation parameter-that number indicates how many neurons you want to have in the neural network upon activation; in this case, we have 150 neurons upon activation of our neural network. The second Dense layer also has a number too-10. What’s the difference between these two numbers? In the first Dense layer, you can have as many neurons as you’d like upon activation while in the second Dense layer, you must have 10 neurons as there are ten unique objects for classifcation (images of the numbers 0-9).

Fitting and Compiling the Model

The last two things we need to do before we deploy our model are to fit it and compile it. How can we do that? Take a look at the code below:

firstNeuralNetwork.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
firstNeuralNetwork.fit(x=trainX,y=trainY, epochs=25)

So, what does all of this code mean? First of all, the optimizer parameter and value set the neural network’s optimization alogrithm-in this case, we’re using Tensorflow’s adam optimizer (for a more in-depth explination on that optimizer, check out this link-https://www.educba.com/tensorflow-adam-optimizer/), though you can experiement with whatever Tensorflow optimizer you like.

The loss parameter and corresponding value set the neural network’s loss function, which is used to help optimize the model’s performance by measuring the discrepancies between the predicted values and the target values. In the context of the MNIST dataset, each element would be considered a target value and the value that the neural network predicts as part of its classification would be the target value. In this example, we’re using the sparse_categorical_crossentropy loss function, which measures the cross-entropy (or contrast or discrepancy) between the predicted values and the actual values.

The metrics parameter and corresponding value (or list in this case) set the metrics-or in this case, metric-that you’d like to use to measure the neural network’s accuracy. In this example, we’re going with the accuracy metric, as this is the easiest metric to understand. Accuracy is also often used as a baseline for other metrics such as precision and f1 score (which is similar to accuracy but it takes false positives and false negatives into account).

In the fit function, you’ll first need to pass in your training datasets for both the X and Y values. As for the epoch parameter and value, an epoch is essentially an iteration through all the training data that isn’t ignored by the Dropout layer. To train a neural network and optimize it for accuracy, iterating through all of the training data once won’t suffice-you’ll need at least 10 iterations through the training data to optimize your neural network (though more epochs couldn’t hurt). In this neural network, we’re using 25 epochs, meaning that we will iterate through the training data 25 times.

Now, let’s see how our neural network performs through each epoch (or iteration):

Epoch 1/25
1875/1875 [==============================] - 5s 2ms/step - loss: 2.3026 - accuracy: 0.1118
Epoch 2/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3028 - accuracy: 0.1137
Epoch 3/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1118
Epoch 4/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3032 - accuracy: 0.1124
Epoch 5/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1114
Epoch 6/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3032 - accuracy: 0.1118
Epoch 7/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1100
Epoch 8/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1125
Epoch 9/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3028 - accuracy: 0.1114
Epoch 10/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3027 - accuracy: 0.1107
Epoch 11/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3028 - accuracy: 0.1129
Epoch 12/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3026 - accuracy: 0.1113
Epoch 13/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3028 - accuracy: 0.1135
Epoch 14/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3032 - accuracy: 0.1124
Epoch 15/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1133
Epoch 16/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3026 - accuracy: 0.1121
Epoch 17/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1124
Epoch 18/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1133
Epoch 19/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1120
Epoch 20/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3028 - accuracy: 0.1134
Epoch 21/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3026 - accuracy: 0.1141
Epoch 22/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3026 - accuracy: 0.1129
Epoch 23/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1126
Epoch 24/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3030 - accuracy: 0.1127
Epoch 25/25
1875/1875 [==============================] - 4s 2ms/step - loss: 2.3037 - accuracy: 0.1117

In this epoch run log, we can see several different metrics for each epoch, such as loss and accuracy. However, the only metric you should focus on is each epoch’s accuracy, as that tells you the accuracy of the neural network throughout each training run. For instance, the first epoch (denoted as Epoch 1/25) had an accuracy of 11.18%. The final epoch (denoted as Epoch 25/25) had an accuracy of 11.17%-all in all, pretty abysmal accruacy for the neural network.

Neural network evaluation time!

Last but not least, it’s neural network evaluation time! To evaluate the accuracy of the overall model (as opposed to individual epochs), all you need is one line of code:

firstNeuralNetwork.evaluate(testX, testY)

313/313 [==============================] - 1s 1ms/step - loss: 2.3026 - accuracy: 0.1045

Just like you saw with the epochs, you’ll see the loss and accuracy metrics. Pay close attention to the accuracy metric, as this will tell you the model’s overall accuracy, which is still pretty bad at 10.45%.

I know this may seem confusing, but remember when you’re fitting & compiling the model to use the training dataset (for both the X and Y axes). When you’re evaluating the model’s accuracy, use the testing dataset (for both the X and Y axes).

Yes, I know the accuracy of this neural network sucked. However, the aim of this lesson was not to build the best neural network out there-rather, my aim was to teach you the basics of neural network creation so that you all knew the basic concepts of neural networks. A lot of the concepts we discussed in this post-activation algorithms, epochs, dropout rate-can be experimented with to your liking in order to optimize the neural network’s accuracy.

Final code and some parting words for 2022

So, I know we had A LOT of code for this lesson. In case you wanted to run the code in the order we discussed it, here’s the entire script below for your convinience (outputs not included):

import tensorflow as tf
import keras as kr
import tensorflow_datasets as tfds

(trainX, trainY), (testX, testY) = mnist.load_data()

trainX.shape
testX.shape
trainY.shape
testY.shape

import matplotlib.pyplot as plt
imageNum = 1500
plt.imshow(trainX[imageNum], cmap='magma')

import matplotlib.pyplot as plt
imageNum = 3332
plt.imshow(testX[imageNum], cmap='magma')

firstNeuralNetwork = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28,28)),
    tf.keras.layers.Dense(150, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10)
])

firstNeuralNetwork.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
firstNeuralNetwork.fit(x=trainX,y=trainY, epochs=25)

firstNeuralNetwork.evaluate(testX, testY)

Thanks for coming along on this coding journey in 2022! Hope you all sharpened your skills and/or learned something new along the way this year! Have a very happy holiday season and rest assured-I will be back in 2023 with brand new coding content (and a little something special for my blog’s 5th anniversary)!

Michael