We are going to create a simple image detection algorithm for cat images. This code is liberally taken from Logistic Regression with a Neural Network mindset

Loading Test/Training Images

Test images are from datasets/test_catvnoncat.h5 and training images are from datasets/train_catvnoncat.h5. The function load_dataset handles the loading details.

Here is one of the images from the training set

So, we have 209 images in our training set, but only 50 examples in our testing one. That reads a bit strange. Intuitively, I would expect to train on the smaller set of examples, and then test the performance on the larger set. Oh, and each image is just a 64x64 pixel color image.

First, we will reshape the images so that they are represented as a 1 column vector of bytes, with the RGB color information being interleaved in the result. For instance, the first three numbers of the vector will be the red, blue and green components of the first pixel in a cat image. The reason for doing this is not spelled out up front, but it has to do with how we will process the 1 column vectors below in our algorithms. Putting it in 1 (long) column form now makes things much easier to calculate with in the near future.

Intuitively, separating out the color bytes does not seem too beneficial. A better approach might be to convert to gray scale so that we only operate on one color value between black and white, and not three separate ones.

Next, rescale each of the color component values so that they fall between 0 and 1. Since each color component value is a 8-bit value between 0 and 255, we just divide them all by 255.0 to get the desired range. Again, I'm curious what the outcome would be if we instead work with one color value instead of three.

Build Neural Network of One Node

The main steps for building a Neural Network are:

1. Define the model structure (such as number of input features)
2. Initialize the model's parameters
3. Loop:
   • Calculate current loss (forward propagation)
   • Calculate current gradient (backward propagation)
   • Update parameters (gradient descent)

Forward and Backward propagation

Execute the "forward" and "backward" propagation steps for learning the parameters.

• Given $XX$
• Compute $A=\sigma (w^{T}X+b)=(a^{(1)},a^{(2)},\mathrm{.}\mathrm{.}\mathrm{.},a^{(m-1)},a^{(m)})A\; =\; \backslash sigma(w^T\; X\; +\; b)\; =\; (a^\{(1)\},\; a^\{(2)\},\; ...,\; a^\{(m-1)\},\; a^\{(m)\})$
• Calculate the cost function: $J=-\frac{1}{m}{\sum }_{i=1}^m{y}^{(i)}\log (a^{(i)})+(1-y^{(i)})\log (1-a^{(i)})J\; =\; -\backslash frac\{1\}\{m\}\backslash sum\_\{i=1\}^\{m\}y^\{(i)\}\backslash log(a^\{(i)\})+(1-y^\{(i)\})\backslash log(1-a^\{(i)\})$

The "backward" bit is when we evaluate how well the estimate fits the truth value given as $YY$. Evaluating this fitness is done below using gradient descent which is a fancy name for a search for the global minima of a convex function, which is basically a function that meets various criteria that guarantee there is only one global minima. Here are the partial differential equations for $dd$ and $ww$ which define the gradiant that will move along the cost function as we attempt to locate the minima.

$$\frac{\mathrm{\partial }J}{\mathrm{\partial }w}=\frac{1}{m}X(A-Y{)}^T\backslash frac\{\backslash partial\; J\}\{\backslash partial\; w\}\; =\; \backslash frac\{1\}\{m\}X(A-Y)^T$$ $$\frac{\mathrm{\partial }J}{\mathrm{\partial }b}=\frac{1}{m}\sum _{i=1}^m(a^{(i)}-y^{(i)})\backslash frac\{\backslash partial\; J\}\{\backslash partial\; b\}\; =\; \backslash frac\{1\}\{m\}\; \backslash sum\_\{i=1\}^m\; (a^\{(i)\}-y^\{(i)\})$$

The $\sigma \backslash sigma$ function is the sigmoid function, which maps a real input $ZZ$ to the range (0, 1). Both ends are open since 0 and 1 are never actually reached, but it is a nice and simple way to map values to a binary 0/1 domain. Note that there must be some reason for doing the mapping -- just blindly applying $\sigma \backslash sigma$ will probably not lead to very good results.

The propagate function holds the cost function and generates the gradient value updates for a given iteration of the learning.

Optimization

We will repeatedly evaluate a cost function, which by careful selection is a convex function -- one that always has a global minimum. It is this global minimum which we will seek to reach, by updating w and b values with their derivatives and a learning rate parameter. Currently, the latter is fixed, but one could easily see a use in having it not so that converging toward the minima does not overshoot and begin oscillating back and forth over it. Below we also record the cost scalar in each iteration so we can see how they converge later on.

Prediction

Calculate estimate $\widehat{Y}=A=\sigma (w^{T}X+b)\backslash hat\{Y\}\; =\; A\; =\; \backslash sigma(w^T\; X\; +\; b)$ Convert the entries of A into 0 (if activation <= 0.5) or 1 (if activation > 0.5), stores the predictions in a vector Y_prediction. Here we use the nifty Numpy where operator which lets us map a comparison on a set of values instead of looping over them in Python.

Build Model

Implement the model function:

• Y_prediction_test are the predictions on the test set
• Y_prediction_train are the predictions on the train set
• w, b, costs are the outputs of optimize()

Run the following cell to train your model.

Results

Let's evaluate how well our predicitions matched those of the test data. First, go through and determine what were false positives (FP) -- predictions of a cat where the test data said there probably was not a cat (< 50% chance) -- and what were false negatives -- predictions where we said there was no cat, but test data said there was (> 50% chance).

First, display the false positive images.

Next, display the false negative images.

Cost Reduction

Here is a plot showing how the cost values fell as we iterated in the optimization step.

Additional Analysis

We can rerun the above for different iteration amounts and learning rate values $\alpha \backslash alpha$.

So this is kind of silly, because the learning rate is a scalar that is multiplied to the $ww$ and $bb$ derivatives while searching for the global minima. A smaller learning rate just means that we crawl more slowly towards that magical goal. We could be smarter and evaluate the changes in the derivatives across iterations with a given learning rate, and if they go from negative to positive or vice-versa, then we have crossed the global minima and we can refine our search as long as there are iterations left to use.

And this is just the behavior we see above in the $0.010.01$ learning rate cost curve. The wild oscillations are due to the extreme slopes of the cost function; as we get closer to the minima the oscillations quickly subside and we more or less arrive close to the minima value. With the lower learning rate values, we don't see the early oscillations, but we also don't ever arrive at the same minima because we run out of iterations before we can get there.

We can easily see this by making the iteration count proportional to the learning rate -- num_iterations = int(15 / lr) -- in the call to our model function above.

(NOTE: this will take some time to run due to the explosion in the iteration counts):

Note that with the change in interation counts, we now arrive at the exact same cost minima for all of the learning rate values, but this effort is wasted for the smaller learning rates, as they essentially waste time to reach a point we already knew about with a larger learning rate.