# Convolution and Deconvolution Revisited

In an earlier post, I had explained convolution and deconvolution in deep neural networks. The purpose of this post is to demo these operations using PyTorch. Before doing that, we will visit different operations associated with a convolution.

#### Convolution Operation

Convolution is an operation on two functions of real valued arguments. One of these functions, considered a signal, is an n-dimensional array of numbers, for example a 3-dimensional array of numbers representing a color image. The second function is a kernel or filter whose size is typically much smaller than the input array size. The array representing the kernel function is called kernel mask. The purpose of the convolution operation is to transform the input into a new array with the aim of highlighting some property of the input array. Thus, convolution can be viewed as feature extraction, and the transformed array is often called feature map where feature implies a particular characteristic of the input extracted by the kernel.

The convolution operation is performed by moving the kernel mask over the signal array and calculating the kernel response at each location. To understand the convolution operation, let’s consider a 3-dimensional input array representing the red, green, and blue channels of a colored image patch and a 3×3 convolution filter as shown below. To perform convolution at a particular position of the input array, we place the center of the convolution mask at the desired position and perform element by element multiplication between the signal array elements and the convolution mask elements followed by summation for each input channel as shown in the figure below. The responses from three channels are then added to produce the output of the convolution operation. The response over the entire input array is obtained by moving the mask center one step at a time and repeating the calculations.

##### Stride and Dilation

We generally move the kernel mask over the input array to the next pixel. However, we can skip a pixel or two in between when moving the mask. The parameter stride determines how the mask is moved during convolution. A stride of 1 means moving to the next pixel with no skipping of pixels/cells and a stride of 2 means moving by two pixels. A stride value other than the default value of 1 means convolution response will be calculated at fewer positions. This means the size of the resulting feature map will be smaller than the input even with padding. Thus, setting a suitable value for stride allows us to down sample the convolution result. The figure below shows the positions where a 3×3 mask would be placed with the default stride value of 1 (blue cells) and with a stride value of 2 (Cells marked with X), when there is no padding. Clearly, stride of 2 will down sample the input to produce a smaller feature map.

##### Pooling and Rectification

A typical convolutional neural network (CNN) is used for classification. In such a network, you will find a large number of convolution layers. Since convolution is a linear operation, we need to insert some nonlinearity between two consecutive convolution layers. Thus, the output of convolution layer is rectified via running it through ReLU (Rectifier Linear Unit). The rectified output of each convolutional layer is followed by a pooling layer whose task is to down sample the convolution result. This is done by replacing a block of convolution layer cells with a single cell. For example, the convolution layer output can be divided into adjacent groups of 2×2 blocks to be replaced by the 2×2 block average. This is called average pooling. When a 2×2 blocks is replaced by the maximum value of the block, the resulting pooling is known as max pooling. Irrespective of the type of pooling used, the basic advantage of pooling is the resulting down sampling which in turn speeds up the computation and minimizes the variance in data moving forward.

#### Convolution Layer Visualization

With the above introduction to the different operations involved with a single convolution layer, lets try to put together a demo to show the effect of different parameters on convolution operation. To do the demo, lets get an image that we will use.


from PIL import Image
import matplotlib.pyplot as plt
%matplotlib inline
pil_image = Image.open('data/chair.jpg')
plt.imshow(pil_image)


The image size is 240X180. Next, we import the necessary libraries. Since PyTorch accepts tensors, the image read earlier will be converted to a tensor. We are going to use four convolution filters. These will not be learned but set by defining a numpy array. The code for this part including the visualization of the filters is shown below.

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
from torchvision import datasets, models, transforms
# Transform PIL image to a tensor
transform = transforms.ToTensor()
img = transform(pil_image)
#Define filters
filter_array = np.array([[-1, -0.5,0, 0.5, 1], [-1, -0.5,0, 0.5, 1],
[-1, -0.5, 0, 0.5, 1], [-1, -0.5,0, 0.5, 1],
[-1, -0.5,0, 0.5, 1]])
filter_1 = filter_array
filter_2 = -filter_1
filter_3 = filter_1.T
filter_4 = -filter_3
filters = np.array([filter_1, filter_2, filter_3, filter_4])
#Visualize filters
fig = plt.figure(figsize=(12, 6))
fig.subplots_adjust(left=0, right=0.5, bottom=0.8, top=1, hspace=0.05, wspace=0.05)
for i in range(4):
ax = fig.add_subplot(1, 4, i+1, xticks=[], yticks=[])
ax.imshow(filters[i], cmap='hot')
ax.set_title('Filter %s' % str(i+1))


Now, we will set up a two-layer convolution network to perform convolution. The code for this is given below.

class DemoNet(nn.Module):

def __init__(self, wt1,wt2):
super(DemoNet, self).__init__()
# We initialize the weights of the convolutional layer as the 4 defined filters
self.conv1 = nn.Conv2d(3, 4, kernel_size=5,stride=1,dilation=1, bias=False)
self.conv1.wt1 = torch.nn.Parameter(wt1)
# define a pooling layer
self.pool1 = nn.MaxPool2d(2, 2)
#Define another conv layer
self.conv2 = nn.Conv2d(4,4,kernel_size=5, bias =False)
self.conv2.wt2 = torch.nn.Parameter(wt2)
self.pool2 = nn.MaxPool2d(2,2)

def forward(self, x):
# calculates the output of a convolutional layer
# pre- and post-activation
conv1_x = self.conv1(x)
activated1_x = F.relu(conv1_x)
# applies pooling layer
pooled1_x = self.pool1(activated1_x)
conv2_x = self.conv2(pooled1_x)
activated2_x = F.relu(conv2_x)
pooled2_x = self.pool2(activated2_x)
# returns all layers
return conv1_x, activated1_x, pooled1_x, conv2_x,activated2_x,pooled2_x



Next, we define a function that will be used to visualize the output of the convolution layer filters.

def visualize_layer(layer, n_filters= 4):
fig = plt.figure(figsize=(12, 12))

for i in range(n_filters):
ax.imshow(np.squeeze(layer[0,i].data.numpy()))
ax.set_title('Filter %s' % str(i+1))


Now, we are ready to instantiate our network, feed the input image, and compute the output at different layers. Just to make the second layer filters different, we will add a random perturbation as shown below.

wt1 = torch.from_numpy(filters).unsqueeze(1).type(torch.FloatTensor)
wt2 = torch.from_numpy(filters).unsqueeze(1).type(torch.FloatTensor)+0.5*torch.randn(5,5)
model = DemoNet(wt1,wt2)
#Compute output
conv1_x, activated1_layer, pooled1_layer, conv2_x,activated2_layer,pooled2_layer = model.forward(img.unsqueeze(0))


Lets now visualize the output of the first convolution layer. The first row below shows the outputs of the four filters before rectification. The second row of four images is the output of the first convolution layer after rectification.

visualize_layer(conv1_x)
visualize_layer(activated1_layer)


Looking at the images in the first row, we notice that filters 1 and 2 produce complimentary response; so is done by filters3 and 4. Further, some of the image features are highlighted in the rectified output. Next, we visualize the second convolution layer in a similar manner.

visualize_layer(conv2_x)
visualize_layer(activated2_layer)


Although all images are displayed at same size, the tick marks on axes indicate that the images at the output of the second layer filters are half of the input image size because of pooling. The mostly dark image in the second row indicates filter2 producing mostly negative values that are getting rectified to 0. To see how changing the stride value from 1 to 2 will change the output, we set the stride to 2 and run the network again. The first row of the images show the rectified output from the first layer and the second row of images are the rectified output from the second layer with the updated stride value of 2 for both layers. With stride of 2, the output at second layer is heavily down sampled.

Now, lets see the effect of dilation. With a dilation value of 3, the result at the first and second layers after rectification appear as shown below. In this case, image features appear prominently compared to output without dilation.

#### 1×1 Convolution

A 1×1 convolution is often confusing because its utility is not obvious. A 1×1 convolution applied to a single image will only scale the pixel values by a factor of the 1×1 convolution weight; thus, it is unclear what benefit might be there of such a convolution. Well! to understand what benefit might be there, lets consider m input channels over which 1×1 convolution is to be applied. In this case, the 1×1 convolution operation can be expressed with the following equation where input(k) stands for the k-th input channel:

$f_{ij} = \sum_{ij}\sum_{k}^m input(k)_{ij}$

As this equation indicates, 1×1 convolution aggregates the input channel values along the depth axis; thus it is often called the depth convolution. This is also illustrated in the figure below.

The main usage of 1×1 convolution is in reducing computation or dimensionality reduction by reshaping input before filtering. Suppose at some intermediate stage in your convolution network, you have 64 filtered images or feature maps of size 28×28 pixels. You want to apply 16 different convolution masks of size 3×3 to these 28x28x64 images. This will require 28*28*16*3*3*64 (7225344) operations. Instead of directly applying 16, 3×3 masks on 64 channels of incoming images, we first reshape the incoming images to 28x28x4 via 4, 1×1 convolution filters. This will require 28*28*4*1*1*64 (200704) operations. Next, applying 16, 3×3 filters on the reshaped input will require 28*28*4*3*3*16 (451584) operations. Adding these two sets of operations, we can see that reshaping via 1×1 convolution requires about 90% fewer operations.

Lets now perform 1×1 convolution on the output of our demo network. To do this, we add another convolution layer to our network and make necessary changes to the network definition. The result of 1×1 convolution is then the feature map shown below. This output is the sum of four rectified images after pooling at the second convolution layer of our demo network.

#### Deconvolution

The use of the term deconvolution in deep learning is different from its meaning in signal and image processing. While convolution without padding results in a smaller sized output, deconvolution increases the output size. With stride values greater than 1, deconvolution is used as a way of up sampling the data stream. This appears to be its main usage in deep learning. Both the convolution and deconvolution operations in deep learning are actually implemented as matrix multiplication operations and the deconvolution is actually transposed convolution. I would direct interested readers to my earlier post on this topic where I explain how convolution and deconvolution operations are carried out as matrix multiplications.  Here I will just show the result of deconvolution operation performed on the output of our two-layer demo network. The first image is with the default stride value of 1 and the second image is with a stride value of 2. In the second image, the size of the original image has been recovered. I use the following code to perform the deconvolution operation:

inputs_to_decon = activated2_layer