Build an Image Classifier with CNNs: A Practical Deep Learning Project

A convolutional neural network (CNN) classifies images by first learning to detect small local patterns — edges, textures, simple shapes — then combining those patterns into progressively larger and more abstract features, and finally turning that summary into a decision about what the image shows. Building one from scratch is one of the best ways to actually understand how computer vision works under the hood. 🖼️

This post builds a complete, working image classifier from nothing, using the same architecture pattern documented in PyTorch's own official tutorial. Every piece — the convolution, the pooling, the flattening, the final classifier — gets explained in plain language first, then shown as real, runnable code. By the end you'll have a model that actually trains and a solid mental model of why each piece is there. 🧱

📑 In This Post

• 1. Foundations: What Makes CNNs Good at Images
• 2. Convolutional Layers: Learning Local Patterns
• 3. Pooling: Shrinking While Keeping What Matters
• 4. From Feature Maps to Class Scores
• 5. Real Example: PyTorch's Official CIFAR-10 Classifier
• 6. Implementation: A Complete Working CNN
• 7. Enterprise Rollout: From Notebook to Reliable Pipeline
• 8. Common Mistakes
• 9. FAQ
• 10. References & Further Reading
• 11. Summary

🔀 Quick Comparison: Convolutional Layer vs. Fully Connected Layer

Property	Convolutional layer	Fully connected layer
Connectivity	Each output looks at a small local region of the input	Every output connects to every input
Parameters	Shared across the whole image (a small filter reused everywhere)	A separate weight for every input-output pair
Spatial awareness	Preserves and exploits spatial structure	Treats input as a flat list of numbers
Typical role	Extracting features from images	Turning extracted features into a final decision

1. Foundations: What Makes CNNs Good at Images

💭 Analogy first: imagine trying to find every cat ear in a giant photo by memorizing the exact pixel position of every possible ear shape — an impossible task. Now imagine instead learning what an "ear shape" looks like once, then sliding that same knowledge across every part of the photo to check everywhere. That second approach is exactly what a convolutional layer does.

A plain fully connected network treats an image as a long, flat list of numbers, with a separate learned weight connecting every single pixel to every neuron in the next layer. This ignores something obvious and important: nearby pixels are related, and a useful pattern (an edge, a curve, a texture) can appear anywhere in the image. A CNN is built around a different assumption — learn a small set of pattern detectors once, and reuse them everywhere in the image — which is both far more parameter-efficient and much better matched to how visual patterns actually work.

What fails without this design: a fully connected network trained on images typically needs vastly more parameters to reach comparable performance, is more prone to overfitting on limited data, and has no built-in notion that a cat in the top-left corner and a cat in the bottom-right corner are recognizable by the same underlying features.

🎯 Use this when you need to explain, in one sentence, why CNNs exist instead of just using a regular fully connected network on images.

2. Convolutional Layers: Learning Local Patterns

💭 Analogy first: a stencil cut with a specific shape lights up a matching pattern wherever you place it on a page, regardless of position. A convolutional filter works the same way — it's a small grid of learned numbers that produces a strong response wherever it finds a matching pattern in the image, and a weak response everywhere else.

What it does: a convolutional layer slides a small filter (also called a kernel) across the input image, computing one number at each position by multiplying the filter's values against the pixels underneath it and summing the result. Doing this for every position produces a "feature map" — a smaller grid showing where in the image that particular pattern was detected.

Why it's needed: a single filter shared across the whole image means the network only needs to learn one set of weights to detect a pattern everywhere it appears, rather than a separate set of weights for every possible location — a dramatic reduction in the number of parameters compared to a fully connected layer over the same input.

How it works, step by step, across a typical CNN:

1. The first convolutional layer typically learns simple patterns — edges, color contrasts, basic textures — since that's what's directly visible in raw pixel values.
2. An activation function, most commonly ReLU, is applied after the convolution to introduce non-linearity, without which stacking more layers would mathematically collapse into something no more powerful than a single layer.
3. Later convolutional layers take the previous layer's feature maps as input, and because they now see combinations of simple patterns, they learn progressively more complex features — corners, textures, and eventually object parts.
4. Multiple filters are learned per layer, each producing its own feature map, so a single layer can simultaneously detect many different patterns.

What fails without stacking multiple convolutional layers: a single convolutional layer can only detect the simple, low-level patterns visible directly in raw pixels; recognizing a whole object requires combining many such patterns, which is exactly what depth (multiple stacked layers) provides.

3. Pooling: Shrinking While Keeping What Matters

💭 Analogy first: when you shrink a photo into a thumbnail, you lose fine detail but keep enough of the overall shape to still recognize what's in it. Pooling does something similar to a feature map — it shrinks the grid while trying to keep the strongest, most important signal.

What it does: a pooling layer reduces the size of a feature map by summarizing small neighborhoods into a single value — the most common version, max pooling, takes the largest value in each small region.

Why it's needed: shrinking feature maps as they pass through the network reduces the amount of computation and memory needed in later layers, and it also makes the network somewhat less sensitive to the exact pixel position of a detected feature — if the pattern shifts by a pixel or two, max pooling in that region often still picks up the same strong signal.

What fails without pooling (or a similar downsampling mechanism): feature maps would stay large through every layer, meaning far more parameters and computation in every subsequent layer, and the network would become more sensitive to tiny, irrelevant shifts in exactly where a pattern appears.

💡 Trade-off: pooling throws away information by design — that's the entire point — so applying it too aggressively, too early, or too many times can discard detail the network actually needed, especially for tasks where fine-grained spatial precision matters (such as detecting very small objects).

4. From Feature Maps to Class Scores

After several rounds of convolution and pooling, the network has a stack of small feature maps summarizing what patterns were found and roughly where. Two more steps turn this into an actual class prediction.

Flatten: the multi-dimensional stack of feature maps is reshaped into one long, flat vector of numbers, since fully connected layers expect a simple list of inputs rather than a spatial grid.

Fully connected layers: one or more fully connected layers take that flattened vector and combine it into a final set of scores — one number per possible class. The class with the highest score is the model's prediction.

Why the split between convolution and fully connected layers makes sense: convolutional layers are good at extracting spatial features efficiently; fully connected layers are good at combining a fixed-size summary of those features into a final decision. Using each for what it's good at is more efficient than trying to do everything with one type of layer.

5. Real Example: PyTorch's Official CIFAR-10 Classifier

PyTorch's own "Training a Classifier" tutorial builds exactly this pattern on the CIFAR-10 dataset, which the tutorial describes as containing 10 classes — plane, car, bird, cat, deer, dog, frog, horse, ship, and truck — with each image sized 3×32×32 (3 color channels, 32 by 32 pixels).

The tutorial's documented preprocessing step is worth noting directly: images loaded via torchvision start as PILImage objects with pixel values in the [0, 1] range, and the tutorial transforms them into tensors normalized to the [-1, 1] range before training.

The documented model itself follows the exact pattern described in Sections 2 through 4: two convolutional layers (the first mapping 3 input channels to 6 output channels with a 5×5 filter, the second mapping 6 channels to 16 with another 5×5 filter), a max pooling layer reused after each convolution, and three fully connected layers reducing the flattened features down to 120, then 84, then finally 10 outputs — one score per CIFAR-10 class. The documented forward pass applies ReLU and pooling after each convolution, flattens the result, and passes it through the three fully connected layers in sequence.

✅ Worked example: tracing the documented architecture's shapes: a 32×32 input shrinks to 28×28 after the first 5×5 convolution (no padding), then to 14×14 after pooling, then to 10×10 after the second convolution, then to 5×5 after the second pooling — which is exactly why the tutorial's first fully connected layer is defined with an input size of 16 × 5 × 5, matching the 16 channels at 5×5 spatial resolution that reach the flatten step.

6. Implementation: A Complete Working CNN

Here is a complete, runnable implementation following the same documented pattern, generalized slightly with configurable channel sizes and clear comments explaining each shape transformation.

import torch import torch.nn as nn import torch.nn.functional as F class SimpleCNN(nn.Module): def __init__(self, num_classes=10): super().__init__() # Block 1: 3 input channels (RGB) -> 6 feature maps self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, kernel_size=5) self.pool = nn.MaxPool2d(kernel_size=2, stride=2) # Block 2: 6 -> 16 feature maps self.conv2 = nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5) # Classifier head: flattened 16*5*5 features -> class scores self.fc1 = nn.Linear(16 * 5 * 5, 120) self.fc2 = nn.Linear(120, 84) self.fc3 = nn.Linear(84, num_classes) def forward(self, x): x = self.pool(F.relu(self.conv1(x))) # 32x32 -> 28x28 -> 14x14 x = self.pool(F.relu(self.conv2(x))) # 14x14 -> 10x10 -> 5x5 x = torch.flatten(x, 1) # (batch, 16, 5, 5) -> (batch, 400) x = F.relu(self.fc1(x)) x = F.relu(self.fc2(x)) return self.fc3(x) # raw class scores (logits) model = SimpleCNN(num_classes=10) loss_fn = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) # Training step, following the same five-step core as any model: # images, labels = next(iter(train_loader)) # preds = model(images) # loss = loss_fn(preds, labels) # loss.backward() # optimizer.step() # optimizer.zero_grad()

Notice this uses nn.CrossEntropyLoss, which expects raw, unnormalized class scores (logits) directly from the final layer, not probabilities — the loss function applies its own softmax internally, so the model's forward method should not apply softmax itself.

7. Enterprise Rollout: From Notebook to Reliable Pipeline

A CNN that trains successfully in a notebook is a different thing from an image classification system a team can maintain and trust in production.

Standardized preprocessing, versioned: the exact resize, crop, and normalization values used at training time must be reproduced identically at inference time — a mismatch here silently degrades accuracy without any error, since the model still runs, just on inputs it wasn't trained to expect.

Held-out test set discipline: keep a fixed, versioned test set never used for any training or hyperparameter decisions, so reported accuracy reflects genuine generalization rather than accumulated, indirect overfitting to the same evaluation data over many iterations.

Data augmentation as a documented, versioned pipeline step: augmentation choices (random crops, flips, color jitter) materially affect what the model learns to be invariant to, and should be tracked alongside the model checkpoint just like any other training configuration.

Monitoring for input drift in production: a classifier trained on one image distribution (say, professional product photos) can degrade significantly on a different distribution (user-uploaded phone photos) even if labels are the same — track prediction confidence distributions over time as an early warning signal.

Access controls and rollback: a new model version should be validated against the fixed test set and ideally a canary slice of real traffic before fully replacing a production classifier, with a clear path to revert if accuracy regresses.

✅ Practical pattern: save the exact preprocessing/transform configuration in the same file or metadata record as the model checkpoint, not as separate, easy-to-lose tribal knowledge — this is one of the most common sources of silent accuracy loss when models move from training to serving.

8. Common Mistakes

Mismatched fully connected input size. Forgetting to account for how convolution and pooling shrink spatial dimensions leads to a fully connected layer defined with the wrong input size — as Section 5's worked example shows, this size must be calculated (or verified with a dummy forward pass) rather than guessed.

Applying softmax before CrossEntropyLoss. Since PyTorch's nn.CrossEntropyLoss applies softmax internally, applying it a second time in the model's forward pass distorts the loss calculation, typically leading to slower or unstable training without an obvious error message pointing at the cause.

Inconsistent normalization between training and inference. If training used one normalization scheme and a deployed inference pipeline uses another (or none at all), the model receives inputs shaped differently from what it learned on, silently degrading accuracy in production.

Overfitting a small dataset without augmentation or regularization. CNNs have many parameters and can memorize a small training set almost perfectly while generalizing poorly — watch the training-versus-validation loss gap described in earlier debugging discussions, and add data augmentation or dropout if it appears.

Forgetting model.eval() during evaluation. If the network includes layers like dropout or batch normalization, leaving the model in training mode during evaluation produces noisy, non-reproducible predictions, exactly as covered for training loops generally.

❓ FAQ

Why use convolution instead of just flattening the image and using regular layers?

A regular fully connected layer needs a separate weight for every pixel-to-neuron connection, which is both far more parameters and ignores that useful patterns can appear anywhere in the image. Convolution reuses the same small set of learned weights across every position, which is both more efficient and better matched to how visual patterns work.

How do I know what size to make my fully connected layer after the convolutional layers?

Calculate it from how each convolution and pooling layer changes the spatial dimensions, as shown in Section 5's worked example, or simply run a dummy input through just the convolutional part of the network and check the resulting tensor's shape directly before defining the fully connected layer.

Do I need a GPU to build and train a simple CNN like this one?

No — a small CNN on a small dataset like CIFAR-10 can train on a CPU, just more slowly. A GPU becomes much more valuable as the model, image resolution, or dataset size grows, since convolution is highly parallelizable work that GPUs are specifically built to accelerate.

What's the difference between a feature map and the original image?

The original image has 3 color channels representing red, green, and blue intensity. A feature map produced by a convolutional layer instead represents how strongly a specific learned pattern was detected at each position — the "channels" in later layers no longer correspond to colors, but to learned pattern detectors.

Should I always use max pooling, or are there alternatives?

Max pooling is the most common and the one used in the documented tutorial architecture in this post, but average pooling (taking the mean instead of the maximum in each region) and strided convolutions (letting the convolution itself do the downsampling) are both established alternatives with their own trade-offs, beyond this post's beginner scope.

🔗 References & Further Reading

• PyTorch: Training a Classifier (CIFAR-10 tutorial) — the documented CIFAR-10 dataset details, normalization approach, and the reference CNN architecture used throughout this post.

PyTorch is a trademark of its respective owners.

📝 Summary

• CNNs exploit the idea that useful visual patterns can appear anywhere in an image, sharing learned filters across the whole input instead of learning a separate weight per pixel.
• Convolutional layers detect local patterns; stacking them lets the network build from simple edges to complex, abstract features.
• Pooling shrinks feature maps while keeping the strongest signal, reducing computation and adding some tolerance to small shifts in position.
• Flattening plus fully connected layers turn the final feature maps into class scores.
• PyTorch's official CIFAR-10 tutorial documents a concrete, minimal version of exactly this architecture.
• A working implementation is a small, clearly structured PyTorch module following the same conv-pool-conv-pool-flatten-fc pattern.
• Enterprise deployment requires consistent preprocessing, versioned test sets, and monitoring for input distribution drift.
• Most CNN bugs come from shape mismatches after conv/pool layers or inconsistent preprocessing between training and inference.

Thanks for reading — may your feature maps be sharp and your class scores be confident. 🚀