Back to Posts

Gradients and Roses

How a childhood book taught me about deep learning and the art of identifying invisible patterns in data.

October 2025 Read on Medium

"It is only with the heart that one can see rightly; what is essential is invisible to the eye."

Antoine de Saint-Exupery, The Little Prince

Just as the Little Prince learns to see what truly matters, neural networks learn to see patterns invisible to simple rules.

That line has lived with me for years. I've always been the person who tries to read between the lines, to look beyond surface detail for structure and meaning. That is why neural networks feel so personal to me... they are proof that there is signal beneath the noise, form hidden inside apparent chaos.

The Loss Function: Distance from Your Rose

You left your rose on Asteroid B-612. Every planet you visit, you feel how far you are from truly understanding her. This distance... this longing... is your loss function.

When you're on the wrong planets (bad parameters), you feel lost and far from the truth. When you find planets that teach you about love, responsibility, and connection (good parameters), you feel closer to understanding your rose.

The loss function measures: "How far am I from truly understanding?"

Derivatives and Gradients: The Fox's Wisdom

"One sees clearly only with the heart," said the fox. But how do you know which direction to travel next?

The gradient is like the fox's guidance... it tells you which neighboring planet will bring you closer to understanding. The fox doesn't show you the entire universe at once; he shows you the next step.

The gradient points toward the planet where you'll feel less lost.

Just as the fox taught the Little Prince through small, patient steps ("You must be very patient. First you will sit down at a little distance from me..."), the gradient guides you one small step at a time.

In mathematical terms: If you're on a planet and want to know which direction reduces your confusion (loss), you calculate the derivative. It's the fox whispering: "Go this way, not that way."

Gradient Descent: Following the Fox's Path

"Goodbye," said the fox. "And now here is my secret, a very simple secret: It is only with the heart that one can see rightly; what is essential is invisible to the eye."

Gradient descent is following the fox's teaching: always move toward clearer understanding.

The process:

  1. Observe where you are (calculate loss)
  2. Ask the fox which way to go (calculate gradient)
  3. Take one step in that direction (update parameters): To be specific the gradient points in the direction of steepest increase in loss; gradient descent steps in the negative of that direction to reduce loss.
  4. Repeat until you understand (converge)

You don't need to see the whole universe at once. You only need to know: "Which neighbor planet brings me closer to my rose?"

Autograd: The Little Prince's Memory Book

The Little Prince kept a journal of his travels. Every planet he visited, every lesson learned, every step taken was recorded. When he needed to find his way back or understand how he got somewhere, he could trace his path through the pages.

Autograd is your magical journal that remembers every step of your journey.

Imagine this sequence:

  1. You left Asteroid B-612 (input)
  2. Visited the King's planet and learned about authority (first layer)
  3. Then the Conceited Man's planet and learned about vanity (second layer)
  4. Then the Drunkard's planet and learned about shame (third layer)
  5. Finally arrived at the Geographer's planet (output)

Now you ask: "How much did my starting point affect where I ended up?"

Without autograd, you'd have to manually retrace every step, remember every lesson, and calculate how each planet influenced the next. Exhausting!

With autograd, your magical journal automatically traces backward through all your visits, measuring how each planet's lesson contributed to your final understanding. One command - loss.backward() - and the journal reveals everything.

The Computational Graph: Your Star Map

Every operation in PyTorch creates a star on your map, connected by routes:

import torch

# Your starting point
x = torch.tensor(2.0, requires_grad=True)

# Your journey through the planets
y = x ** 2          # King's planet: square your understanding
z = y * 3           # Conceited Man's planet: amplify by 3
w = z + 5           # Businessman's planet: add 5 stars
loss = w ** 2       # Lamplighter's planet: square again

Your star map:

B-612 (2.0) -> King -> (4.0) -> Conceited Man -> (12.0) -> Businessman -> (17.0) -> Lamplighter -> (289.0)

When you call loss.backward(), your journal traces backward through this star map, calculating how each planet affected your final position.

Learning Rate: The Size of Your Jumps Between Planets

"My flower is ephemeral," thought the Little Prince, "and she has only four thorns to defend herself against the world! And I have left her on my planet, all alone!"

When you're desperate to return home, you might want to take huge leaps between planets. But:

  • Too large (high learning rate): You jump wildly, overshooting good planets, bouncing back and forth across the universe, never settling anywhere meaningful.
  • Too small (low learning rate): You take tiny, careful steps. Safe, but you'll need thousands of years to find your way home.
  • Just right: You travel with purpose... quick enough to make progress, careful enough not to miss important lessons.

The Little Prince traveled at just the right pace... fast enough to learn, slow enough to truly understand each planet's lesson.

Note: From here, I'm adding a little to the story to make the concept clearer. This is a departure from the original plot... bear with me while I borrow the Little Prince's world to explain the idea.

Momentum: Carrying Lessons from Planet to Planet

The Little Prince does not forget what each planet teaches him. He carries those lessons forward, so the next step begins with a little of the last one's direction. That accumulated guidance is momentum: it smooths zig-zags, keeps progress steady through flat regions, and helps him glide past small distractions instead of stopping to re-decide at every step.

Maintain a running velocity v_t and step with it:

v_t = mu * v_{t-1} - eta * grad(theta_t)
theta_{t+1} = theta_t + v_t

Nesterov momentum is like glancing ahead with the lessons you've accumulated... peeking where that carry-forward would take you, and taking the gradient there before you step.

Epochs: Reading Your Book Multiple Times

The Little Prince read his favorite passages over and over. Each time he read about the fox, he understood something new.

  • One epoch = reading through your entire book of training examples once
  • First reading (epoch 1): You grasp the basic story
  • Second reading (epoch 2): You notice subtle details
  • Third reading (epoch 3): You understand the deeper meanings
  • ...
  • Fiftieth reading (epoch 50): You've internalized the wisdom

Each time you revisit the same planets (training data), you learn something slightly different, refining your understanding.

Activation Functions: The Roses' Thorns

"It is such a secret place, the land of tears."

Activation functions decide which signals are strong enough to pass through... like thorns deciding what gets close to the rose.

  • ReLU (Rectified Linear Unit): Simple thorns. "If you come with kindness (positive), pass through. If you come with ill intent (negative), you're blocked!"
  • Sigmoid: Gentle thorns. "Everyone may approach, but I'll modulate how close you get... between 0 (far away) and 1 (touching the rose)."
  • Tanh: Like sigmoid but allows negative space: between -1 (repelled) and 1 (attracted).
  • Softmax: For the final choice. "Among all these roses, which probabilities describe which is yours?" Forces choices to sum to 1.

Without activation functions, neural networks would be like roses without thorns... unable to be selective, unable to learn complex patterns.

Regularization: The Essential and the Non-Essential

"What is essential is invisible to the eye," the fox teaches.

In machine learning, regularization is how we help a model focus on essentials instead of memorizing trivia. It adds a gentle preference for simpler, more stable explanations so the model performs well on new data... not just the examples it has already seen.

Why this matters

  • If training accuracy climbs while validation accuracy stalls or drops, the model is overfitting... it has learned the garden by heart but not what makes a rose a rose.
  • Regularization narrows the set of explanations the model can choose, nudging it toward patterns that generalize.

Common tools (how to think about them)

  • L2 / weight decay: keep weights modest so no single feature dominates. Works like trimming baobabs before they take over the planet. (Tip: with AdamW, weight decay is decoupled and usually preferable to adding L2 to the loss.)
  • L1: encourages sparsity... many connections become exactly zero. Good when you believe only a few features truly matter.
  • Dropout: randomly "forget" some pathways during training so the model can't rely on any one shortcut. It learns robust explanations that still work when some paths are missing.
  • Early stopping: stop traveling once the lessons stop improving on validation data; don't overstay on any one planet.
  • Data augmentation: show varied views of the same truth (e.g., flips/crops for images, noise/word-drop for text) so the model learns the essence, not the accidental detail.
  • Label smoothing (classification): avoid over-confident, brittle predictions by distributing a tiny bit of probability mass to non-target classes.

How to pick (practical heuristics)

  • Start simple: weight decay + early stopping. Add dropout for deeper nets or small datasets. Use augmentation when you can create faithful variations of your inputs.
  • If validation loss is noisy and accuracy swings, try a touch more weight decay or dropout; if learning stalls, reduce them or increase data/augmentation.
  • For transformer fine-tunes, modest weight decay (e.g., 0.01) and label smoothing (e.g., 0.1) are common; keep dropout small.

Regularization ultimately is daily care: pulling up baobabs, cleaning volcanoes, and tending the rose. You're not changing who the rose is; you're removing distractions so her essence stands out.

Putting It All Together: The Little Prince's Complete Journey

Training a neural network is like the Little Prince's journey:

  1. Start on a random planet (initialize weights randomly)
  2. Feel how lost you are (calculate loss)
  3. Ask the fox which way to go (calculate gradients via backpropagation and autograd)
  4. Take a step toward understanding (update parameters using optimizer)
  5. Remember your momentum (don't zigzag)
  6. Examine groups of roses together (mini-batches)
  7. Use your thorns to be selective (activation functions)
  8. Focus on what's essential (regularization)
  9. Read your book many times (multiple epochs)
  10. Watch for the geographer's trap (monitor for overfitting)
  11. Know when you've found your rose (convergence)

Not sure if this is going to be useful to anyone else, but this little essay has always been a fun way for me to reflect on core deep learning concepts.