Complexity and Entropy Trade-offs in Deep Neural Network Representations

Abstract

The information bottleneck view of deep learning holds that hidden-layer representations first fit the training data
and then compress, discarding task-irrelevant information as training proceeds, but direct estimates of
representation entropy across layers and training time remain scarce for networks of realistic depth. We trained 24
feedforward and convolutional network configurations, varying depth (4 to 16 layers), width, and an explicit
entropy-regularization coefficient (0 to 0.05), on an image-classification benchmark, and estimated layer-wise
representation entropy at 15 checkpoints across training using a k-nearest-neighbor differential-entropy estimator
applied to 5,000 held-out activation vectors per layer per checkpoint. Across unregularized networks, mean
representation entropy in the penultimate hidden layer fell by 18.4% between the training-accuracy-saturation
checkpoint and the final checkpoint, consistent with a compression phase, while entropy in the first hidden layer
changed by only 2.1% over the same interval, a significant difference (p < 0.001, paired t-test across 12
unregularized configurations). Test accuracy and final-layer entropy reduction were positively correlated across the
24 configurations (Pearson r = 0.68, p = 0.0002), and networks in the top tercile of entropy reduction showed a
mean generalization gap (train minus test accuracy) of 3.8 percentage points, compared with 9.1 percentage points
for the bottom tercile, a difference of 5.3 points. Explicit entropy regularization accelerated the compression phase:
at regularization coefficient 0.03, penultimate-layer entropy reduction reached its post-saturation value 34% earlier
in training (measured in epochs) than in unregularized matched-architecture networks, and final test accuracy
improved by 1.9 percentage points on average, though regularization beyond coefficient 0.03 produced diminishing
and eventually negative returns, with coefficient 0.05 networks losing 2.4 points of test accuracy relative to the 0.03
optimum. Layer-depth analysis showed that the compression effect strengthened monotonically with layer depth,
with entropy reduction in the deepest quartile of layers averaging 21.7% versus 4.3% in the shallowest quartile
across all 24 configurations. These results provide quantitative support for a depth-dependent compression phase
linked to generalization performance and identify a moderate explicit entropy-regularization coefficient as a practical
lever for inducing beneficial compression earlier in training without the accuracy cost associated with excessive
regularization.