Uncertainty in Deep Learning

Review of PhD thesis of Dr. Yarin Gal
ML
Author

Zeel B Patel

Published

March 5, 2022

import torch

1 - Introduction

1.1 - Deep Learning

We define a single layer network as the following:

class SingleLayerNetwork(torch.nn.Module):
    def __init__(self, Q, D, K):
        """
        Q: number of features
        D: number of outputs
        K: number of hidden features
        """
        super().__init__()
        self.input = torch.nn.Linear(Q, K) # Transforms Q features into K hidden features
        self.output = torch.nn.Linear(K, D) # Transforms K hidden features to D output features
        self.non_lin_transform = torch.nn.ReLU() # A non-linear transformation
        
    def forward(self, X):
        """
        X: input (N x Q)
        """
        self.linear_transformed_X = self.input(X)  # (N, Q) -> (N, K)
        self.non_lin_transformed_X = self.non_lin_transform(linear_transformed_X)  # (N, K) -> (N, K)
        output = self.output(self.non_lin_transformed_X)  # (N, K) -> (N, D)
        return output
Q = 10 # Number of features
N = 100 # Number of samples
D = 15 # Number of outputs
K = 32 # Number of hidden features

X = torch.rand(N, Q) # Input
Y = torch.rand(N, D) # Output
model = SingleLayerNetwork(Q=Q, D=D, K=K)
model
SingleLayerNetwork(
  (input): Linear(in_features=10, out_features=32, bias=True)
  (output): Linear(in_features=32, out_features=15, bias=True)
  (non_lin_transform): ReLU()
)
for name, value in model.named_parameters():
    print(name, value.shape)
input.weight torch.Size([32, 10])
input.bias torch.Size([32])
output.weight torch.Size([15, 32])
output.bias torch.Size([15])

ReLU is does not contain any parameters here so it is merely a function.

1.2 Model Uncertainty

In which cases we want our model to be uncertain?

  • When it encounters a out-of-the-distribution data
  • When training data is noisy (irreducible/aleatoric uncertainty)
  • When we have multiple predictors (model/epistemic uncertainty)