How to initialize weights in PyTorch?

The Solution to How to initialize weights in PyTorch? is

Single layer

To initialize the weights of a single layer, use a function from torch.nn.init. For instance:

conv1 = torch.nn.Conv2d(...)

Alternatively, you can modify the parameters by writing to (which is a torch.Tensor). Example:

The same applies for biases:

nn.Sequential or custom nn.Module

Pass an initialization function to torch.nn.Module.apply. It will initialize the weights in the entire nn.Module recursively.

apply(fn): Applies fn recursively to every submodule (as returned by .children()) as well as self. Typical use includes initializing the parameters of a model (see also torch-nn-init).


def init_weights(m):
    if type(m) == nn.Linear:

net = nn.Sequential(nn.Linear(2, 2), nn.Linear(2, 2))

~ Answered on 2018-03-22 16:34:42

Most Viewed Questions: