examples/word_language_model/main.py · a9b28923c42c6d7c6f691716f597510ed0204c90 · llvm / distiller

7 years ago

Language model: replace the optimizer and LR-decay scheduler · a9b28923

Neta Zmora authored 7 years ago

Replace the original "homebrew" optimizer and LR-decay schedule with
PyTorch's SGD and ReduceLROnPlateau.
SGD with momentum=0 and weight_decay=0, and ReduceLROnPlateau with
patience=0 and factor=0.5 will give the same behavior as in the
original PyTorch example.

Having a standard optimizer and LR-decay schedule gives us the
flexibility to experiment with these during the training process.

a9b28923

History

Language model: replace the optimizer and LR-decay scheduler

Neta Zmora authored 7 years ago

Replace the original "homebrew" optimizer and LR-decay schedule with
PyTorch's SGD and ReduceLROnPlateau.
SGD with momentum=0 and weight_decay=0, and ReduceLROnPlateau with
patience=0 and factor=0.5 will give the same behavior as in the
original PyTorch example.

Having a standard optimizer and LR-decay schedule gives us the
flexibility to experiment with these during the training process.

main.py 15.59 KiB