hive.utils.torch_utils module

hive.utils.torch_utils.numpify(t)[source]

Convert object to a numpy array.

Parameters

t (np.ndarray | torch.Tensor | obj) – Converts object to np.ndarray.

class hive.utils.torch_utils.RMSpropTF(params, lr=0.01, alpha=0.9, eps=1e-10, weight_decay=0, momentum=0.0, centered=False, decoupled_decay=False, lr_in_momentum=True)[source]

Bases: Optimizer

Direct cut-paste from rwhightman/pytorch-image-models. https://github.com/rwightman/pytorch-image-models/blob/f7d210d759beb00a3d0834a3ce2d93f6e17f3d38/timm/optim/rmsprop_tf.py Licensed under Apache 2.0, https://github.com/rwightman/pytorch-image-models/blob/master/LICENSE

Implements RMSprop algorithm (TensorFlow style epsilon)

NOTE: This is a direct cut-and-paste of PyTorch RMSprop with eps applied before sqrt and a few other modifications to closer match Tensorflow for matching hyper-params. Noteworthy changes include:

  1. Epsilon applied inside square-root

  2. square_avg initialized to ones

  3. LR scaling of update accumulated in momentum buffer

Proposed by G. Hinton in his course. The centered version first appears in Generating Sequences With Recurrent Neural Networks.

Parameters

  • params (iterable) – iterable of parameters to optimize or dicts defining parameter groups

  • lr (float, optional) – learning rate (default: 1e-2)

  • momentum (float, optional) – momentum factor (default: 0)

  • alpha (float, optional) – smoothing (decay) constant (default: 0.9)

  • eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-10)

  • centered (bool, optional) – if True, compute the centered RMSProp, the gradient is normalized by an estimation of its variance

  • weight_decay (float, optional) – weight decay (L2 penalty) (default: 0)

  • decoupled_decay (bool, optional) – decoupled weight decay as per https://arxiv.org/abs/1711.05101

  • lr_in_momentum (bool, optional) – learning rate scaling is included in the momentum buffer update as per defaults in Tensorflow

step(closure=None)[source]

Performs a single optimization step.

Parameters

closure (callable, optional) – A closure that reevaluates the model and returns the loss.