cerebras.modelzoo.common.utils.model.mup_utils.LRAdjustmentGroup#

class cerebras.modelzoo.common.utils.model.mup_utils.LRAdjustmentGroup(param_filter, scale=1.0)[source]#

Bases: object

Stores data for a group of params that share a learning rate scalar. Stores a callable that returns True if a given model param corresponds to the group. Additionally, it stores the scale that should be applied to the LR of the model params that correspond to the group.

param_filter: A string or a list of strings that contains glob expressions used to match whether a given model param name belongs to the group.

scale: The scale that should be applied to the LR of this group

Methods

set_scale

previous

cerebras.modelzoo.common.utils.model.mup_utils.scale_initializers_by_dimension

next

cerebras.modelzoo.common.utils.model.transformer_utils