Main Content

cosineLearnRate

Cosine learning rate schedule

Since R2024b

expand all in page

    Description

    A cosine learning rate schedule object drops the learning rate using a cosine curve and incorporates warm restarts.

    Tip

    For most tasks, setting the Period argument to the number of training steps can yield better results. To easily use a cosine learning rate schedule that drops the learning rate over the length of the training process, specify the LearnRateSchedule argument of the trainingOptions function as "cosine".

    Creation

    Description

    schedule = cosineLearnRate creates a cyclicalLearnRate object.

    example

    schedule = cosineLearnRate(Name=Value) specifies optional properties using one or more name-value arguments. For example, InitialFactor=1.5 specifies initial scaling of the learning rate with a factor of 1.5.

    example

    Properties

    expand all

    Initial scaling factor, specified as a positive scalar.

    The software starts by scaling the base learning rate by InitialFactor and transitions toward scaling the learning rate by FinalFactor.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Final scaling factor, specified as a positive scalar.

    The software starts by scaling the base learning rate by InitialFactor and transitions toward scaling the learning rate by FinalFactor.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Number of steps in the cycle, specified as a positive integer.

    Tip

    For most tasks, setting the Period argument to the number of training steps can yield better results. To easily use a cosine learning rate schedule that drops the learning rate over the length of the training process, specify the LearnRateSchedule argument of the trainingOptions function as "cosine".

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Period growth factor, specified as a positive scalar.

    After Period steps, the software increases the period by scaling it by PeriodGrowthFactor.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

    Frequency unit, specified as "epoch" or "iteration".

    If FrequencyUnit is "epoch", then the software updates the learning rate every epoch and each iteration of the epoch uses the same learning rate. If FrequencyUnit is "iteration", then the software updates the learning rate every iteration.

    This property is read-only.

    Number of steps to drop learning rate, specified as Inf.

    cosineLearnRate objects are infinite learning rate schedules, so NumSteps is Inf.

    Data Types: double

    Examples

    collapse all

    Open Live Script

    Create a cosine learning rate schedule with the default settings.

    schedule = cosineLearnRate
    schedule = 
  cosineLearnRate with properties:

         InitialFactor: 1
           FinalFactor: 1.0000e-05
                Period: 30
    PeriodGrowthFactor: 2
         FrequencyUnit: "epoch"
              NumSteps: Inf

    Specify this schedule as a training option.

    options = trainingOptions("adam",LearnRateSchedule=schedule);

    For most tasks, setting the Period argument to the number of training steps can yield better results. As an alternative to creating a cosine learning rate object, to easily use a cosine learning rate schedule that drops the learning rate over the length of the training process, specify the LearnRateSchedule argument of the trainingOptions function as "cosine".

    options = trainingOptions("adam",LearnRateSchedule="cosine");
    Open Live Script

    Create a cosine learning rate schedule that drops the learning rate for an initial period of 100 iterations, starting with an initial scaling factor of 0.5.

    schedule = cosineLearnRate( ...
    InitialFactor=0.5, ...
    Period=100, ...
    FrequencyUnit="iteration")
    schedule = 
  cosineLearnRate with properties:

         InitialFactor: 0.5000
           FinalFactor: 1.0000e-05
                Period: 100
    PeriodGrowthFactor: 2
         FrequencyUnit: "iteration"
              NumSteps: Inf

    Specify this schedule as a training option.

    options = trainingOptions("adam",LearnRateSchedule=schedule);

    Algorithms

    expand all

    Version History

    Introduced in R2024b

    See Also

    | | | | | | |

    Topics