cyclicalLearnRate

Cyclical learning rate schedule

Since R2024b

Description

A cyclical learning rate schedule periodically increases and decreases the learning rate.

Tip

To easily use a cyclical learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "cyclical".

Creation

Description

schedule = cyclicalLearnRate creates a cyclicalLearnRate object.

This syntax is equivalent to setting the LearnRateSchedule argument of the trainingOptions function to "cyclical".

example

schedule = cyclicalLearnRate(Name=Value) specifies additional properties using one or more name-value arguments. For example, MaxFactor=2 specifies a maximum scaling factor of 2.

example

Properties

expand all

`MaxFactor` — Maximum scaling factor
`5` (default) | positive scalar

Maximum scaling factor, specified as a positive scalar.

`Period` — Number of steps in cycle
`10` (default) | positive integer

Number of steps in the cycle, specified as a positive integer.

`StepRatio` — Ratio of time spent increasing versus decreasing learning rate
`0.5` (default) | scalar in the range (0,1)

Ratio of time spent increasing versus decreasing the learning rate, specified as a scalar in the range (0,1).

Data Types: single | double

`FrequencyUnit` — Frequency unit
`"epoch"` (default) | `"iteration"`

Frequency unit, specified as "epoch" or "iteration".

If FrequencyUnit is "epoch", then the software updates the learning rate every epoch and each iteration of the epoch uses the same learning rate. If FrequencyUnit is "iteration", then the software updates the learning rate every iteration.

`NumSteps` — Number of steps to drop learning rate
`Inf` (default)

This property is read-only.

Number of steps to drop learning rate, specified as Inf.

cyclicalLearnRate objects are infinite learning rate schedules, so NumSteps is Inf.

Data Types: double

Examples

collapse all

Create Cyclical Learning Rate Schedule

Open Live Script

Create a cyclical learning rate schedule with the default settings.

schedule = cyclicalLearnRate

schedule = 
  cyclicalLearnRate with properties:

        MaxFactor: 5
           Period: 10
        StepRatio: 0.5000
    FrequencyUnit: "epoch"
         NumSteps: Inf

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

As an alternative to creating a cyclical learning rate schedule object, to easily use a cyclical learning rate schedule with the default options, specify the LearnRateSchedule argument of the trainingOptions function as "piecewise".

options = trainingOptions("adam",LearnRateSchedule="cyclical");

Customize Cyclical Learning Rate Schedule

Open Live Script

Create a cyclical learning rate schedule that cycles for a period of 100 iterations.

schedule = cyclicalLearnRate( ...
    Period=100, ...
    FrequencyUnit="iteration")

schedule = 
  cyclicalLearnRate with properties:

        MaxFactor: 5
           Period: 100
        StepRatio: 0.5000
    FrequencyUnit: "iteration"
         NumSteps: Inf

Specify this schedule as a training option.

options = trainingOptions("adam",LearnRateSchedule=schedule);

Algorithms

expand all

Cyclical Learning Rate Schedule

A cyclical learning rate schedule periodically increases and decreases the learning rate.

The software uses this formula to calculate the learning rate:

$α = {\begin{matrix} α_{0} (1 + (γ_{N} - 1) \frac{(k - 1) \mod P}{⌈ q P ⌉}) & if (k - 1 \mod P) < ⌈ q P ⌉ \\ α_{0} (γ_{N} - (γ_{N} - 1) \frac{((k - 1) \mod P) - ⌈ q P ⌉}{P - ⌈ q P ⌉}) & if (k - 1 \mod P) \geq ⌈ q P ⌉ \end{matrix},$

where:

α₀ is the base learning rate, specified by the InitialLearnRate option of the trainingOptions function.
γ_N is the maximum scaling factor, specified by the MaxFactor argument.
P is the period, specified by the Period argument.
q is the step ratio, specified by the StepRatio argument.

If FrequencyUnit is "iteration", then the variable k denotes the schedule iteration number. The value of k can be different from the training iteration number. For example, if you train using {"warmup",schedule}, where schedule is a cyclicalLearnRate object with FrequencyUnit set to "iteration", then at training iteration 15, the cyclicalLearnRate object uses k with a value of 10 because the warm-up learning rate schedule runs for 5 training iterations.

If FrequencyUnit is "epoch", then the variable k denotes the schedule epoch number. In this case, the schedule uses the same learning rate for each iteration of the epoch. The value of k can be different from the training epoch number. For example, if you train using {warmupLearnRate(FrequencyUnit="epoch"),schedule}, where schedule is a cyclicalLearnRate object with FrequencyUnit set to "epoch", then at training epoch 15, the cyclicalLearnRate object uses k with a value of 10 because the warm-up learning rate schedule processes the first 5 epochs.

This plot shows an overview of the cyclical learning rate schedule with the default settings.

Version History

Introduced in R2024b

cyclicalLearnRate

Description

Creation

Description

Properties

`MaxFactor` — Maximum scaling factor
`5` (default) | positive scalar

`Period` — Number of steps in cycle
`10` (default) | positive integer

`StepRatio` — Ratio of time spent increasing versus decreasing learning rate
`0.5` (default) | scalar in the range (0,1)

`FrequencyUnit` — Frequency unit
`"epoch"` (default) | `"iteration"`

`NumSteps` — Number of steps to drop learning rate
`Inf` (default)

Examples

Create Cyclical Learning Rate Schedule

Customize Cyclical Learning Rate Schedule

Algorithms

Cyclical Learning Rate Schedule

Version History

See Also

Topics

cyclicalLearnRate

Description

Creation

Description

Properties

MaxFactor — Maximum scaling factor 5 (default) | positive scalar

Period — Number of steps in cycle 10 (default) | positive integer

StepRatio — Ratio of time spent increasing versus decreasing learning rate 0.5 (default) | scalar in the range (0,1)

FrequencyUnit — Frequency unit "epoch" (default) | "iteration"

NumSteps — Number of steps to drop learning rate Inf (default)

Examples

Create Cyclical Learning Rate Schedule

Customize Cyclical Learning Rate Schedule

Algorithms

Cyclical Learning Rate Schedule

Version History

See Also

Topics

`MaxFactor` — Maximum scaling factor
`5` (default) | positive scalar

`Period` — Number of steps in cycle
`10` (default) | positive integer

`StepRatio` — Ratio of time spent increasing versus decreasing learning rate
`0.5` (default) | scalar in the range (0,1)

`FrequencyUnit` — Frequency unit
`"epoch"` (default) | `"iteration"`

`NumSteps` — Number of steps to drop learning rate
`Inf` (default)