The number of guided modes of a waveguide (for example, an optical fiber) depends on the optical wavelength: the shorter the wavelength, the more modes can be guided. For long wavelengths, there may be only a single guided mode (→ single-mode fibers) or even none at all, whereas multimode behavior is obtained at shorter wavelengths.
When a particular mode ceases to exist beyond a certain wavelength, that wavelength is called its cut-off wavelength. For an optical fiber, the cut-off wavelength for the LP11 mode sets a limit to the single-mode regime, as below that wavelength there is at least the LP01 and the LP11 mode.
Just below the cut-off wavelength, the mode properties often vary substantially. Typically, the mode radius (and thus the effective mode area) increases sharply near the cut-off, and the fraction of power propagating within the waveguide core decreases accordingly. That effect is shown in Figure 1 for a multimode step-index fiber; similar behavior occurs for fibers with other transverse refractive index profiles.
For LPlm modes of a fiber, only for l = 0 the fraction of the power guided in the core goes to zero when approaching the cut-off. For modes with higher l, the mode size stays finite there.
Fibers with not radially symmetric designs (and strongly bent fibers) can have polarization-dependent cut-off wavelengths.
Just below its cut-off wavelength, the bend losses of a mode can become very high due to the increased mode area. Therefore, even for moderate bending of the fiber one may obtain sharply increasing propagation losses near the cut-off wavelength. Therefore, cut-off wavelengths can not always be precisely determined in experiments.