Encyclopedia

… the photonics community’s trusted resource!

Transverse Mode Instability of Fiber Amplifiers: Now Covered in Encyclopedia, Case Study and RP Fiber Power Software

Posted on 2026-10-19 as part of the Photonics Spotlight (available as e-mail newsletter!)

Permanent link: https://www.rp-photonics.com/spotlight_2026_10_19.html

Author: Dr. Rüdiger Paschotta, RP Photonics AG

Abstract: Transverse mode instability (TMI) is one of the main limitations of high-power fiber amplifiers. We have now dealt with it extensively, resulting in a comprehensive encyclopedia article, a detailed case study and an extension of our simulation software RP Fiber Power for fast estimates of TMI thresholds.

Content quality and neutrality are maintained according to our editorial policy.

Since about 2010, it has become clear that the output power of single-mode fiber amplifiers is limited not only by nonlinear optical effects, but also by transverse mode instability (TMI): Above a certain threshold power, the beam quality suddenly degrades, as power is transferred from the fundamental mode to a higher-order mode, with fluctuations on millisecond time scales. For anyone developing kilowatt-class fiber amplifiers, this is a central issue.

Users of our simulation software RP Fiber Power occasionally asked whether it can simulate TMI. For a long time, the honest answer was essentially “no”. But we now decided to deal with the topic extensively. The results are a comprehensive encyclopedia article, a detailed case study and a substantial extension of the software.

Why It Took a While

For several reasons, TMI is a complicated issue:

  • It involves many interacting effects: the laser gain of the fundamental and higher-order modes and its saturation, the heat generation in the doped core, heat diffusion, the thermo-optic effect, and the guiding properties of the fiber. None of these is difficult for us in isolation, but their interplay is subtle. For example, what matters is not the heat load itself, but how strongly the heat generation is modulated by the interference pattern of the modes – and that modulation is strongly reduced by gain saturation.
  • Full dynamic simulations of TMI are very demanding: They need to resolve frequency offsets of a few kilohertz between the modes – compared with optical frequencies of hundreds of terahertz – and processes on millisecond time scales. Such a simulation can take hours for a single configuration, which is rather inconvenient for exploring design options.
  • Finally, the scientific literature is not easy to digest. Two main pictures of the mechanism have been developed: One describes TMI as stimulated thermal Rayleigh scattering, with a frequency-shifted higher-order mode amplified by a moving thermal grating. The other, developed at the Friedrich Schiller University Jena, emphasizes phase shifts between the modal intensity pattern and the thermal grating, which can be caused e.g. by pump power fluctuations. Both essentially describe the same underlying physics, but emphasize different ways in which the instability is initiated. In addition, different papers use different conventions, and signs and factors of 2 are easily confused.

As an inaccurate model may do more harm than good, careful validation was essential.

Our Approach: Fast Threshold Estimates

Instead of a full dynamic simulation, we decided to use a linear stability analysis of the steady state of the amplifier: Based on the calculated distributions of optical powers and excitation, the software calculates the heat modulation caused by a weak higher-order mode, the resulting thermal refractive index grating, and the amplification of the higher-order mode as a function of its frequency offset. The threshold is the pump power (or signal output power) at which a weak initial perturbation (e.g. from quantum noise) grows to a significant power fraction at the fiber end.

Such an estimate takes only a few seconds. Comparisons with published model calculations show deviations of up to about 20%, which is quite satisfactory considering the uncertainties of some input parameters. And since the calculation is fast, one can easily explore how the threshold depends on many parameters – which is highly useful for device design.

What Is New

Encyclopedia article: The new article on transverse mode instability explains in depth the observed behavior and the physical mechanism, including the relation between the different pictures, and the role of gain saturation. It discusses the many factors influencing the threshold and the known mitigation methods, the special situations in fiber lasers and pulsed amplifiers, and the modeling of TMI. We believe that it will help many physicists and engineers to gain a proper overview on this field of research and its important engineering consequences.

Case study: Our new case study works through a concrete example: an ytterbium-doped amplifier with a 25 µm core, delivering about 1 kW of output power.

Case Studies

Transverse Mode Instability in a Kilowatt Fiber Amplifier

We find out up to which output power an ytterbium-doped fiber amplifier with 25 µm core can be operated before transverse mode instability sets in, which part of the fiber is responsible, and which measures really help – with some surprises.

#amplifiers#TMI#high power

Software: RP Fiber Power has been extended in several ways:

  • An include file for TMI threshold estimates can be used in your own scripts. It is documented in detail, including the underlying equations, the numerical methods, the validation and the limitations.
  • Two demo files using the include file show its application to ytterbium-doped and thulium-doped fiber amplifiers, including a visualization of the thermal index grating and an interesting parameter study.
  • The Power Forms for continuous-wave fiber amplifiers and for fiber lasers with amplifiers now offer an optional TMI threshold estimate for each amplifier stage – just tick a checkbox and enter a few additional parameters, such as the numerical aperture of the core. Background absorption, e.g. from photodarkening, can also be taken into account.
thermal refractive index grating in a fiber amplifier
Figure 1: Calculated thermal refractive index grating in an xz cut through the fiber of the case study. The index maxima (red) lag slightly behind the maxima of the modal intensity pattern (black lines) – the key to the power transfer.

Some Interesting Findings

Working with the model gave us a number of interesting insights. Here are a few of them, all from the case study, and well in line with various reports in the literature:

  • The instability builds up mainly in a relatively short part of the fiber near the input – not where the heat load is highest, and not near the output where the signal power is highest. In the example, the higher-order mode power grows by 13 orders of magnitude within the first 1.5 m, and only by 4 more orders within the remaining 2.5 m, where the heat load is still substantial.
LP01 and LP11 powers versus position
Figure 2: Powers of LP01 and LP11 versus position, for 1 kW pump power and at the TMI threshold.
  • Gain saturation is the key to understanding many dependencies. A higher signal input power, (partial) counter-pumping and pumping away from the absorption peak at 976 nm all raise the threshold, essentially by strengthening the saturation. For example, pumping at 915 nm or 981 nm instead of 976 nm (with correspondingly longer fiber) raises the threshold by a factor of 1.75 or 2.4, respectively – in line with experimental findings.
  • A background absorption of only 0.01 dB/m, e.g. from photodarkening, lowers the threshold by about 20%. The reason is that the heat from absorption follows the intensity pattern fully, while the modulation of the quantum defect heating is strongly reduced by gain saturation.
  • Additional losses of the higher-order mode, e.g. from fiber bending, help less than one might expect: 10 dB/m raises the threshold by only 25%.
  • Spreading the heat over a longer fiber with lower doping concentration (with the same pump absorption) does not help at all, although it halves the heat load per unit length.
  • The calculated safety margin of an amplifier operated somewhat below the threshold may be smaller than it appears: In the example, it corresponds to a factor of only about 7000 in the power of the initial perturbation. Pump intensity noise in the kilohertz range, acting on a small static higher-order mode content from the launch, can provide such stronger seeds.

Of course, such results are estimates, with uncertainties e.g. concerning the actual noise level and the losses of the higher-order mode. As usual, trends found when comparing design options are more reliable than absolute values. In practice, a measured threshold of an existing amplifier can be used for calibrating uncertain parameters; the model then predicts the effects of design changes.

How to Get It

The TMI extension is available for all versions of RP Fiber Power. If you are an existing user, simply notify us, and you get a free update. Note that the Power Forms, which offer the most convenient access to the TMI estimates, are available only in V8. (If you don't have that yet, consider an upgrade!)

Feedback is always welcome – in particular, experience with measured TMI thresholds, which can help to further test and calibrate the model. Just use our software inquiries page or send me a LinkedIn message.


This article is a posting of the Photonics Spotlight, authored by Dr. Rüdiger Paschotta. You may link to this page and cite it, because its location is permanent. See also the RP Photonics Encyclopedia.

Note that you can also receive the articles in the form of a newsletter or with an RSS feed.

Questions and Comments from Users

Here you can submit questions and comments. As far as they get accepted by the author, they will appear above this paragraph together with the author’s answer. The author will decide on acceptance based on certain criteria. Essentially, the issue must be of sufficiently broad interest.

Please do not enter personal data here. (See also our privacy declaration.) If you wish to receive personal feedback or consultancy from the author, please contact him, e.g. via email.

Spam check:

By submitting the information, you give your consent to the potential publication of your inputs on our website according to our rules. (If you later retract your consent, we will delete those inputs.) As your inputs are first reviewed by the author, they may be published with some delay.