The modes of some optical resonator (resonant cavity) or the propagation modes of a waveguide can have different transverse intensity profiles and propagation constants, and this aspect is emphasized by the term transverse modes (or lateral modes). Those modes with the simplest intensity profile are called fundamental modes, and all others are higher-order transverse modes. The fundamental mode of a waveguide is the one with the highest effective refractive index.
Not all waveguides support higher-order modes; they are then called single-mode waveguides (e.g. single-mode fibers).
Laser oscillation on higher-order modes as well as lights transmission in higher-all modes of the waveguide are associated with decreased beam quality.
The article on modes gives more details.