AbstractThe modulation and demodulation methods of the optical Differential Quadrature Phase Shift Keying (DQPSK), Optical Differential Phase Shift Keying (DPSK) and the different modulation formats including nonreturn-to-zero (NRZ), return-to-zero (RZ), carrier-suppressed return-to-zero (CS-RZ) are introduced, studying the ability of anti-dispersion for different optical signals in the 40 Gb/s high speed transmission system. The system is simulated in optical fiber by way of dispersion compensation. And then analyzed the compensation effect of the different dispersion compensation method for different phase modulation formats. It is shown that the ability of anti-dispersion and anti-PMD is better in the CS-RZ-DQPSK modulation format, and this format has the smallest eye-opening penalty (EOP) with a wider range of power into the optical fiber.