Advanced Digital Signal Processing For Next-Generation Coherent Optical Communication Systems