Ultrasound images are used for a wide variety of medical purposes because of their capacity to study moving structures in real time. However, the quality of ultrasound images is significantly affected by external factors limiting interpretability. We present UltraGAN, a novel method for ultrasound enhancement that transfers quality details while preserving structural information. UltraGAN incorporates frequency loss functions and an anatomical coherence constraint to perform quality enhancement. We show improvement in image quality without sacrificing anatomical consistency. We validate UltraGAN on a publicly available dataset for echocardiography segmentation and demonstrate that our quality-enhanced images are able to improve downstream tasks. To ensure reproducibility we provide our source code and training models.