Audio Editing in the Era of
Foundation Models: A Survey

AACL-IJCNLP 2026

Changhao Pan1,*, Yifei Fan1,*, Fan Zhuo1,*, Yifu Chen1, Wenxiang Guo1, Yu Zhang2, Ruiqi Li2, Zhiyuan Zhu1, Rui Yang1, Shengpeng Ji3, Chenyuhao Wen1, Jiayang Xu1, Ke Lei1, Xiaoda Yang1, Jingyu Lu1, Zhou Zhao1,†
1 Zhejiang University
2 ByteDance
3 Hunyuan Team, Tencent

* Equal contribution   † Corresponding author

Star us on GitHub Paper arXiv BibTeX

We present a survey of foundation-model-based audio editing across speech, music, and general audio. We connect editing goals with model architectures, training methods, and publicly available resources, with a unified view of what an edit changes and what it should preserve.

  • Task taxonomy. We organize audio editing into acoustic, semantic, and instance editing, clarifying the targets and preservation requirements of each task.
  • Model architectures. We review codec language models, diffusion models, and flow-matching models, linking their representations and generation mechanisms to editing capabilities.
  • Training methods. We distinguish methods that learn editing from data from training-free methods that steer pretrained generators without parameter updates.
  • Public resources. We curate released editing models, datasets, generation and annotation tools, benchmarks, and metrics to support research and implementation.

Explore Audio Editing

Our taxonomy distinguishes changes to acoustic attributes, interpretable content or expression, and individual sound sources. The same categories apply across speech, music, and general audio.

Figure 1. Taxonomy of audio editing tasks. Select the figure to enlarge it.

Select an operation to listen to its audio editing samples.

Audio Examples

Choose an audio domain and compare each input with the editing instruction and output.

Loading audio examples…

Methods at a Glance

Model architectures

  • Codec language models represent audio as discrete tokens and edit through conditional prediction, infilling, or selective regeneration. .
  • Diffusion models iteratively denoise audio representations, using conditioning, inversion, attention control, or masks to guide the edit. .
  • Flow-matching models learn continuous transport toward audio; conditioning or trajectory manipulation supports reconstruction and editing. .

These architectures can be combined, for example by using a language model to condition a diffusion or flow-matching decoder.

Resource Explorer

Browse the public resources curated in MM-Speech/AudioEditSurvey. Select a resource type, filter its entries, and click a column heading to sort the table.

Loading resources…

Catalog snapshot: · Download all resources

Open Challenges and Future Directions

Foundation-model-based audio editing still faces several system-level challenges:

  1. Complex editing. Real-world audio entangles semantic events, speaker identity, acoustic attributes, background ambience, rhythm, spatial cues, and reverberation. Future systems should support precise source/event localization, attribute-level modification, and reliable preservation of non-target content across speech, music, and general audio.
  2. Robustness in open-domain settings. Editing models should remain reliable under noise, reverberation, overlapping sources, long-form context, and ambiguous instructions. Better instruction grounding, long-context modeling, iterative refinement, and self-verification are important directions toward robust real-world editing.
  3. Faithful and editing-specific evaluation. Existing evaluation often mixes generation quality with editing quality. Future benchmarks should explicitly annotate edit targets, operations, preservation regions, and relevant control signals, allowing edit success and non-target preservation to be evaluated separately.
  4. Safety, copyright, and misuse prevention. Modern editing systems can realistically modify speech content, speaker identity, emotion, environmental sounds, and music. Practical deployment therefore requires complementary mechanisms for provenance, watermarking, manipulated-audio detection, and responsible data licensing.

BibTeX

If you find this survey or repository useful, please cite our paper:

@article{pan2026audio,
  title={Audio Editing in the Era of Foundation Models: A Survey},
  author={Pan, Changhao and Fan, Yifei and Zhuo, Fan and Chen, Yifu and Guo, Wenxiang and Zhang, Yu and Li, Ruiqi and Zhu, Zhiyuan and Yang, Rui and Ji, Shengpeng and others},
  journal={arXiv preprint arXiv:2606.23139},
  year={2026}
}

Contributing

This repository is meant to keep growing. If an audio editing model, dataset, tool, benchmark, or metric is missing, please open an issue or a pull request.

Survey figure