What is diffsinger?

DiffSinger is a research project that teaches a computer to generate singing voices from lyrics and musical notes. Given a piece of sheet music or MIDI input along with the words to be sung, the system produces an audio output that sounds like a human singer. The same underlying approach also powers DiffSpeech, a companion system that converts plain text into spoken audio without any musical component. The core idea behind the project is a technique called shallow diffusion. Diffusion models work by starting from noise and gradually refining it into something meaningful, which can produce high-quality results but tends to be slow. Shallow diffusion is a shortcut: instead of starting from pure noise, the process begins partway through using a simpler model's output as a starting point. The paper describing this approach was accepted at AAAI 2022, a major academic conference for artificial intelligence research. The system processes audio in stages. First it converts lyrics and pitch information into an intermediate representation called a mel spectrogram, which is a way of visualizing sound frequencies over time. A separate component called a vocoder then turns that representation into actual audio waveforms. Several vocoder options are supported depending on whether music or speech is being generated. The repository is the official code release from the paper's authors, written in Python using PyTorch. Running it requires a machine with an NVIDIA GPU, and the setup instructions list specific CUDA versions for different GPU models. Live demos are available on Hugging Face where anyone can try the singing and speech synthesis in a browser without installing anything. A community-maintained fork called DiffSinger by Team Openvpi has continued development beyond this original research release, and the README points to that project for users who want more actively updated software.