What is asr-evaluation?

ASR Evaluation Framework is a benchmarking tool for comparing automatic speech recognition (ASR) models, software that converts spoken audio into text. If you are trying to decide which speech-to-text model to use in an application, this framework runs them all on the same audio samples and measures how well each one performs, how fast it is, and how well it holds up under difficult conditions like background noise or accented speech. The framework supports five models: IBM Granite, OpenAI Whisper, NVIDIA Canary, Distil-Whisper, and Wav2Vec2. For each model, it calculates Word Error Rate (the percentage of words the model got wrong), Character Error Rate (a finer-grained version of the same idea), overall accuracy, and Real-Time Factor (how long transcription takes relative to the length of the audio, a value below 1.0 means the model can keep up in real time). Tests cover over 15 scenarios including clean speech, office or street background noise, fast or slow speakers, technical vocabulary, and phone-quality audio. You run it from the command line with flags like --accuracy, --speed, or --all. Results are saved as structured JSON files with a consistent schema, making it easy to compare runs or import results into another tool. The code is written in Python and integrates with the Hugging Face ecosystem for loading models. You would use this before deploying a voice feature to check which model gives the best tradeoff between speed and accuracy for your specific use case. The full README is longer than what was provided.