Train a visual question answering model that answers questions about images using MMF's pre-built components.
Run experiments on the Hateful Memes dataset using MMF as the official starter codebase.
Swap out individual components like model architecture or dataset loader to test new multimodal AI ideas.
Scale a multimodal AI training run across multiple GPUs or machines using MMF's distributed training support.
| facebookresearch/mmf | google/seq2seq | lucidrains/dalle-pytorch | |
|---|---|---|---|
| Stars | 5,629 | 5,629 | 5,629 |
| Language | Python | Python | Python |
| Setup difficulty | moderate | hard | hard |
| Complexity | 4/5 | 4/5 | 5/5 |
| Audience | researcher | researcher | researcher |
Figures from each repo's GitHub metadata at analysis time.
Full documentation and installation instructions are at mmf.sh, not in the repo, requires GPU for meaningful experiments.
MMF is a research framework from Facebook AI Research for building and experimenting with AI models that work with both images and text at the same time. This area of research is called multimodal AI, because it combines multiple types of input (visual and language) rather than just one. For example, a multimodal model might answer questions about a photo, generate captions for images, or detect hateful content that pairs an image with text. The framework is built on top of PyTorch, a widely used AI development library. It is designed to be modular, meaning researchers can swap out individual components like the dataset loader, the model architecture, or the training loop without rewriting everything else. It also supports distributed training, which means it can spread work across multiple machines or graphics cards to handle large experiments faster. MMF includes reference implementations of several published research models, and it has been used internally at Facebook for a number of AI research projects. It also served as the official starter codebase for several public AI challenges including the Hateful Memes challenge and the TextVQA challenge, where teams compete to build better models for understanding text inside images. The project was previously called Pythia before being renamed to MMF. Installation instructions and full documentation live at mmf.sh rather than in the repository itself. The README is brief and points to the external documentation site for most setup and usage details.
MMF is a Facebook AI research framework for building AI models that understand both images and text together, used for tasks like visual question answering and hateful content detection.
Mainly Python. The stack also includes Python, PyTorch.
Open source under a permissive license, check mmf.sh for the specific terms.
Setup difficulty is rated moderate, with roughly 1h+ to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
Verify against the repo before relying on details.