whatisgithub

What is xformers?

facebookresearch/xformers — explained in plain English

Analysis updated 2026-06-24

★ 10,456PythonAudience · researcherComplexity · 4/5LicenseSetup · hard

In one sentence

A Python library from Meta with optimized GPU building blocks for Transformer models, offering memory-efficient attention and other components that make AI models faster and less memory-hungry than standard PyTorch implementations.

Mindmap

mindmap
  root((xFormers))
    What it does
      Faster Transformers
      GPU memory savings
      Drop-in components
    Key components
      Memory-efficient attention
      Layer normalization
      SwiGLU activation
      Fused linear layer
    Requirements
      PyTorch
      Nvidia GPU CUDA
      Linux or Windows
    Use cases
      Language models
      Vision Transformers
      Research experiments
Click or tap to explore — scroll the page freely

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

What do people build with it?

USE CASE 1

Drop xFormers' memory-efficient attention into an existing PyTorch Transformer model to cut GPU memory usage on long sequences.

USE CASE 2

Speed up training of a large language model or vision Transformer by replacing standard attention with optimized GPU kernels.

USE CASE 3

Use the SwiGLU or fused linear layer components to reduce compute cost in a custom model architecture.

What is it built with?

PythonPyTorchCUDAC++

How does it compare?

facebookresearch/xformersmegvii-basedetection/yoloxlivekit/agents
Stars10,45610,45710,463
LanguagePythonPythonPython
Setup difficultyhardmoderatemoderate
Complexity4/53/53/5
Audienceresearcherresearcherdeveloper

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · hard Time to first run · 1h+

Requires a CUDA-capable Nvidia GPU, AMD support is experimental, PyTorch version must match the prebuilt package exactly or you must compile from source.

BSD-style license, use, modify, and distribute freely including for commercial purposes, with attribution to the original authors.

So what is it?

xFormers is a Python library from Meta's research team that provides building blocks for working with Transformer models, which are the architecture behind most modern AI language and vision systems. The library is aimed at researchers and engineers who want to experiment with or optimize these models without being limited to what ships in standard tools like PyTorch. The core value the library offers is speed and memory efficiency. Transformer models, especially large ones, require a lot of GPU memory and computation. xFormers includes custom GPU code (called kernels) that makes certain operations faster or less memory-hungry than the standard implementations. The most prominent example is its memory-efficient attention operation, which the README claims can be up to 10 times faster than a standard approach while still producing exact results, not an approximation. Beyond attention, the library includes optimized versions of other common operations used inside these models: layer normalization, dropout combined with activation functions, a fused linear layer, and a component called SwiGLU used in some newer architectures. These components are designed to be used independently, so you can drop one into an existing project without having to adopt the whole library. Installation requires a compatible version of PyTorch and a CUDA-capable GPU (NVIDIA hardware on Linux or Windows). AMD GPU support is listed as experimental. The library is also available from source if you need to pair it with a specific PyTorch version not covered by the prebuilt packages. The project is primarily a research tool and is used across both language and vision work at Meta. It carries a BSD-style open-source license and includes attribution to several other open-source projects whose code or ideas it builds on.

Copy-paste prompts

Prompt 1
I have a PyTorch Transformer model that runs out of GPU memory on long sequences. Show me how to replace the standard attention layer with xFormers' memory-efficient attention.
Prompt 2
How do I install xFormers to match my specific PyTorch and CUDA version, and verify it's actually using the compiled GPU kernels and not a slow fallback?
Prompt 3
Write me a short benchmarking script that compares the speed and peak memory of standard PyTorch attention versus xFormers attention on my GPU.
Prompt 4
Help me integrate the xFormers SwiGLU activation into my custom feedforward layer in a PyTorch Transformer model.

Frequently asked questions

What is xformers?

A Python library from Meta with optimized GPU building blocks for Transformer models, offering memory-efficient attention and other components that make AI models faster and less memory-hungry than standard PyTorch implementations.

What language is xformers written in?

Mainly Python. The stack also includes Python, PyTorch, CUDA.

What license does xformers use?

BSD-style license, use, modify, and distribute freely including for commercial purposes, with attribution to the original authors.

How hard is xformers to set up?

Setup difficulty is rated hard, with roughly 1h+ to a first successful run.

Who is xformers for?

Mainly researcher.

Open on GitHub → Ask about another repo

This repo across BitVibe Labs

Verify against the repo before relying on details.

What are similar repos?