What is fetchit?

FetchIt is a command line tool that downloads web based documentation and combines it into a single clean Markdown file, or optionally a matching folder tree of Markdown files. Instead of manually copying pages from a documentation website, a person points FetchIt at a starting URL and it crawls the linked pages, cleans them up, and writes the result to a file the user chooses. While crawling, FetchIt strips out page headers, footers, sidebars, and navigation menus so only the actual content remains, then converts the leftover HTML into standard Markdown formatting. It downloads several pages at the same time to speed things up, and it can save images from the pages onto the local disk while updating the links to point at those saved copies, or it can skip saving images and keep the original web links instead. It also tries to detect which programming language a code block is written in automatically, and it can save its progress partway through a large crawl so the job can be resumed later instead of starting over. Under the hood, FetchIt combines Python and C++. A small C++ extension handles fast parsing of sitemaps and quick URL checks, while Python code manages the network requests, retries, and the actual HTML to Markdown conversion. The C++ part compiles automatically the first time the tool is run, so most users do not need to think about it directly. Installation is available through pip as fetchit-md, through the Arch Linux AUR, through a system-wide install using make, or inside a local Python virtual environment. Running it requires Python 3.8 or newer along with a C++ compiler such as GCC or Clang and the make utility. Once installed, a user runs the fetchit command with a starting URL and options like output file name, how many pages to fetch at once, how many times to retry a failed page, and whether to save output as one file or as a directory tree of files. A typical example points FetchIt at a documentation site such as Android's Compose docs, tells it to fetch 20 pages at a time, and saves the combined result to a single Markdown file along with a local images folder. The README does not mention a software license.