Tux Machines

Inference is 3x Faster in Linux than in Windows

Posted by Roy Schestowitz on Oct 04, 2024

Recently, a developer shared how he switched from Windows to Linux after 30 years and saw remarkable results for AI-specific tasks. The person, who goes by the name Inevitable-Start-653, mentioned that he had six 24GB graphics cards, pushing the limits of what’s typically used in consumer-grade setups.

As more GPUs were added to the system, the performance hit on Windows became increasingly noticeable. Despite using top-notch inferencing software like Oobabooga’s Textgen, the Windows operating system’s overhead proved to be a significant bottleneck.

Read on