Intel Unveils AI Inference Software Optimized for Xeon CPUs

Intel on Oct. 16 unveiled Intel Inference Microservice, software designed to maximize artificial intelligence (AI) inference performance on its Xeon server central processing units (CPUs).

The containerized software enables rapid optimization of large language models (LLMs) for CPUs. It includes libraries containing the code, engines and configurations needed to run open-source LLMs.

When launched, the software detects the Intel processor and applies prevalidated and tuned AI inference settings, reducing reliance on specialists in AI inference optimization. Major AI frameworks, including the OpenAI SDK, LangChain, LlamaIndex, LiteLLM and Haystack, support the OpenAI API specification. The software uses the open-source vLLM as its default inference engine.

Optimizing existing LLMs for hardware previously took a week. Settings for computational precision, workload distribution across CPU cores, memory allocation across non-uniform memory access (NUMA) nodes and hardware instruction set selection vary depending on the combination of AI model and processor.

Intel plans to expand Inference Microservice support to its entire graphics processing unit (GPU) lineup, including the D-series GPUs designed for AI inference.