●Change8

b9931

📦 llama-cppView on GitHub →
✨ 2 features🔧 2 symbols

Summary

This release introduces significant performance optimizations for Mixture-of-Experts (MoE) prefill on OpenCL via ragged-tile FP16 GEMM skipping and quarter-granularity tile-skipping.

Migration Steps

  1. To opt out of the new OpenCL MoE ragged FP16 GEMM optimization, set the environment variable GGML_OPENCL_MOE_RAGGED_FP16=0.
  2. To change the granularity for the new OpenCL quarter-granularity ragged MoE tile-skip, set the environment variable GGML_OPENCL_MOE_RAGGED_GRAN={8,16,32} (default is quarter).

✨ New Features

  • Implemented FP16 GEMM optimization for ragged-tile MoE prefill on OpenCL by skipping padded expert tiles.
  • Introduced quarter-granularity ragged MoE tile-skip (8-column skip-groups) for OpenCL MoE GEMMs, allowing independent skipping of empty trailing skip-groups.

Affected Symbols

opencl MoE GEMMscl_init