Apple's New Video Language Model Outperforms Larger Models

Apple has introduced SlowFast-LLaVA-1.5, a new family of video large language models (LLMs) designed for efficient long-form video understanding, according to a study published on arXiv. The model employs a two-stream SlowFast mechanism, which allows it to efficiently process long-range temporal contexts while maintaining high computational efficiency.

The SlowFast-LLaVA-1.5 model family ranges from 1 billion to 7 billion parameters and has been optimized through a streamlined training pipeline using publicly available datasets. This approach enables the models to outperform much larger models across various video tasks, achieving state-of-the-art results on benchmarks like LongVideoBench and MLVU.

Apple's model addresses several limitations of existing video LLMs, such as reliance on long context windows and complex training pipelines. By integrating video perception into pre-trained LLMs, SlowFast-LLaVA-1.5 not only excels in video tasks but also performs well on image tasks, including benchmarks for knowledge and math reasoning. The model is open-source and available on platforms like GitHub and Hugging Face.