Arxiv paper - MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

In this episode, we discuss MetaMorph: Multimodal Understanding and Generation via Instruction Tuning by Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, Zhuang Liu. The paper introduces Visual-Predictive Instruction Tuning (VPiT), which enhances pretrained large language models to generate both text and visual tokens by training on mixed image and text data. The study finds that visual generation naturally arises from improved visual understanding and that understanding data is more effective than generation data for enhancing both capabilities. Using VPiT, the authors develop the MetaMorph model, which achieves strong performance in visual understanding and generation by leveraging the inherent vision capabilities of language models through simple instruction tuning.