Ciyar da AI gaba da LLM multimodal: darussa daga MM1

Tutar Apple MM1 .class="img-fluid clearfix"

Gabatarwa

Haɗin sarrafa harshe na halitta da gane hoto ya samar da LLM multimodal. A paper ɗin MM1, Apple ya gabatar da jerin model na AI da ke haɗa fahimtar gani da fahimtar harshe. Binciken ya gwada zaɓin architecture, cakuda pre-training data, da abubuwan da ke ƙayyade performance.

Muhimmancin MM1 ba wai demo ba ne. Muhimmancinsa shi ne ya nuna wane ɓangare na tsarin model ne ke da tasiri, yadda ake shirya data, da wane trade-off injiniya ya kamata a auna.

.class="m-10 w-100"

Tasowar AI multimodal

AI ya ci gaba a manyan hanyoyi biyu: fahimtar harshe da fahimtar hoto. LLM sun sauya yadda machine ke fahimta da rubuta harshe. Computer vision kuma ya inganta yadda machine ke fitar da ma’ana daga hoto. LLM multimodal yana haɗa waɗannan abubuwa biyu domin model ya yi aiki da rubutu da hoto a lokaci guda.

Wannan ya buɗe hanya ga assistants masu fahimtar allo, nazarin takardu, kayan koyarwa, visual search, da ƙirƙirar abun multimedia. Amma matsalar ba wai karɓar hoto kawai ba ce. Matsalar ita ce haɗa visual representation da language model cikin hanyar da za ta ba da sakamako mai ma’ana.

.class="m-10 w-100"

Binciken MM1: muhimmin mataki a AI multimodal

Binciken MM1: Methods Analysis & Insights from Multimodal LLM Pre-training ⧉ ya zama muhimmin abin dubawa wajen fahimtar MLLM pre-training. Masu binciken Apple sun gwada image encoder, vision-language connector, image resolution, da data composition.

Hanya da manufofi

MM1 ya yi amfani da gwaji mai tsauri. Masu binciken sun gwada architecture daban-daban da cakuda data daban-daban, sannan suka auna tasirinsu kan few-shot learning. Wannan yana da muhimmanci saboda a amfani na zahiri ba koyaushe ake da manyan labelled examples ba.

Manufar ita ce gano design da zai ba model damar koyo daga ƙaramin misali, ya tsaya daidai, kuma ya haɗa visual context da umarnin harshe.

.class="m-10 w-100"

Sakamako da darussa

Darasi na farko shi ne data mix yana da muhimmanci. Haɗa image-caption data, interleaved image-text data, da text-only data ya ba da sakamako mafi kyau. Model yana bukatar nau’in data daban-daban domin ya koyi alaƙa tsakanin abu a hoto, mahallin takarda, da umarnin harshe.

Darasi na biyu shi ne scale ba parameter count kawai ba ne. MM1 ya gwada dense models har zuwa 30B parameters da mixture-of-experts variants. Amma binciken ya nuna image resolution na iya yin tasiri fiye da girman model. A multimodal AI, ingancin visual input wani ɓangare ne na performance.

Image encoder ma yana da nauyi. ResNet ko ViT suna shafar yadda model ke fitar da visual features. Vision-language connector kuma yana haɗa features ɗin da language model domin su zama context mai amfani.

.class="m-10 w-100"

Gine-ginen MM1 da tsarin multimodal learning

Gine-ginen model MM1 .class="m-10 w-100"

Diagram ɗin yana nuna tsarin MM1. Image input yana shiga Image Encoder. Text input yana shiga pre-trained LLM transformer. Visual features suna wucewa zuwa VL Connector, wanda ke haɗa su da textual representation. Wannan multimodal fusion yana ba model damar yin visual question answering da captioning bayan supervised fine-tuning.

Pre-training data ya ƙunshi 45% interleaved data, 45% captions, da 10% text-only data. Wannan ya nuna cewa multimodal learning ba ƙara hoto ga language model kawai ba ne; tsara data yana cikin architecture.

.class="m-10 w-100"

MM1 a matsayin benchmark na AI multimodal

MM1 benchmark ne saboda yana gwada shawarar architecture da ta dace da amfani na gaske. Model ɗin ya nuna ƙarfi a visual question answering, image captioning, da aikin da ke buƙatar fahimtar hoto tare da harshe.

Ƙarfinsa shi ne samar da rubutu mai ma’ana daga visual input. Idan aka ba shi hoton titi mai cunkoso, zai iya bayyana yanayi, gine-gine, mutane, da ayyuka. Wannan shi ne ainihin darajar multimodal AI: fahimtar context, ba gane object kaɗai ba.

Tasiri da gaba

MM1 ya ba da tushe ga MLLM architecture mafi ƙarfi. A gaba, ana bukatar connector mai daidaitawa, attention mai inganci, da evaluation mafi tsauri don yanayin amfani na zahiri.

Mu ƙirƙiri gobe maimakon damuwa da jiya. — Steve Jobs

Aikace-aikace sun haɗa da assistants masu fahimtar allo, kayan koyarwa, nazarin takardu, da ƙirƙirar abun ciki. Amma model multimodal ya fi wahalar tantancewa. Ƙarin modality yana nufin ƙarin aikin validation.

Babban mataki na gaba a AI shi ne machine da ke fahimtar duniya a kusa da su sosai, har ma su iya yin reasoning kan data da ba su taɓa gani ba. — Yann LeCun

.class="m-10 w-100"

Kammalawa

MM1 muhimmin bincike ne a ci gaban LLM multimodal. Ya nuna architecture, data quality, image resolution, da vision-language connector suna ƙayyade ikon model. Ba girman model kaɗai ne amsa ba; dole a auna data pipeline da haɗin modalities.

Model irin MM1 na iya sa hulɗar mutum da machine ta zama mafi halitta. Amma hakan yana bukatar injiniya mai tsari, evaluation, da governance.

Don karanta asalin paper, duba: MM1: Methods Analysis & Insights from Multimodal LLM Pre-training ⧉

Bita ta ƙarshe 2026-07-16.

Sake buga wannan labari

Ciyar da AI gaba da LLM multimodal: darussa daga MM1 — Sebastien Rousseau

Nazari kan binciken MM1 na Apple: LLM multimodal, gine-gine, dabarun pre-training, ingancin hoto, da ikon few-shot.

An lasisin wannan labari a karkashin Creative Commons Attribution 4.0 International. Sake bugawa na bukatar nuna asalin URL na asali.

Ciyar da AI gaba da LLM multimodal: darussa daga MM1 — Sebastien Rousseau

Nazari kan binciken MM1 na Apple: LLM multimodal, gine-gine, dabarun pre-training, ingancin hoto, da ikon few-shot.

Originally published at https://sebastienrousseau.com/ha/2024-03-18-advancing-ai-with-multimodal-llms-insights-from-mm1/ by Sebastien Rousseau.
Licensed under CC-BY-4.0.

SEBASTIEN ROUSSEAU FOUNDER · ENGINEER