使用 Transformers 中的 IDEFICS 完成图像理解与多模态推理任务实战指南【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformers导读本指南聚焦于 Hugging Face Transformers 中的 IDEFICS——一个基于 Flamingo 架构的开放访问视觉-语言大模型讲解如何在不微调的前提下直接用它完成图像描述、视觉问答、图像分类、Few-shot 提示、图像引导文本生成及多轮对话等任务。读完本文你将掌握 IDEFICS 的加载含 4-bit 量化加载、Processor 的使用方式、generate的实用参数配置以及批量和会话场景下的完整调用方案。IDEFICS 是 DeepMind Flamingo 论文的开源复现属于大规模预训练模型直接用于多任务路线的代表性模型它接受任意图像序列与文本作为输入输出连贯的文本可用于回答图像相关问题、描述视觉内容、基于多张图片创作故事等。目前 Hub 上提供两个规模的检查点800 亿参数与90 亿参数且都配有针对对话场景微调过的 instruct 版本。模型用途广泛但参数量庞大需要相当的算力与基础设施是否选择大模型直用而非任务专用小模型微调取决于你的具体使用场景。环境准备开始前请确认已安装所需依赖pip install -q bitsandbytes sentencepiece accelerate transformerstransformers模型与处理器实现accelerate支持device_mapauto的设备自动调度bitsandbytes4-bit 量化加载sentencepieceIDEFICS 文本端使用的 Llama 风格分词器依赖。注意使用非量化检查点运行本文示例至少需要20GB 可用 GPU 显存9B 模型即已如此80B 模型要求更高。加载模型IdeficsForVisionText2Text与AutoProcessor首先指定 9B 检查点 checkpoint HuggingFaceM4/idefics-9b与 Transformers 中其他多模态模型一致需要同时加载处理器Processor与模型本体。IDEFICS 的处理器将 [LlamaTokenizer] 与 IDEFICS 图像处理器封装在同一个Processor中统一负责文本与图像输入的预处理。从源码看该封装位于 processing_idefics.py 中的IdeficsProcessor类继承自ProcessorMixin它内部维护了image图像占位 token 的 ID并在处理过程中把图片 URL 或PIL.Image对象转换成pixel_values等模型可消费的输入。 import torch from transformers import IdeficsForVisionText2Text, AutoProcessor processor AutoProcessor.from_pretrained(checkpoint) model IdeficsForVisionText2Text.from_pretrained(checkpoint, dtypetorch.bfloat16, device_mapauto)dtypetorch.bfloat16以 bfloat16 半精度加载权重显著降低显存占用device_mapauto由accelerate自动决定权重在各设备GPU/CPU上的最优放置与加载方式。IdeficsForVisionText2Text在 modeling_idefics.py 中定义它由IdeficsModel主干与lm_headIdeficsDecoupledLinear输出层构成。该实现还继承了GenerationMixin因此可以直接调用model.generate(...)。值得注意的实现细节是当配置了additional_vocab_size用于image等额外 token时语言模型头与embed_tokens之间采用解耦权重绑定见_tied_weights_keys额外词表始终可训练。4-bit 量化加载显存紧张时可以加载模型的量化版本把BitsAndBytesConfig传给from_pretrained在加载过程中就地压缩权重。 import torch from transformers import IdeficsForVisionText2Text, AutoProcessor, BitsAndBytesConfig quantization_config BitsAndBytesConfig( ... load_in_4bitTrue, ... bnb_4bit_compute_dtypetorch.float16, ... ) processor AutoProcessor.from_pretrained(checkpoint) model IdeficsForVisionText2Text.from_pretrained( ... checkpoint, ... quantization_configquantization_config, ... device_mapauto ... )4-bit 量化将权重压缩到 4 bit 存储、以 float16 精度计算可大幅降低显存需求是在消费级显卡上运行 9B 多模态模型的主要手段。load_in_4bitTrue需配合bitsandbytes库使用量化加载通常要求搭配device_map进行设备放置。输入格式文本与图像的混合 PromptIDEFICS 的输入是文本 图片交错组成的 Prompt 列表这是其架构的关键用法。图片可以以两种形式出现在列表中图片 URL 字符串模型可在线抓取PIL.Image图像对象。处理器在遇到图片时会自动在序列中注入fake_token_around_imageimagefake_token_around_image标记把文本 token 与视觉特征连接起来该行为由IdeficsProcessor.__call__实现见 processing_idefics.py。处理后的输入包含input_ids、attention_mask、pixel_values、image_attention_mask等条目可直接传给model.generate。从结构上看图像端由 vision.py 中的视觉编码器提取特征再经 perceiver.py 的IdeficsPerceiverResamplerPerceiver Resampler对定长 latent 做交叉注意力压缩为固定数量视觉 token最后注入文本解码器层间cross_layer_interval控制交叉注意力层的间隔见 configuration_idefics.py。图像预处理方面image_processing_idefics.py 定义了标准化的均值/方差IDEFICS_STANDARD_MEAN/STD与 224×224 的默认输入尺寸。图像描述Image Captioning图像描述是根据给定图片预测文字说明的任务常见应用包括辅助视障人士浏览在线图片内容。IDEFICS 接受文本与图像混合的 Prompt但做纯图像描述时无需提供文本 Prompt——仅传入预处理后的图像即可。没有文本 Prompt 时模型从 BOSBeginning-of-sequencetoken 开始生成从而输出图片的 caption。 prompt [ ... https://images.unsplash.com/photo-1583160247711-2191776b4b91?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3542q80, ... ] inputs processor(prompt, return_tensorspt).to(cuda) bad_words_ids processor.tokenizer([image, fake_token_around_image], add_special_tokensFalse).input_ids generated_ids model.generate(**inputs, max_new_tokens10, bad_words_idsbad_words_ids) generated_text processor.batch_decode(generated_ids, skip_special_tokensTrue) print(generated_text[0]) A puppy in a flower bed为什么需要bad_words_ids增大max_new_tokens时模型可能错误地继续生成新的image或fake_token_around_image占位 token模型并不能真正生成图片。在generate调用中传入bad_words_ids可以显式禁止这些 token 被解码输出。你可以像本示例这样在调用时临时设置也可以参考文本生成策略指南将其固化到GenerationConfig中。提示式图像描述Prompted Image Captioning通过提供文本提示可以扩展图像描述模型会结合图像继续补全文本。把文本与图像 Prompt 作为单个列表传给处理器即可 prompt [ ... https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3501q80, ... This is an image of , ... ] inputs processor(prompt, return_tensorspt).to(cuda) bad_words_ids processor.tokenizer([image, fake_token_around_image], add_special_tokensFalse).input_ids generated_ids model.generate(**inputs, max_new_tokens10, bad_words_idsbad_words_ids) generated_text processor.batch_decode(generated_ids, skip_special_tokensTrue) print(generated_text[0]) This is an image of the Eiffel Tower in Paris, France.这里的技巧是把未完成的句子如This is an image of作为文本提示接在图片之后模型就会沿着该句子的语义继续生成将描述内容精确引导到期望的句式。Few-shot 提示上下文学习IDEFICS 在零样本下已有不错表现但某些任务要求特定输出格式或带有额外约束。此时可以使用Few-shot prompting在上下文in-context中学习在 Prompt 中给出若干图片-描述示例模型会模仿示例的格式输出结果。下面用一张埃菲尔铁塔的图片作为示例先让模型认识图片中的物体 有趣冷知识这一输出模式再对自由女神像图片施加同样的格式要求 prompt [User:, ... https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3501q80, ... Describe this image.\nAssistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building.\n, ... User:, ... https://images.unsplash.com/photo-1524099163253-32b7f0256868?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3387q80, ... Describe this image.\nAssistant: ... ] inputs processor(prompt, return_tensorspt).to(cuda) bad_words_ids processor.tokenizer([image, fake_token_around_image], add_special_tokensFalse).input_ids generated_ids model.generate(**inputs, max_new_tokens30, bad_words_idsbad_words_ids) generated_text processor.batch_decode(generated_ids, skip_special_tokensTrue) print(generated_text[0]) User: Describe this image. Assistant: An image of the Eiffel Tower at night. Fun fact: the Eiffel Tower is the same height as an 81-storey building. User: Describe this image. Assistant: An image of the Statue of Liberty. Fun fact: the Statue of Liberty is 151 feet tall.模型仅从一个示例即 1-shot就学会了任务执行方式。对于更复杂的任务可以自由尝试更多示例3-shot、5-shot 等把User:... / Assistant:...式的回合结构扩展到多个示例。视觉问答Visual Question Answering, VQAVQA 是根据图像回答开放式问题的任务。除无障碍应用外还可用于教育对视觉材料进行推理、服务根据图片回答商品相关问题、图像检索等场景。通过在 Prompt 中给出恰当的指令可以把模型从图像描述引导到视觉问答 prompt [ ... Instruction: Provide an answer to the question. Use the image to answer.\n, ... https://images.unsplash.com/photo-1623944889288-cd147dbb517c?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3540q80, ... Question: Where are these people and whats the weather like? Answer: ... ] inputs processor(prompt, return_tensorspt).to(cuda) bad_words_ids processor.tokenizer([image, fake_token_around_image], add_special_tokensFalse).input_ids generated_ids model.generate(**inputs, max_new_tokens20, bad_words_idsbad_words_ids) generated_text processor.batch_decode(generated_ids, skip_special_tokensTrue) print(generated_text[0]) Instruction: Provide an answer to the question. Use the image to answer. Question: Where are these people and whats the weather like? Answer: Theyre in a park in New York City, and its a beautiful day.Prompt 中Instruction 图片 Question/Answer的结构化模板是发挥大模型指令跟随能力的关键。图像分类Image ClassificationIDEFICS 即使没有针对特定类别标注数据做过显式训练也能将图像划分到不同类别它利用给定类别列表 图文理解能力推断图像最可能归属的类别。例如给定一张蔬菜摊图片可指示模型从候选类别中单选 categories [animals,vegetables, city landscape, cars, office] prompt [fInstruction: Classify the following image into a single category from the following list: {categories}.\n, ... https://images.unsplash.com/photo-1471193945509-9ad0617afabf?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3540q80, ... Category: ... ] inputs processor(prompt, return_tensorspt).to(cuda) bad_words_ids processor.tokenizer([image, fake_token_around_image], add_special_tokensFalse).input_ids generated_ids model.generate(**inputs, max_new_tokens6, bad_words_idsbad_words_ids) generated_text processor.batch_decode(generated_ids, skip_special_tokensTrue) print(generated_text[0]) Instruction: Classify the following image into a single category from the following list: [animals, vegetables, city landscape, cars, office]. Category: Vegetables除单类别分类外还可以通过改写指令让模型做多标签rank分类。这种方式把分类任务转化为受约束的文本生成任务无需训练分类头。图像引导的文本生成Image-Guided Text Generation面向更创造性的应用可以用图像引导文本生成基于图像创作产品描述、广告文案、场景说明等。下面让 IDEFICS 根据一张带南瓜的红门图片写一个故事 prompt [Instruction: Use the image to write a story. \n, ... https://images.unsplash.com/photo-1517086822157-2b0358e7684a?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw2203q80, ... Story: \n] inputs processor(prompt, return_tensorspt).to(cuda) bad_words_ids processor.tokenizer([image, fake_token_around_image], add_special_tokensFalse).input_ids generated_ids model.generate(**inputs, num_beams2, max_new_tokens200, bad_words_idsbad_words_ids) generated_text processor.batch_decode(generated_ids, skip_special_tokensTrue) print(generated_text[0]) Instruction: Use the image to write a story. Story: Once upon a time, there was a little girl who lived in a house with a red door. She loved her red door. It was the prettiest door in the whole world. One day, the little girl was playing in her yard when she noticed a man standing on her doorstep. He was wearing a long black coat and a top hat. The little girl ran inside and told her mother about the man. Her mother said, Dont worry, honey. Hes just a friendly ghost. The little girl wasnt sure if she believed her mother, but she went outside anyway. When she got to the door, the man was gone. The next day, the little girl was playing in her yard again when she noticed the man standing on her doorstep. He was wearing a long black coat and a top hat. The little girl ran模型注意到了门前的南瓜顺势编出了一个万圣节风格的鬼怪故事。此处使用了num_beams2束搜索且max_new_tokens200允许长文本输出。对于这类长输出微调文本生成策略beam search 束宽、采样温度、top-p 等能显著提升生成质量详见文本生成策略指南。批量模式推理此前所有示例均为单样本推理。IDEFICS 支持以几乎相同的方式处理一批样本直接把Prompt 列表的列表传给处理器即可。注意外层列表是 batch 维度内层列表是单个样本中交错的文本/图片条目 prompts [ ... [ https://images.unsplash.com/photo-1543349689-9a4d426bee8e?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3501q80, ... This is an image of , ... ], ... [ https://images.unsplash.com/photo-1623944889288-cd147dbb517c?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3540q80, ... This is an image of , ... ], ... [ https://images.unsplash.com/photo-1471193945509-9ad0617afabf?ixlibrb-4.0.3ixidM3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3Dautoformatfitcropw3540q80, ... This is an image of , ... ], ... ] inputs processor(prompts, return_tensorspt).to(cuda) bad_words_ids processor.tokenizer([image, fake_token_around_image], add_special_tokensFalse).input_ids generated_ids model.generate(**inputs, max_new_tokens10, bad_words_idsbad_words_ids) generated_text processor.batch_decode(generated_ids, skip_special_tokensTrue) for i,t in enumerate(generated_text): ... print(f{i}:\n{t}\n) 0: This is an image of the Eiffel Tower in Paris, France. 1: This is an image of a couple on a picnic blanket. 2: This is an image of a vegetable stand.从处理器的默认配置processing_idefics.py看padding默认为longest即同一 batch 内的样本会自动按最长序列补齐因此批量输入无需手动指定 padding 策略。会话场景IDEFICS Instruct面向对话场景可在 Hub 上使用微调过的 instruct 版本检查点HuggingFaceM4/idefics-80b-instruct与HuggingFaceM4/idefics-9b-instruct。这些检查点是对应基础模型在监督数据集与指令数据集上联合微调的结果在保持下游性能的同时让模型更适配多轮对话。对话的使用方式与基础模型非常相似关键在于end_of_utterance回合结束标记的使用——处理器支持通过add_end_of_utterance_token控制是否自动追加该标记 import torch from transformers import IdeficsForVisionText2Text, AutoProcessor device cuda if torch.cuda.is_available() else cpu checkpoint HuggingFaceM4/idefics-9b-instruct model IdeficsForVisionText2Text.from_pretrained(checkpoint, dtypetorch.bfloat16).to(device) processor AutoProcessor.from_pretrained(checkpoint) prompts [ ... [ ... User: What is in this image?, ... https://upload.wikimedia.org/wikipedia/commons/8/86/Id%C3%A9fix.JPG, ... end_of_utterance, ... \nAssistant: This picture depicts Idefix, the dog of Obelix in Asterix and Obelix. Idefix is running on the ground.end_of_utterance, ... \nUser:, ... https://static.wikia.nocookie.net/asterix/images/2/25/R22b.gif/revision/latest?cb20110815073052, ... And who is that?end_of_utterance, ... \nAssistant:, ... ], ... ] # --batched mode inputs processor(prompts, add_end_of_utterance_tokenFalse, return_tensorspt).to(device) # --single sample mode # inputs processor(prompts[0], return_tensorspt).to(device) # Generation args exit_condition processor.tokenizer(end_of_utterance, add_special_tokensFalse).input_ids bad_words_ids processor.tokenizer([image, fake_token_around_image], add_special_tokensFalse).input_ids generated_ids model.generate(**inputs, eos_token_idexit_condition, bad_words_idsbad_words_ids, max_length100) generated_text processor.batch_decode(generated_ids, skip_special_tokensTrue) for i, t in enumerate(generated_text): ... print(f{i}:\n{t}\n)对话场景的几个关键点构造多轮上下文把历史回合以User: ... end_of_utterance/\nAssistant: ... end_of_utterance的格式交错写入 Prompt 列表并在末尾以\nAssistant:收尾模型会接着生成助手回复eos_token_id设为end_of_utterance将回合结束标记作为停止生成条件模型答完一轮即停止add_end_of_utterance_tokenFalse由于 Prompt 中已手动写入end_of_utterance关闭处理器的自动追加以免重复批量与单样本两种调用方式批量模式传prompts外层 batch、内层回合列表单样本模式直接传prompts[0]。模型结构与配置速览下表汇总了 IDEFICS 配置中的核心字段及其默认值见 configuration_idefics.py帮助理解模型行为配置项默认值说明vocab_size32000文本词表大小additional_vocab_size0额外词表如image占位 token额外 token 始终可训练hidden_size4096隐藏层维度num_hidden_layers32文本解码器层数cross_layer_interval1文本到图像的交叉注意力层间隔use_resamplerFalse是否使用 Perceiver Resampler 压缩视觉 tokenfreeze_text_layersTrue是否冻结文本层微调相关freeze_vision_layersTrue是否冻结视觉层微调相关max_position_embeddings2048最大位置编码长度IDEFICS 的图像输入默认缩放到 224×224使用 CLIP 风格的标准化参数IDEFICS_STANDARD_MEAN/STD见 image_processing_idefics.py。仓库中还提供了完整的自动化测试用于验证上述行为模型前向与生成逻辑见 test_modeling_idefics.py处理器文本/图像混排与 token 注入见 test_processing_idefics.py图像预处理见 test_image_processing_idefics.py可作为深入学习与回归验证的参考。小结IDEFICS 展示了大模型直用范式在多模态任务上的威力无需针对每个任务微调专用模型仅通过精心构造的 Prompt图像与文本交错、Few-shot 示例、指令模板、回合标记即可覆盖图像描述、VQA、图像分类、创意生成与多轮对话等广泛场景。实际部署时请根据显存预算在全精度/bfloat16与4-bit 量化加载方式之间取舍并在长文本生成时通过bad_words_ids、eos_token_id与文本生成策略束搜索、采样参数控制输出质量与终止条件。【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
