Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-2024-05-13· OpenAIAi2 WildChat-4.8M (AI2)

你是一个资深的翻译师,从事人工智能特别是多模态大模型领域翻译工作多年,请将下列的英文翻译成中文

  1. User

    你是一个资深的翻译师,从事人工智能特别是多模态大模型领域翻译工作多年,请将下列的英文翻译成中文

  2. Assistant

    好的,请提供您需要翻译的英文内容,我将尽力为您翻译。

  3. User

    In NLP, studies show that the quality of instruction-following data largely affects the capability of the resulting instruction-following models [61]. For visual instruction tuning, LLaVA [36] is the pioneer to leverage text-only GPT-4 to expand the existing COCO [35] bounding box and caption dataset to a multi- modal instruction-following dataset that contains three types of instruction-following data: conversational-style QA, detailed description, and complex reasoning. LLaVA’s pipeline has been employed to expand to textual understanding [57], million-scales [58], and region-level conversations [8]. InstructBLIP [14] incorporates academic-task-oriented VQA datasets to further enhance the model’s visual capabilities. Conversely, [7] identifies that such naive data merging can result in models that tend to overfit to VQA datasets and thus are unable to participate in natural conversations. The authors further propose to leverage the LLaVA pipeline to convert VQA datasets to a conversational style. While this proves effective for training, it introduces added complexities in data scaling. However, in NLP, the FLAN family [13, 51] shows that adding a large number of academic language tasks for instruction tuning can effectively improve the generalization ability. In light of this, we consider investigating the root cause of the inability to balance between natural conversations and academic tasks in multimodal models

  4. Assistant

    在自然语言处理(NLP)领域,研究表明,指令跟随数据的质量在很大程度上影响了指令跟随模型的能力 [61]。在视觉指令调优方面,LLaVA [36] 首创利用纯文本的 GPT-4 扩展现有的 COCO [35] 边界框和标题数据集,形成一个包含三种类型指令跟随数据的多模态指令跟随数据集:对话风格的问答、详细描述和复杂推理。LLaVA 的流程已经被应用于扩展到文本理解 [57]、百万规模 [58] 和区域级对话 [8]。InstructBLIP [14] 则结合了面向学术任务的视觉问答(VQA)数据集,以进一步增强模型的视觉能力。相反,文章 [7] 指出,这种简单的数据合并会导致模型过拟合于 VQA 数据集,从而无法参与自然对话。作者进一步提出利用 LLaVA 的流程,将 VQA 数据集转换为对话风格。尽管这在训练中证明是有效的,但在数据规模上引入了更多复杂性。然而,在 NLP 中,FLAN 系列研究 [13, 51] 显示,增加大量学术语言任务进行指令调优可以有效地提高泛化能力。有鉴于此,我们考虑探讨在多模态模型中无法平衡自然对话与学术任务的根本原因。

  5. User

    We find that the inability [7] to balance between short and long-form VQA for approaches like InstructBLIP [14], which leverages instruction following data that includes both natural responses and short-answers, is mainly due to the fol- lowing reasons. First, ambiguous prompts on the response format. For example, Q: {Question} A: {Answer}. Such prompts do not clearly indicate the desired output format, and can overfit an LLM behaviorally to short-form answers even for natural visual conversations. Second, not finetuning the LLM. The first issue is worsened by InstructBLIP only finetuning the Qformer for instruction-tuning. It requires the Qformer’s visual output tokens to control the length of the LLM’s output to be either long-form or short-form, as in prefix tuning [33], but Qformer may lack the capability of properly doing so, due to its limited capacity compared with LLMs like LLaMA.

  6. Assistant

    我们发现像 InstructBLIP [14] 这样的方法,在平衡短回答和长回答的视觉问答(VQA)方面存在困难,这种方法利用了包含自然响应和短回答的指令跟随数据,主要原因如下。首先,响应格式的提示不明确。比如,Q: {问题} A: {回答}。这样的提示并没有明确指示期望的输出格式,导致大模型行为上容易倾向于短回答,即使是在自然的视觉对话中也是如此。其次,没有对大语言模型(LLM)进行微调。InstructBLIP 只对 Qformer 进行了指令调优,这进一步加剧了第一个问题。它要求 Qformer 的视觉输出令牌来控制 LLM 的输出长度是长格式还是短格式,如同前缀调优 [33],但由于与像 LLaMA 这样的大语言模型相比,Qformer 的能力有限,可能无法适当地完成这项任务。

  7. User

    Answer with the option’s letter from the given choices directly

  8. Assistant

    Sure, please provide the choices and the question.

  9. User

    翻译成中文 Answer with the option’s letter from the given choices directly

  10. Assistant

    请直接用给定选项中的字母作答。

  11. User

    In Sec. 3.3, we observe the advantage that scaling up the input image resolution improves the model’s capabilities. However, the image resolution of the existing open source CLIP vision encoders is limited to 3362, preventing the support of higher resolution images by simply replacing the vision encoder as we did in Sec. 3.3. In this section, we present an early exploration of scaling the LMM to higher resolutions, while maintaining the data efficiency of LLaVA-1.5. When using ViT [15] as the vision encoder, to scale up the resolution, previous approaches mostly choose to per- form positional embedding interpolation [3, 32] and adapt the ViT backbone to the new resolution during finetuning. However, this usually requires the model to be finetuned on a large-scale image-text paired dataset [3, 32], and limits the resolution of the image to a fixed size that the LMM can accept during inference

  12. Assistant

    在第3.3节中,我们观察到提高输入图像分辨率可以提升模型的能力。然而,现有开源的CLIP视觉编码器的图像分辨率限制在3362,这使得无法像第3.3节中那样简单地通过替换视觉编码器来支持更高分辨率的图像。在本节中,我们介绍了将LMM扩展到更高分辨率的初步探索,同时保持LLaVA-1.5的数据效率。当使用ViT [15]作为视觉编码器时,为了提高分辨率,之前的方法大多选择进行位置嵌入插值 [3, 32],并在微调过程中使ViT主干适应新的分辨率。然而,这通常需要在大规模图像-文本配对数据集上对模型进行微调 [3, 32],并限制了在推理过程中LMM可以接受的图像分辨率为固定大小。

  13. User

    Instead, as shown in Fig. 2, we overcome this by dividing the image into smaller image patches of the resolution that the vision encoder is originally trained for, and encode them independently. After obtaining the feature maps of individual patches, we then combine them into a single large feature map of the target resolution, and feed that into the LLM. To provide the LLM with the global context and to reduce the artifact of the split-encode-merge operation, we addition- ally concatenate the feature of a downsampled image to the merged feature map. This allows us to scale the input to any arbitrary resolution and maintain the data efficiency of LLaVA-1.5. We call this resulting model LLaVA-1.5-HD

  14. Assistant

    相反,如图2所示,我们通过将图像划分为更小的图像块,这些图像块的分辨率是视觉编码器原本训练的分辨率,并独立地对它们进行编码,从而克服了这一问题。在获取各个块的特征图后,我们将它们组合成一个目标分辨率的单一大特征图,并将其输入到大语言模型(LLM)中。为了向LLM提供全局上下文并减少分割-编码-合并操作的伪影,我们还将降采样图像的特征与合并后的特征图连接起来。这样就使我们能够将输入扩展到任意分辨率,并保持LLaVA-1.5的数据效率。我们将这种结果模型称为LLaVA-1.5-HD。

  15. User

    For recent benchmarks proposed for instruction-following LMMs, POPE [34] evaluates model’s degree of hallucination on three sampled subsets of COCO [35]: random, common, and adversarial and we report the F1 score on all three splits. Other benchmarks evaluate the model’s capabilities on a wide range of domains and applications, with different response formats. MME-Perception [17] evaluates model’s visual perception with yes/no questions. MMBench [37] evaluates model’s answer robustness with all-round shuffling on multiple choice answers. MMBench-CN [37] is the Chinese-translated version of MMBench. SEED-Bench [27] evaluates model’s performance on both images and videos with multiple choice, and we sample the frame in the middle to evaluate the accuracy on videos. LLaVA-Bench-in-the-Wild [36] and MM-Vet [55] evaluate model’s capabilities in engaging in visual conversations on a diverse range of tasks, and evaluates the correctness and the helpfulness of the response with GPT-4 evaluation.

  16. Assistant

    对于最近为指令跟随的大模型(LMMs)提出的基准测试,POPE [34] 评估了模型在COCO [35]的三个采样子集(随机、常见和对抗性)上的幻觉程度,并报告了这三个子集上的F1得分。其他基准测试则在不同领域和应用中评估模型的能力,使用不同的响应格式。MME-Perception [17] 使用是/否问题评估模型的视觉感知能力。MMBench [37] 通过对多选答案进行全面打乱,评估模型的回答鲁棒性。MMBench-CN [37] 是MMBench的中文版。SEED-Bench [27] 通过多项选择题评估模型在图像和视频上的表现,我们取中间的帧来评估视频的准确性。LLaVA-Bench-in-the-Wild [36] 和MM-Vet [55] 评估模型在多样化任务中参与视觉对话的能力,并通过GPT-4评估响应的正确性和有用性。

  17. User

    LLaVA-1.5 can detect and answer tricky questions when prompted to verify the question.

  18. Assistant

    LLaVA-1.5 可以在被提示验证问题时检测并回答棘手问题。

  19. User

    In NLP, findings [49] suggest that the capability of the base LLM can affect its instruction-tuned successors. In this section, we explore two families of LLMs and study their contribution to the final model’s multimodal capability: LLaMA-based (Vicuna v1.1, Vicuna-v1.3) and LLaMA-2-based (Vicuna-v1.5, LLaMA-2-Chat). Vicuna-v1.3 and Vicuna-v1.5 use the same ∼150K ShareGPT [46] data (2×that used in v1.1). Unlike Vicuna series that is only trained with supervised instruction finetuning (SFT), LLaMA 2-Chat is further optimized with reinforcement-learning from human-feedback (RLHF). We visualize the relative performance of these variants in Fig. 3.

  20. Assistant

    在自然语言处理(NLP)领域,研究发现 [49] 表明基础大语言模型(LLM)的能力可以影响其经过指令调优的后续模型。在本节中,我们探讨了两大家族的LLM,并研究它们对最终模型多模态能力的贡献:基于LLaMA的(Vicuna v1.1, Vicuna-v1.3)和基于LLaMA-2的(Vicuna-v1.5, LLaMA-2-Chat)。Vicuna-v1.3和Vicuna-v1.5使用了相同的约15万份ShareGPT [46]数据(是v1.1使用数据的两倍)。与仅通过监督指令微调(SFT)训练的Vicuna系列不同,LLaMA 2-Chat进一步通过从人类反馈中进行强化学习(RLHF)进行优化。我们在图3中可视化了这些变体的相对性能。

  21. User

    First, we find that Vicuna-v1.5 achieves the best overall performance, and LLaMA-2-based models generally perform better than LLaMA-1-based, suggesting the importance of the base language model. This is further evidenced by the results on MMBench-CN [37]: despite Vicuna-v1.3 and v1.5 using the same ShareGPT data for instruction tuning, the performance in generalization to Chinese of Vicuna-v1.3 is significantly worse than v1.5. Second, language instruction-tuning matters on specific capabilities that are required by each dataset. For example, although LLaMA-2-Chat and Vicuna-v1.5 achieves almost the same performance on MMBench, the generalization to MMBench-CN [37] of LLaMA-2-Chat is worse than Vicuna- v1.5, which is partly due to that the most SFT/RLHF data of LLaMA-2-Chat is in English and does not contain as many multilingual data as in ShareGPT. Furthermore, TextVQA requires both the model’s capability of identifying the text characters in the images, and also processing the noisy outputs from the OCR engine; such noise may be more commonly observed in the ShareGPT data, which is collected in-the-wild from daily usage of ChatGPT

  22. Assistant

    首先,我们发现 Vicuna-v1.5 达到了最佳的整体性能,且基于 LLaMA-2 的模型总体上比基于 LLaMA-1 的模型表现更好,这表明基础语言模型的重要性。这一点在 MMBench-CN [37] 的结果中得到了进一步证明:尽管 Vicuna-v1.3 和 v1.5 使用相同的 ShareGPT 数据进行指令调优,但 Vicuna-v1.3 在汉语泛化表现上明显不如 v1.5。其次,语言指令调优对于每个数据集需要的特定能力非常重要。例如,虽然 LLaMA-2-Chat 和 Vicuna-v1.5 在 MMBench 上的表现几乎相同,但 LLaMA-2-Chat 在 MMBench-CN [37] 的泛化效果比 Vicuna-v1.5 要差,部分原因是 LLaMA-2-Chat 的大部分 SFT/RLHF 数据是英语的,没有像 ShareGPT 那样包含大量的多语言数据。此外,TextVQA 不仅要求模型具备识别图像中文字字符的能力,还需要处理来自 OCR 引擎的噪声输出;这种噪声可能在 ShareGPT 数据中更常见,因为这些数据是在实际使用 ChatGPT 时收集的。

  23. User

    Despite the data efficiency of LLaVA-1.5 when compared with approaches like InstructBLIP [14], the training of LLaVA-1.5 still doubles when compared with LLaVA. In this section, we conduct experiments for further improving the data efficiency by randomly sub-sampling the training data mixture of LLaVA-1.5, with a sampling ratio ranging from 0.1 to 0.5. We visualize the relative performance of different sampling variants in Fig. 4. First, the full data mixture provides the best knowledge coverage, and allows the model to achieve the best overall performance. To our surprise, with only 50% of the samples, the model still maintains more than 98% of the full dataset performance. This suggests that there is room for further improvements in data efficiency. Second, when downsampling the dataset to 50%, the model’s performance on MMBench, ScienceQA, and POPE does not decrease at all, and it even slightly improves on MMBench. Similarly, the model’s performance remains steady when further downscaling the data from 50% to 30%. These results show promise of having the less-is-more [61] benefit for multimodal models as well

  24. Assistant

    尽管 LLaVA-1.5 在数据效率方面相比 InstructBLIP(如InstructBLIP [14])的方法更有优势,但与 LLaVA 相比,LLaVA-1.5 的训练时间仍然翻倍。在本节中,我们通过随机抽样 LLaVA-1.5 的训练数据集,抽样比例从 0.1 到 0.5 之间,来进一步提高数据效率。我们在图 4 中可视化了不同抽样变体的相对性能。首先,完整的数据混合提供了最佳的知识覆盖,使模型能够达到最佳的整体表现。令人惊讶的是,仅使用 50% 的样本,模型仍能保持超过 98% 的完整数据集性能。这表明在数据效率方面还有进一步提高的空间。其次,将数据集抽样降至 50% 时,模型在 MMBench、ScienceQA 和 POPE 上的性能没有任何下降,甚至在 MMBench 上略有提升。同样地,当进一步将数据从 50% 降至 30% 时,模型的性能保持稳定。这些结果表明,多模态模型也有可能享受到“少即是多”的好处 [61]。

  25. User

    Hallucination is an important issue to tackle for LLMs and LMMs. Often in LMMs, we attribute the model’s hallucination to the errors or hallucinations in the training dataset. For example, the detailed descriptions in LLaVA-Instruct [36] may contain a small amount of hallucinated content, and it is believed that training on such data may have caused the model to hallucinate when asked to “describe the image in detail”. However, we find that such hallucination is signifi- cantly reduced, when we scale the model’s inputs to higher resolutions like 4482. This finding is interesting as it suggests that the LMMs may be robust to a few such errors in the training data. However, when the input resolution is not sufficient for the model to discern all details in the training data, and the amount of data that is at that granularity beyond the model’s capability becomes large enough, the model learns to hallucinate. This further suggests that there needs to be a balance between improving the data annotation with more details and the model’s capability to properly process the information at such granularities. We hope this finding provides a reference for future work in terms of dealing with hallucination and the scaling of the models and data.

  26. Assistant

    幻觉是大语言模型(LLMs)和多模态大模型(LMMs)需要解决的重要问题。通常在 LMMs 中,我们将模型的幻觉归因为训练数据集中的错误或幻觉。例如,LLaVA-Instruct [36] 中的详细描述可能包含少量虚构内容,且认为在这种数据上训练可能导致模型在被要求“详细描述图像”时产生幻觉。然而,我们发现,当将模型输入扩展到更高分辨率(如 4482)时,这种幻觉显著减少。这一发现很有趣,因为它表明 LMMs 可能对训练数据中的少量错误具有一定的鲁棒性。然而,当输入分辨率不足以让模型辨别训练数据中的所有细节,且超出模型能力的细节数据量足够大时,模型会学会幻觉。这进一步表明,需要在改进数据标注以包含更多细节与模型处理这些细节能力之间找到平衡。我们希望这一发现能为未来在处理幻觉以及模型和数据扩展方面提供参考。

  27. User

    Compositional Capabilities

  28. Assistant

    组合能力

  29. User

    We demonstrate interesting compositional capabilities in LLaVA-1.5: the model trained on a set of tasks independently generalizes to tasks that require a combination of these capabilities without explicit joint training. We note some of the findings below First, we observe an improved language capability in visual conversations after including the ShareGPT [46] data, including the multimodal multilingual capability as discussed in Sec. 4.3. Moreover, the model is more capable at providing longer and more detailed responses in visual conversations. Second, the additional visual knowledge from the academic-task-oriented datasets, improves the visual groundness of LLaVA-1.5’s responses in visual conversations, as evidenced quantitatively by the improved results on MMVet [55] and LLaVA-Wild [36] in Table 4. However, there is still difficulty in achieving ideal performance for some tasks that require a certain combination of capabilities. For example, being able to correctly answer the attribute of a certain object in VQA, does not guarantee an accurate depiction of that object attribute in a detailed description of the whole image. Furthermore, the capability of engaing in conversations with certain foreign languages (e.g. Korean) still falls behind. See appendix for examples. These findings suggest that the compositional capabilities of LMMs can be leveraged to improve the model’s performance without significantly increasing the data by exhaustively including all task combinations. Yet, it can be further investigated, and a deeper understanding of the mechanism behind the compositional capabilities of LMMs can further improve the capability and the data efficiency of LLaVA-1.5.

  30. Assistant

    我们展示了 LLaVA-1.5 的有趣组合能力:模型在一组任务上独立训练后,可以在无需显式联合训练的情况下泛化到需要这些能力组合的任务。我们在以下部分记录了一些发现。

    首先,在包括 ShareGPT [46] 数据后,我们观察到视觉对话中的语言能力得到了提升,包括第四节3小节中讨论的多模态多语言能力。此外,模型在视觉对话中更能提供更长且更详细的回答。其次,来自学术任务导向数据集的额外视觉知识,提高了 LLaVA-1.5 在视觉对话中回答的视觉基础性,这在表4中的 MMVet [55] 和 LLaVA-Wild [36] 的改进结果中得到了定量证明。

    然而,对于一些需要特定能力组合的任务,仍然难以达到理想的性能。例如,能够在 VQA 中正确回答某个对象的属性,并不保证在对整个图像的详细描述中能够准确描绘该对象的属性。此外,使用某些外语(例如韩语)进行对话的能力仍然滞后。具体案例请参见附录。

    这些发现表明,可以利用 LMMs 的组合能力来提升模型性能,而无需通过全面涵盖所有任务组合来显著增加数据量。然而,这还需要进一步研究,深入理解 LMMs 组合能力背后的机制,可以进一步提升 LLaVA-1.5 的能力和数据效率。

  31. User

    In this paper, we take a step towards demystifying the design of large multimodal models, and propose a simple, effective, and data-efficient baseline, LLaVA-1.5, for large multimodal models. In addition, we explore the open problems in visual instruction tuning, scale LMMs to higher resolutions, and present some intriguing findings in terms of model hallucination and compositional capabilities for LMMs. We hope these improved and easily-reproducible baselines as well as the new findings will provide a reference for future research in open-source LMM.

  32. Assistant

    在本文中,我们迈出了一步,致力于揭示大规模多模态模型的设计,并提出了一个简单、有效且数据高效的基线模型 LLaVA-1.5,用于大规模多模态模型。此外,我们探索了视觉指令调优中的开放问题,扩展了LMMs到更高的分辨率,并展示了一些关于模型幻觉和组合能力方面的有趣发现。我们希望这些改进和易于复制的基线模型以及新发现能为未来开源LMM的研究提供参考。

  33. User

    Limitations. Despite the promising results demonstrated by LLaVA-1.5, it still has limitations including prolonged training for high-resolution images, lack of multiple-image understanding, limited problem solving capabilities in cer- tain fields. It is not exempt from producing hallucinations, and should be used with caution in critical applications (e.g. medical). See appendix for a detailed discussion. Acknowledgements. This work was supported in part by NSF CAREER IIS2150012, and Institute of Information & communications Technology Planning & Evaluation(IITP) grants funded by the Korea government(MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration) and (No. RS- 2022-00187238, Development of Large Korean Language Model Technology for Efficient Pre-training).

  34. Assistant

    局限性:尽管 LLaVA-1.5 展示了令人鼓舞的结果,它仍然存在一些局限性,包括高分辨率图像的训练时间过长、对多图像理解的缺乏、在某些领域问题解决能力的有限性。它不能避免产生幻觉,因此在关键应用(如医疗)中应谨慎使用。详细讨论请参见附录。

    致谢:本工作部分得到了 NSF CAREER IIS2150012 和韩国政府(MSIT)资助的信息与通信技术规划与评估研究所(IITP)资助项目的支持(项目编号 2022-0-00871,AI 代理合作的 AI 自主性和知识增强开发;项目编号 RS-2022-00187238,韩语大模型技术高效预训练开发)。

  35. User

    We predefine a set of resolutions to support up to six grids (1x1, 1x2, 1x3, 1x4, 1x5, 1x6, 2x2, 2x3, and their transpose). This system allows for a maximum resolution of 672x448 (or 448x672). Two criteria are enforced in the target resolution selection: (1) Detail preservation: the selected resolution preserves as much detail from the original image as possible; (2) Resource efficiency: the resolution should not be excessively large to avoid unneces- sary consumption of pixels and memory (e.g. it should not select 4482for a 2242input image)

  36. Assistant

    我们预定义了一组分辨率,以支持多达六个网格(1x1、1x2、1x3、1x4、1x5、1x6、2x2、2x3及其转置)。该系统允许最大分辨率为672x448(或448x672)。在目标分辨率选择上执行两个标准:(1) 细节保留:所选分辨率尽可能多地保留原始图像的细节;(2) 资源效率:分辨率不应过大,以避免不必要的像素和内存消耗(例如,不应为224x224的输入图像选择448x448的分辨率)。

  37. User

    We perform three steps of postprocessing to ensure that the final features can be processed effectively and efficiently by the language model. (1) Padding removal. Features corresponding exclusively to the paddings are dis- carded. This reduces the number of visual tokens processed by the language model and improves the efficiency. (2) Row end Tokens. We append a special token to the end of each row of features, to provide an explicit indication of the shape of the image. Unlike the original LLaVA and LLaVA-1.5 that uses a fixed resolution, we now use a variable resolution for the image features of LLaVA-1.5-HD, such indication allows the language model to capture the exact shape and the size of the image for each sample. (3) Flattening. Finally, we flatten the image feature map and feed it into the language model along with language token features.

  38. Assistant

    我们执行三步后处理,以确保最终特征能够被语言模型有效且高效地处理。(1) 去除填充。丢弃与填充对应的特征。这减少了语言模型处理的视觉标记数量,从而提高了效率。(2) 行结束标记。我们在每行特征的末尾附加一个特殊标记,以明确表示图像的形状。与使用固定分辨率的原始 LLaVA 和 LLaVA-1.5 不同,我们现在为 LLaVA-1.5-HD 的图像特征使用可变分辨率,这样的指示使语言模型能够捕捉每个样本的图像确切形状和大小。(3) 展平。最后,我们将图像特征映射展平,并将其与语言标记特征一起输入语言模型。

  39. User
    1. For all VQA datasets, QA pairs from the same training image are merged into a single conversation.
    2. For ShareGPT [46], we filter out invalid conversations as [12]. Unlike Vicuna [12], long conversations that surpass 2048 tokens are truncated rather than splitting to multiple conversations. This results in ∼40K conversations.
    3. Each QA pair in A-OKVQA [45] is augmented ktimes, where kis the number of choices per question, to counterbalance the lack of multiple-choice data.
    4. 80K conversations are sampled from OCRVQA [42].
  40. Assistant
    1. 对于所有视觉问答(VQA)数据集,来自同一训练图像的问答对被合并为一个对话。
    2. 对于 ShareGPT [46],我们按照 [12] 过滤掉无效对话。不像 Vicuna [12],对于超过2048个标记的长对话,我们将其截断而不是分割成多个对话。这导致约40K个对话。
    3. A-OKVQA [45] 中的每个问答对被增强k次,其中k是每个问题的选择数量,以平衡多选数据的缺乏。
    4. 从 OCRVQA [42] 中抽取了80K个对话。
  41. User
    1. For Visual Genome, we sample 10 annotations for images with additional annotations.
    2. For RefCOCO, conversations are dissected into segments, each containing fewer than 10 conversations.
    3. We obverse that language conversations are often longer than visual ones. For each batch, we sample conversations only from a single modality, and this speeds up the training by 25%, and does not affect the final outcome. All data splits are concatenated together and sampled with the same probability. We present the response formatting prompts of the final instruction-following data mixtures in Table 7 and the response format prompts used for each evaluation benchmark in Table 8.
  42. Assistant
    1. 对于 Visual Genome,我们为有额外注释的图像抽取10个注释。
    2. 对于 RefCOCO,对话被分解成多个片段,每个片段包含少于10个对话。
    3. 我们观察到语言对话通常比视觉对话更长。对于每个批处理,我们仅从单一模态中抽取对话,这使训练速度提高了25%,且不影响最终结果。所有数据分割都被连接在一起,并以相同的概率进行抽样。我们在表7中展示了最终指令跟随数据混合的响应格式提示,并在表8中展示了用于每个评估基准测试的响应格式提示。
  43. User

    A cartoon-style image of a young girl with a bob haircut, wearing a white kimono, holding a stuffed animal, and posing for a picture, with a color scheme of pink and white, detailed, and trending on artstation.

  44. Assistant

    一张卡通风格的图片,描绘了一个留着波波头发型的年轻女孩,穿着白色和服,手里抱着一个毛绒玩具,正在为拍照摆姿势。这幅图的颜色方案以粉色和白色为主,细致入微,并在 ArtStation 上趋势流行。