You may call one or more functions to assist with the user query. You are provided with…
- User
You may call one or more functions to assist with the user query.
You are provided with function signatures within XML tags: {"type": "function", "function": {"name_for_human": "mobile_use", "name": "mobile_use", "description": "Use a touchscreen to interact with a mobile device, and take screenshots.
- This is an interface to a mobile device with touchscreen. You can perform actions like clicking, typing, swiping, etc.
- Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions.
- The screen's resolution is 672x1428.", "parameters": {"properties": {"action": {"description": "The action to perform. The available actions are:
key: Perform a key event on the mobile device.- This supports adb's
keyeventsyntax. - Examples: "volume_up", "volume_down", "power", "camera", "clear".
- This supports adb's
click: Click the point on the screen with coordinate (x, y).long_press: Press the point on the screen with coordinate (x, y) for specified seconds.swipe: Swipe from the starting point with coordinate (x, y) to the end point with coordinates2 (x2, y2).type: Input the specified text into the activated input box.system_button: Press the system button.open: Open an app on the device.wait: Wait specified seconds for the change to happen.terminate: Terminate the current task and report its completion status.", "enum": ["key", "click", "long_press", "swipe", "type", "system_button", "open", "wait", "terminate"], "type": "string"}, "coordinate": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only byaction=click,action=long_press, andaction=swipe.", "type": "array"}, "coordinate2": {"description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only byaction=swipe.", "type": "array"}, "text": {"description": "Required only byaction=key,action=type, andaction=open.", "type": "string"}, "time": {"description": "The seconds to wait. Required only byaction=long_pressandaction=wait.", "type": "number"}, "button": {"description": "Back means returning to the previous interface, Home means returning to the desktop, Menu means opening the application background menu, and Enter means pressing the enter. Required only byaction=system_button", "enum": ["Back", "Home", "Menu", "Enter"], "type": "string"}, "status": {"description": "The status of the task. Required only byaction=terminate.", "type": "string", "enum": ["success", "failure"]}}, "required": ["action"], "type": "object"}, "args_format": "Format the arguments as a JSON object."}}
For each function call, return a json object with function name and arguments: {"name": , "arguments": }
以上是我定义的用于GUI操控的action space,在实际使用过程中,有时候会遇到广告、弹窗等遮挡,需要先通过click处理到遮挡,才可以继续完成原有的用户指令。在这种情况下,除了click的坐标点,还需要某种方式能够通知框架侧当前的click操作实际是在处理广告等遮挡,而不是真正的用户指令。请帮我修改上述action space定义,从而能够达到预期效果。
Assistant
为了区分正常用户指令与处理广告/弹窗的点击操作,可以在
mobile_use函数的参数中新增一个ad_context字段,用于标记点击的意图。修改后的函数定义如下:
修改后的
mobile_use函数定义{ "type": "function", "function": { "name_for_human": "mobile_use", "name": "mobile_use", "description": "Use a touchscreen to interact with a mobile device, and take screenshots.\n* This is an interface to a mobile device with touchscreen. You can perform actions like clicking, typing, swiping, etc.\n* Some applications may take time to start or process actions, so you may need to wait and take successive screenshots to see the results of your actions.\n* The screen's resolution is 672x1428.", "parameters": { "properties": { "action": { "description": "The action to perform. The available actions are: key, click, long_press, swipe, type, system_button, open, wait, terminate.", "enum": ["key", "click", "long_press", "swipe", "type", "system_button", "open", "wait", "terminate"], "type": "string" }, "coordinate": { "description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=click`, `action=long_press`, and `action=swipe`.", "type": "array" }, "coordinate2": { "description": "(x, y): The x (pixels from the left edge) and y (pixels from the top edge) coordinates to move the mouse to. Required only by `action=swipe`.", "type": "array" }, "text": { "description": "Required only by `action=key`, `action=type`, and `action=open`.", "type": "string" }, "time": { "description": "The seconds to wait. Required only by `action=long_press` and `action=wait`.", "type": "number" }, "button": { "description": "Back means returning to the previous interface, Home means returning to the desktop, Menu means opening the application background menu, and Enter means pressing the enter. Required only by `action=system_button`", "enum": ["Back", "Home", "Menu", "Enter"], "type": "string" }, "status": { "description": "The status of the task. Required only by `action=terminate`.", "type": "string", "enum": ["success", "failure"] }, "ad_context": { "description": "Indicates if the action is for handling an ad/popup. 'ad' for ad dismissal, 'normal' for main task.", "type": "string", "enum": ["normal", "ad"] } }, "required": ["action"], "type": "object" }, "args_format": "Format the arguments as a JSON object." } }
使用示例
-
处理广告的点击
{ "name": "mobile_use", "arguments": { "action": "click", "coordinate": [300, 700], "ad_context": "ad" } } -
正常用户指令的点击
{ "name": "mobile_use", "arguments": { "action": "click", "coordinate": [200, 500], "ad_context": "normal" } }
框架侧逻辑说明
- 当
ad_context设置为"ad"时,框架可触发以下行为:- 优先处理广告遮挡(如自动重试或记录日志)。
- 在截图或后续操作中额外标记该点击为“广告处理”。
- 若广告未被正确处理,可触发错误上报或重新尝试。
- 当
ad_context为"normal"或未提供时,按原始逻辑执行。
此修改通过新增参数明确区分操作意图,同时保持原有功能兼容性。
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.