WebWISE: Web Interface Control and Sequential Exploration with Large Language Models

The paper investigates using a Large Language Model (LLM) to automatically perform web software tasks using click, scroll, and text input operations. Previous approaches, such as reinforcement learning (RL) or imitation learning, are inefficient to train and task-specific. Our method uses filtered D...

Full description

Saved in:
Bibliographic Details
Main Authors Tao, Heyi, TV, Sethuraman, Shlapentokh-Rothman, Michal, Hoiem, Derek
Format Journal Article
LanguageEnglish
Published 24.10.2023
Subjects
Computer Science - Artificial Intelligence
Computer Science - Computation and Language
Online AccessGet full text

Cover

More Information
Summary:The paper investigates using a Large Language Model (LLM) to automatically perform web software tasks using click, scroll, and text input operations. Previous approaches, such as reinforcement learning (RL) or imitation learning, are inefficient to train and task-specific. Our method uses filtered Document Object Model (DOM) elements as observations and performs tasks step-by-step, sequentially generating small programs based on the current observations. We use in-context learning, either benefiting from a single manually provided example, or an automatically generated example based on a successful zero-shot trial. We evaluate the proposed method on the MiniWob++ benchmark. With only one in-context example, our WebWISE method achieves similar or better performance than other methods that require many demonstrations or trials.
DOI:10.48550/arxiv.2310.16042