Amazon Unveils Nova Act SDK for Building Web Browser Agents

Amazon AGI Labs has introduced a new AI model, Nova Act, designed to perform actions within web browsers. This release includes a research preview of the Nova Act SDK, available at nova.amazon.com, allowing developers to experiment with an early version of the model. The SDK enables developers to build agents capable of completing tasks in web browsers, such as submitting out-of-office requests or setting up calendar holds.

What Changed Technically and Why It Matters

Traditionally, "agents" have been systems that respond to users in natural language or draw on knowledge bases using Retrieval-Augmented Generation (RAG). However, Nova Act shifts the focus to agents that can execute tasks in digital environments, particularly within web browsers. This change is significant because it addresses a critical gap in current agent capabilities: the ability to handle multi-step, complex workflows without constant human supervision.

Key Features of Nova Act SDK

• Atomic Commands: The SDK provides reliable atomic commands for common actions like searching, checking out, and answering questions about the screen.

  • These commands can be combined to form more complex workflows.
  • Developers can add detailed instructions to these commands (e.g., "don’t accept the insurance upsell").

• API Integration: Nova Act supports calling APIs, allowing agents to interact with external systems seamlessly.

  • This integration is crucial for tasks that require data from multiple sources or actions across different platforms.

• Direct Browser Manipulation: The SDK leverages Playwright for direct browser manipulation, enhancing reliability and accuracy.

  • For example, it can handle sensitive tasks like entering passwords more securely.

• Python Code Interleaving: Developers can insert Python code to enhance the agent's functionality.

  • This includes tests, breakpoints, asserts, and thread pools for parallelization.
  • Even the fastest agents are limited by web page load times, so this feature helps optimize performance.

Use Cases and Benchmarks

Nova Act is particularly useful for tasks that require a series of steps, such as organizing events or handling complex IT tasks. While some use cases are well-suited for today’s technology, multi-step agents prompted with high-level goals still often require human intervention.

• Current Agent Performance: Many agent benchmarks measure model performance on high-level tasks, where state-of-the-art models achieve 30% to 60% accuracy in completing tasks within web browsers.
  • Nova Act aims to improve this by providing reliable building blocks that can be composed into more complex workflows.

Why This Matters for Practitioners

For developers and businesses, the ability to automate complex, multi-step tasks in web environments can significantly boost productivity. By reducing the need for human oversight, Nova Act can help streamline processes and free up valuable time. The SDK's focus on reliability and composability means that developers can build robust agents that handle a wide range of tasks with minimal errors.

Getting Started

To start experimenting with Nova Act, developers can visit nova.amazon.com to access the SDK. The research preview provides documentation, examples, and community support to help you get up and running quickly.