NC State University

Department of Computer Science Colloquia 1999-2000

Date: Thursday, March 30, 2000
Time: 3: 30 PM (refreshments), 4:00 PM (talk)
Place: Withers 402A, NCSU Historical Campus (click for courtesy parking request)

Speaker: Robert St. Amant, Computer Science, NCSU

Ibots: Perception and Action in the User Interface

Abstract: Interactive applications extend human abilities along an enormous number of dimensions. What can we learn from agents that use these same software tools? This project attempts to answer this question, with the the development of a specialized type of interface agent that we call an ibot,for interface softbot.

An ibot controls an interactive system through the graphical user interface, as human users do, without relying on an application programming interface (API) or access to source code. Instead of tailoring interface agents to the APIs of different applications, our goal is to build what Nils Nilsson has called habile agents: general tool-using agents. Our work has led to a programmable substrate for ibots, with sensors, effectors, and skeleton controllers for this purpose. Sensor modules take pixel-level input from the display, run the data through image processing algorithms, and build a representation of visible interface objects.  Effector modules generate mouse and keyboard gestures to manipulate these objects. These sensors and effectors act as eyes and hands to be managed by a controller appropriate for an application domain.

In this talk we discuss the practical and theoretical implications of such an approach to interface agents.

Short Bio: Robert St. Amant  is an assistant professor in the computer science department at North Carolina State University.  His work is a blend of human-computer interaction and artificial intelligence.  His Ph.D. was earned at the University of Massachusetts in 1996, and his current research deals with building intelligent, collaborative tools to help users with complex tasks.

Host: Franc Brglez, Computer Science, NCSU

Colloquia Home Page.