Speaker: Robert St. Amant, Computer Science, NCSU
An ibot controls an interactive system through the graphical user interface, as human users do, without relying on an application programming interface (API) or access to source code. Instead of tailoring interface agents to the APIs of different applications, our goal is to build what Nils Nilsson has called habile agents: general tool-using agents. Our work has led to a programmable substrate for ibots, with sensors, effectors, and skeleton controllers for this purpose. Sensor modules take pixel-level input from the display, run the data through image processing algorithms, and build a representation of visible interface objects. Effector modules generate mouse and keyboard gestures to manipulate these objects. These sensors and effectors act as eyes and hands to be managed by a controller appropriate for an application domain.
In this talk we discuss the practical and theoretical implications of such an approach to interface agents.
Short Bio: Robert St. Amant is an assistant professor in the computer science department at North Carolina State University. His work is a blend of human-computer interaction and artificial intelligence. His Ph.D. was earned at the University of Massachusetts in 1996, and his current research deals with building intelligent, collaborative tools to help users with complex tasks.
Host: Franc Brglez,
Computer Science, NCSU