Home | Forums | Article Search | Subscribe & Shop | Contact Us | Log Out Technically Speaking Email This Print This View My Personal Library What's Cooking March 2004 • Vol.4 Issue 3 Page(s) 103-104 in print issue Add To My Personal Library Technically Speaking An Interview With Eric Horvitz, Microsoft’s Man Working On Tomorrow’s User Experience Jump to first occurrence of: [HORVITZ] by William Van Winkle You wouldn't expect one of the chief architects of tomorrow's user interface to boast a degree in biophysics, and a Ph.D. MD in the area of "decision making under uncertainty," but Eric Horvitz follows an unconventional path. As the senior researcher and group manager of the Adaptive Systems and Interaction Group at Microsoft Research, Horvitz is a point man for crafting technologies that improve how we interact with computers. One of his main pursuits is software that can augment a user's cognition. In this realm, his team has been working on "attentional user interfaces," systems wherein one of the rarest user resources, attention, is tracked and managed such that the entire UI revolves around assisting our beleaguered brains so we can operate more efficiently, not only on the desktop but throughout our lives. CPU: Your attentional interface technologies hinge on the computer being able to assess how "interruptible" a person is at a certain time. Can you give an example of that? Horvitz: Sure. Just now, I got back to you without scheduling a specific time to speak. We both knew that we had a goal of having this conversation, and I'm first free to do this now. I knew you might not be available, and, as you just mentioned, you had to negotiate with your family in real-time about whether taking some time now would work. Per your email, I knew you were leaving tonight for a long trip and have a deadline on another article, so it's likely a busy time for you to be talking. There are also resources to consider with scheduling, such as the fact that you wanted your recorder to capture our conversation. These are examples of pieces of "evidence" that can be used to reason about the cost of interrupting somebody at any moment. The systems that we have developed take into consideration streams of evidence about, for example, what you're doing on your desktop if you're using your computer, what your calendar says, what time of day it is, the day of week, whether or not a conversation is sensed to be going on in the office, and so on. I'm fundamentally interested in methods for making decisions with limited information and doing the best thing you can under uncertainty. I believe such methods lay at the foundations of intelligence. Within the Attentional User Interface project, we work to ascertain the best way to balance the informational benefits and the costs of interruption associated with staying aware of information users might be interested in and handling real-time communications, including the handling of telephone calls. It turns out to be an interesting and challenging area. I believe the way to really get people the maximal benefit of the connectivity we have now is to develop methods that balance guesses as to what information is valuable with guesses about the cost of bothering people. CPU: Can computers really be made smart enough to pull that off that juggling act? Horvitz: Yes, I'm optimistic, and I've been happily living with several of the prototypes. Success relies on a combination of methods. First of all, we don't necessarily want to depend solely on automation. A critical aspect is seated in design, coming up with the right abstractions, you might say, or tools that let people specify what their preferences are about information and disruption. So we need richer languages. I might like to say to a computing system, ‘If someone calls me and my Outlook calendar information shows that I will be meeting with them within an hour, give them high priority, or call urgency, and route their calls appropriately.' If Outlook knows that I met with someone within a week, I might like my system to assign them a medium communication urgency, and if I don't have a meeting with them at all, they get a low priority unless they get priority from somewhere else, for example, from an analysis of organizational relationships. I may wish to give a standing communication priority to peer and direct reports in my organization—or to people I have recently been co-authoring a document. Right now we can't tell communications software these kinds of things with ease. We can do a few things, like give users the ability to say that an email or voicemail message is urgent, but this is very coarse and binary in contrast to having richer information about context, like ‘hold all calls until the meeting I'm in now ends.' One part of the problem involves having richer languages in our interfaces. A second challenge is, how can we reduce the overhead of setting up detailed profiles by hand via developing what you might call an automated personal secretary—an agent working on your behalf, learning models of your patterns of behavior and availability . . . learning models of what you care about and how much you'd like to avoid being interrupted in different settings. CPU: Do you have any examples up and running? Horvitz: Yes. For the last six years or so, we've been prototyping internally at Microsoft variants of a system called Priorities and its descendant, the Notification Platform. Priorities is a system wherein we learn by watching a user work with email or by letting the user tag messages of different urgencies. The system basically learns to assign ‘urgency scores' to email automatically, considering hundreds of features, including words and phrases, as well as relationships between you and the sending company, whether they're in your address book or not, times and dates in email, and so on. We look at all those features and use methods from machine learning to assign an urgency score that captures what we refer to as the Expected Cost of Delayed Review, ECDR. It's a formal measure we use to represent for how much you care to avoid a delay in reviewing a message. I trust the Priorities filter in routing messages to my cell phone more than I could trust another person without having a long, tedious discussion about what I care about, as it's fusing together thousands of distinctions in the email. So instead of having to set up scores of rules like, ‘If the mail is just to me, and I am away for 30 minutes, send an SMS message to my cell phone,' Priorities considers many thousands of things. These inferences are very subtle at times. CPU: So what does Priorities actually do? Horvitz: Priorities was one of our earlier AUI systems, and it considers the urgency of email and my context. It coarsely sees whether or not I'm busy through sensing a number of states such as considering my typing and looking at application activity. If I'm not around the computer, it estimates how long it'll be until I'm back to see the email, finds an expected cost of delayed review for each message coming in, and then decides whether or not it should alert me via my mobile device by forwarding that message. I and others have relied on Priorities as a personalized information service as we travel through the world. I recall the first time I really felt like I began to depend on Priorities as a service working across my devices. I was down in California to speak at Stanford, and I noticed some time into the day that I wasn't getting those little nudges about time-critical email on my cell phone. I had this sense that something was wrong, so I called back and found there was a brief blackout in Redmond, bringing down my Priorities system. I had lost a "utility" in a blackout, analogous to the nuisance of losing electric power at home. In this case, the utility was a pipeline bringing me information about things I wished to remain aware of, provided by the Priorities agent that had come to understand my preferences. I had a colleague bring Priorities back to life for me. CPU: Sign me up for the beta testing group, would you? OK, so if Priorities was an early effort, where are you at now? Horvitz: Components from Priorities for learning personalized models of email urgency and mobile forwarding have already shipped in an exploratory product called the Outlook Mobile Manager. We have received positive feedback from users on that product. We've been working away very steadily at Microsoft Research and more generally at Microsoft on a theme that has come to be called Information Agents. It's work that permeates several products, including the new Longhorn work going on, as well as new features in Office and other products coming out over the next few years. Some portion of the ideas have already percolated into Office 2003, in the new Outlook for example, including a relatively nonintrusive fading alert that comes up to tell you about incoming email and that lets you inspect and react immediately to messages. We first demonstrated that kind of design in the Priorities' user interface awhile back. CPU: Humans are obviously far from omniscient. What impact do you feel limited cognition has on our everyday computing experiences? Horvitz: We're extremely limited creatures. No doubt human beings have very deep, magical abilities. But findings from modern Cognitive Psychology highlight the salient limitations we face in memory, attention, judgment, and other capabilities. Almost everything we do is influenced by these limitations. For example, people often interleave tasks or perform multitasking. In one of our studies we found that almost all task switching comes from external influences. In an office setting, for example, almost all switching among tasks comes from phones ringing, alerts going off, appointments coming, people standing in the doorway asking you something, and so on. Computers are wonderful devices in that they allow us to run multiple applications at any one time. They also have become our major conduits to communications, in particular email and instant messaging, and therefore, while they've provided us with a wonderful general purpose informational Cuisinart of the modern era, they also have amplified the problem of multitasking and disruption. Now, psychologists have shown in experimental settings that, as you push people to multitask, there are tradeoffs. You pay a cost for task swapping at the gain of getting more things done at once. One of the goals in our team is to figure out how computers can help people be more efficient and experience less frustration with switching among tasks and also to help protect them from interruptions. CPU: As we discussed, I'm leaving town tonight. One thing that always frustrates me is the time it takes to get back into a work situation once you step out of it. Do you have any help for that? Horvitz: Yes, and this challenge, on task recovery, is an active area of work at Microsoft Research. If someone's coming back to a task after being away for a while, we ask, how can we make it easier for them? We're interested in methods for encoding and leveraging memory cues. In one approach, we seek to remember the configuration and content of the windows associated with a project. Let's say you're working on a project, then you go on a trip and come back. You want to push one button and have the context of the project regenerated. Or if you shift between two different tasks, such as a physician switching between two patients, we'd like the computer to swap in the critical information, including background and reminders to really bring up that rich context again to minimize the cost of swapping among tasks. This is a rich design challenge. CPU: How might your AUI work someday find use in PC-based gaming? Horvitz: There are a number of links to potential gaming applications. It is feasible to provide methods that would serve as automated coaches for players that could help them to understand where and when to focus their attention. We could also make characters in game settings more lifelike by endowing them with realistic attentional abilities. Integrating models of a user's attention and memory into educational games can enhance their pedagogical effectiveness. We can also work to make artificial worlds more real by understanding how to engage the human mind and visual system. The Qualia project has been an AUI effort focused on the idea that you don't have to render all graphics at 100% fidelity. Instead, you figure out where people are likely attending at any moment and put the rendering cycles there. That is, we work to put the effort of the rendering engine where users are likely to be focusing their attention and where they would notice approximations. We've done gaze-tracking studies of users playing various kinds of games. The goal of that work is to build predictive models that can tell us the probabilities that users will attend to different regions and objects at any moment. The same kind of models can be employed in cognitively-aware compression and prefetching, enabling us to send content more quickly over bandwidth-limited connections. CPU: Will AUI technology eventually be applied throughout Microsoft's software lines? Horvitz: My vision is that the ideas, methods, interfaces, and even standard languages this work spawns might one day be used across many of our products. Having multiple entry points and familiar tools and metaphors across products could make things a lot easier for users. It might be commonplace one day to consider treating an intelligent information agent as an invisible unitary entity across products and devices, a personalized "presence" that works 24/7 for you. The work on our Notification Platform project demonstrates this vision. Notification Platform is a descendant of Priorities that handles multiple devices and multiple sources of information, including email, instant messaging, news, telephone calls, and different kinds of alerts. The idea is that, as you obtain new devices and applications or subscribe to new sources of information, you check them into a unified system that understands your preferences and goals in a way that maintains your privacy. I foresee a world we everyone can afford their own full-time, dedicated "information servants." CPU: What is your ideal vision for computing systems years and years from now? Horvitz: At the foundations of my vision for the future of computing is the prospect for building computing systems that are more like "collaborators" than they are like "tools." I envision that someday computers will grow into "companion systems" that get to know the way you think, your abilities, and preferences and time-varying needs, in a very deep way such that, whether you're working at a desk or are in a mobile setting, you're being supported with valuable computation and information in an insightful, context-sensitive manner. Such companion computing systems would augment a user's cognition by continuing to sense situations and cognitive state, and consider multiple sources of news and information and communications that you care about. They would understand if, when, and how to come forward to inform you or assist you, and they can understand enough about the project that you're working or goal that you have that they know if and how to make a positive contribution. The information–agent work is one piece of a larger dream of computational devices and communication systems evolving into collaborators. Want more information about a topic you found of interest while reading this article? Type a word or phrase that identifies the topic and click "Search" to find relevant articles from within our editorial database. Enter A Subject (key words or a phrase): ALL Words (‘digital’ AND ‘photography’) ANY Words (‘digital’ OR ‘photography’) Exact Match ('digital photography'- all words MUST appear together) Home      Copyright & Legal Information      Privacy Policy      Site Map      Contact Us Copyright © 2010 Sandhills Publishing Company U.S.A. All rights reserved.