Site MapUser Experience for a Better World Site MapUser Experience for a Better World 
|
Dr. Susan Weinschenk: Hello and welcome to HFI's Usability Broadcast Network. Our topic today is the future of usability testing. I am Susan Weinschenk, Chief of Technical Staff and I have with me here Kath Straub, who is the Chief Scientist at Human Factors International. Before we begin with our presentation I want to invite you to submit questions you can do this anytime during the broadcast by using the link that is in the lower right hand corner of the screen. We will answer as many of the questions as we can live on air when we are done with our presentation. So today's topic is Future of Usability Testing, you know when we talk about the future of usability testing we are assuming Kath, that our audience knows about what usability testing is and we are going to make that assumption but probably should define that. Dr. Kath Straub: Okay, so what we are talking about today is usability testing, the process by which people bring representative users into a lab or office setting and ask them to do real tasks on a real prototype or real productive level interface and observe how they interact with that interface. Dr. Susan Weinschenk: So we will use that working definition as we go through. What we want to talk about today though, is some of the new trends in usability testing that we have been using in our practice at Human Factors International and then perhaps even some newer trends that we are just starting to use and are being talked about and you can see our topics we are going to start with remote testing. Now again, we probably better define what we mean by remote testing, how is that different from what you defined as traditional usability testing? Dr. Kath Straub: So in traditional testing we have an observer sitting right next to a participant watching what they are doing but in remote testing though, the observer may be in a different place, in a different city or even in a different continent from the individual who is actually using the interface. Dr. Susan Weinschenk: So a year ago I think you remember we did some research – it was a joint project with HFI, Techsmith and Michigan State University for World Usability Day in the last year and we did a study and the question was do you get the same kind of data as reliable and valid data when we do a remote testing than when you do with a regular traditional usability testing. We did find that actually the data was the same, just as useful. There was one difference though because the comment was that it was more work for the facilitator technologically having to set up remote software and recruiting and that was a little bit more work but the data was the same. Dr. Kath Straub: Yeah, and we have been working with that for a long time so we understand and now know more about what the technological constraints are and how to set that up more efficiently but what found is a really good application for this approach is that if you are doing something on your intranet or something within your organization, this becomes a very easy thing to do because you know what kind of technology your end user will have. So we have applied this in a number of cases where we have been trying to make sure there is representation of different offices across the world or even within the same country, within the same city and there are participants represented and voices heard that might not have been heard. Dr. Susan Weinschenk: Because most people think of remote testing as something you deal with outside and it's because you are in an office, not even an office, you have users across the world. They don't always think of it as doing it internally as part of an internal project. Dr. Kath Straub: Well, internally, that is the easiest way to do. Dr. Susan Weinschenk: Because you don't have to fight with the technology, right? Dr. Kath Straub: Exactly. Dr. Susan Weinschenk: Let's go on to the next topic which is automated testing and a definition of what is meant by automated testing. What we are referring to here is that instead of a traditional usability testing and here again you have the usability engineer sitting there, there is no usability engineer collecting data – not remotely, not sitting there. The data is being collected that the users are actually typing in often with the aid of another piece of software; windows pop up and ask or give instructions and ask about the experience as they are going through. There was a study, a number of studies but one of them done most recently by West and others to see do you get the same results when you use automated testing and traditional testing. So they compared this together and they found out the testing was similar for automated and in-person testing, both quantitatively as well as qualitatively. Dr. Kath Straub: Did they report this anywhere? I have some reservations about what they found. Dr. Susan Weinschenk: You're not sure that they really can make that claim that quantitative and qualitative... Dr. Kath Straub: I'm not really sure that – the quantitative data may be the same but qualitatively I see a difference between having a remote testing, having a professional observing what's really happening and completely automated. Dr. Susan Weinschenk: So let's see what they found and you can maybe point out where you're saying that the qualitative claim might be a little too strong right? Dr. Kath Straub: True. Dr. Susan Weinschenk: So they did find that it provided the same results but in terms of their actual data and completion rate etc. but the remote participants did spend less time per task. Dr. Kath Straub: And that may be important because that tells us something about having an observer in the room, maybe there is a difference in the interaction. If you're watching me, I am going to feel different about I am interacting with the interface. Dr. Susan Weinschenk: Right. Dr. Kath Straub: So I may behave a little bit differently. Dr. Susan Weinschenk: Although you know some people and we will come back to this later when we get to another topic, some people would claim that it is more realistic to not have the observer there. Dr. Kath Straub: That's another way to think about it but I am not sure that we will see that this is supported with the rest of the results that we are going to talk about. Dr. Susan Weinschenk: Okay. Next was that remote participants were 15% more likely to give up on the task than when the usability person was in the room. Dr. Kath Straub: And that's another actually interesting result because that might be more realistic. In a lab situation, if you are looking over my shoulder, I may feel that actually because you are Dr. Susan Weinschenk: that I better finish the task. Dr. Susan Weinschenk: (Laughs) Right. Dr. Kath Straub: So I may actually get a more realistic result here because I feel I can't give up when I would normally give up at home because I am at home. Dr. Susan Weinschenk: Right. So in that case, it would be more realistic. Another finding here is that the automated, the people in the automated condition wrote comments about why they were not able to complete the task but they were only at the, immediate level. They didn't get into underlying, what the researchers felt as secondary issues, as much as the in-person. Dr. Kath Straub: Right. Here's where I have a couple of concerns about the automated testing. We do these kinds of testing tasks ourselves to see what kind of results, at least I do, I don't know if you do. Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: I do these kinds of things. I have a lot of energy at the very beginning of the task and I type a lot but as I get further and further in the task, I type less and my comments become more cryptic and sometimes more humorous. Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: And less useful and what they found actually, they said was that as you said, they saw more of these primary issues but none of the secondary issues so the value that I can bring if I am observing you is that I can ask you questions that might help you understand that you might not be able to think about really happening in terms of your cognition. Dr. Susan Weinschenk: Right, you can do that probing. Dr. Kath Straub: Right. Dr. Susan Weinschenk: Which you absolutely can't do in an automated test. Dr. Kath Straub: Right. Dr. Susan Weinschenk: And lastly, that the usability engineers did catch additional issues that the users themselves didn't catch. Dr. Kath Straub: This makes me feel good ... Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: Because that means we bring value to our process. Dr. Susan Weinschenk: (Laughs) Alright, relevance of this particular study in that? Dr. Kath Straub: So automated usability testing can be useful; can provide that data but you actually don't get exactly the same data. You don't get the probing. You don't get the "thinking-out aloud" and getting insight into why the users are doing what they are doing. Dr. Susan Weinschenk: So let's move on. Now we've got a study that talks about fidelity of prototypes and you know, I don't really give away my age in any way but I do go back to the time when we would talk about doing testing with hand-drawn prototypes. We didn't really have any prototyping tools back then so our choices were either hand drawn or we wait until the whole system is done and you won't remember that, I know. Dr. Kath Straub: No we do, do hand drawn prototypes once in a while, not now but just when we want to be nostalgic. Dr. Susan Weinschenk: (Laughs) So this question about how much fidelity, right, to put it into a prototype when you're testing it, should you use something very, very rough or should you use something that looks like the finished product? This study I thought was interesting by McKirney and others who said, "Well, you're kind of asking the wrong question" because the real issue is not do I use high or low, but it's a little more complicated than that. Dr. Kath Straub: So the question becomes, what is fidelity? Dr. Susan Weinschenk: What is fidelity? And so here are the, they have five measures of fidelity and they are saying is what you should do is mix and match these and make conscious decisions. So do I want, for instance, a high level of visual refinement but perhaps not as much functionality? Is that, for this particular task, going to give me the data that I want? Dr. Kath Straub: And then you would be asking a very different question and you would be asking – so did the graphics or how did the visuals impact my use of the interface rather than did the interface work? Dr. Susan Weinschenk: Right. Dr. Kath Straub: We have several examples where we have seen that the visuals can actually interfere with usability testing, even as small as adding color to your wireframe can really be distracting. Dr. Susan Weinschenk: Right. Dr. Kath Straub: So we have an example of a client we worked with where we just happened to use the default colors which were blue and purple on the wireframe that we printed out... Dr. Susan Weinschenk: Default colors for the? Dr. Kath Straub: For fireworks, I think, the software that we were using, that was the color. Dr. Susan Weinschenk: Oh, the software that you were using. Those were the colors. Dr. Kath Straub: It was the some software and we thought we'd have fun and try it out. Dr. Susan Weinschenk: Right, okay. Dr. Kath Straub: We brought in paper prototypes. We set them down in front of the first participant. It was an intranet study and they looked at that paper prototype and they said, "Purple, green, blue? Those are very cold colors." Dr. Susan Weinschenk: And that was just an artifact of the printing up. Dr. Kath Straub: Well, what ended up having to do was go to the photocopier and copy the wireframes. Dr. Susan Weinschenk: So copy them in black and white so that they didn't have color. Dr. Kath Straub: Otherwise people were just distracted. Dr. Susan Weinschenk: So let's go on and see what their conclusions were. So they found that this mixing of the different prototype dimensions really helped the designers get the right prototype for the test. Dr. Kath Straub: So really what you're saying is that we want to talk about a more complicated way of thinking about fidelity in a prototype means and to be able to vary things on more parameters. Dr. Susan Weinschenk: Yes, I think that's more sophisticated and given you know, the kind of software we have now to be able to do this, it really shouldn't be that hard. So the idea is to make a conscious decision on those five parameters. Dr. Kath Straub: In what way are you going to test, right? Dr. Susan Weinschenk: Right. So our next one has to do with how many designs do we want to test at a time? And this is interesting, you know, sometimes we know that we are going to test more than one design because we are actually doing a comparative test, right? You have two or three different alternative designs that you want to test. Otherwise, I think what we tend to do is you know, we test the one. Usually we say, "Oh, let's test a couple as it's going to help with the data that we collect." Dr. Kath Straub: Right. It turns out that it may help us. Dr. Susan Weinschenk: Maybe we should do that. So this is a study in which they check that. Actually let's go to this page where they describe the materials and the methodology a little bit. Dr. Kath Straub: Right. Dr. Susan Weinschenk: They had three alternate paper prototypes and they don't say whether they are on the five dimensions we just talked about before but actually I remember this you know, particular study. They were fairly low-fi prototypes hand drawn. So they had three paper prototypes and they had, the three designs used the same amount of fidelity and the same functionality but different design decisions – interaction decisions. They looked at ease of use, interface design usability, they looked at how willing were the users to criticize the designs and the number and type of re-design suggestions that they got. What they found was that people gave a higher rating when they were only testing one rather than the three so they actually what I would call, nicer. They tended to say, "Oh, this was good, this prototype was good" and they gave them higher ratings than when they were comparing them. Dr. Kath Straub: It is easier to be more critical when you have something to compare with side-by-side. Dr. Susan Weinschenk: Compare with against, right. The number of positive comments was significantly higher for each design when they saw them individually rather than in the group and there was no increase in the number of re-design suggestions as when they looked at multiple designs. Dr. Kath Straub: So what you're suggesting is more prototypes gives us better data because we get more critical data which is really what we need. Dr. Susan Weinschenk: I think we do. Unless you just want people to say that your design is wonderful and then you go home, right? So when only one design is seen, participants rate it as being higher, they have more positive comments than they do but they don't provide more re-design suggestions either. I think then, it would be more work to come up with multiple designs. Dr. Kath Straub: But I think you have a better sense of who is getting what and what they might need and getting the critical comparisons that you might make. You know, we forgot, there is one comment that I forgot in the last page because one of the things that I thought was really interesting was in the, when it was only one design, they wouldn't reject it. Dr. Kath Straub: Right. Dr. Susan Weinschenk: So they might say they have problems with it but they wouldn't say, "I would never use this" but when they had multiple designs, they would actually point to one of the designs and say, "I wouldn't buy that." Dr. Kath Straub: So you're almost putting them in a forced choice situation. What would you put your money on, this one or this one? Dr. Susan Weinschenk: Yeah. But then I think you get more, I mean that's actually data that you didn't get before. Dr. Kath Straub: Right. Dr. Susan Weinschenk: Alright so now we start to move into some of the, I think, the leading edge that's a little bit, even newer and that has to do with, well, it's interesting, it's challenging, possibly challenging the think-aloud technique which you know, you are not allowed to challenge in the field if usability testing, right? Dr. Kath Straub: Oh, I don't know. It comes up at every webcast, doesn't it? Dr. Susan Weinschenk: (Laughs) the think-aloud technique obviously being the technique during the test, the participant is talking aloud, thinking aloud about what they are doing and that's important. Dr. Kath Straub: Because of the proofs that allow me to know what you are thinking. I'm just watching you from over your shoulder, I don't know why you are making the decisions you are making or why, what you're looking for? Dr. Susan Weinschenk: How you're thinking? Or if you're thinking about it at all, right? Dr. Kath Straub: Exactly, but then the flip side is of course, is that if I am asking you to talk out loud, the play-by-play that you provide me may actually interfere with what you normally do... Dr. Susan Weinschenk: Right. Dr. Kath Straub: Because you are thinking about it at a different level. Dr. Susan Weinschenk: So this was some research that was done about this whole aspect, and they're making a distinction, they're calling it CTA because we need more acronyms. Dr. Kath Straub: (Laughs) Dr. Susan Weinschenk: They call it CTA for "Current Think-Aloud" meaning the person is thinking aloud while they are using the product or are in the test, versus RTA – "Retrospective Think-Aloud". Dr. Kath Straub: Retro's cool these days. Dr. Susan Weinschenk: Right, retro is cool and retrospective means that you don't think out loud while you're doing the test, right? Dr. Kath Straub: Right, so you don't think out loud while you're doing the test but you do it after you are finished and I think that this is an interesting question because now do I have, I mean do I have you do it after I finish the entire test or do I have you do it after you have finished an individual task? Dr. Susan Weinschenk: Right, which I don't think this particular study got into but that's great question. You know, one of the things that we worry about with people talking afterwards is the whole memory associated. They don't necessarily remember or report exactly what happened. Dr. Kath Straub: Well, memory is a fragile thing to start with. Dr. Susan Weinschenk: Right. Dr. Kath Straub: If I have been doing all of these things that are very similar over the course of an hour, I may not remember what I was doing. It's a little bit like remembering where you parked yesterday. If you park in the same place, you remember. Dr. Susan Weinschenk: Right, right. So what they did in this study is that they looked, is they actually used eye tracking to compare this. So they looked at whether if you waited for the retrospective and compared it with the eye-tracking, did the two things match? So when people say that, "This is what I did and this is where I looked" they really base it on the eye-tracking data. So what they found out was actually that RTA was fairly reliable and they even mentioned in here, we have a number for them, there is only a small amount of what we call fabrication. It doesn't mean people are intentionally lying but just a mismatch between what they remembered and what they had actually done. Dr. Kath Straub: So maybe if the duration is short enough between when I do the task and the time that I reported back. Dr. Susan Weinschenk: Right. I don't remember that in the study. I'll have to back and see if they talk about that. They found that RTA, that retrospective was reliable both for simple and complex tasks, that the verbalizations that they got in retrospective you know, were very rich and with a fair amount of explanation but looking at the last one, at number five, they did admit some information. So the AOI sequences meaning the data from the eye-tracking about where they, about where they had exactly looked, not all of that detail was in the retrospective. Dr. Kath Straub: Well, eye-tracking is actually capturing a much richer set of data that we would typically find. Typically, we're just looking over your shoulder and trying to figure out where you are looking on the screen but with the eye-tracking I can see where you are looking and often where you are looking is many more places than you think. Dr. Susan Weinschenk: Really? So you mean that people's experience of you know, "I think I looked here and here and here" but the data would actually show... Dr. Kath Straub: Right, in fact you can take videotapes of where you can ask people to follow where they think they are looking at with fingers and actually overlay that on eye-tracking and show that they are looking at many more places in the screen, sometimes for very short periods of time but still you're getting exposure to that and if you're getting exposure to that information, it may have some influence over what you do or how you think or what you do next. Dr. Susan Weinschenk: So she's getting into a topic and we are going to talk about it before we are done but not quite yet, which has to do with how conscious are we really, of what we are doing, right? Dr. Kath Straub: Right. Dr. Susan Weinschenk: We'll come back to that. So the conclusion from this study then, the relevance is that you can rely on the retrospective, it does not have to be current think-aloud. This next bullet kind of gets into an interesting one, Kath, because why would you? I mean what's wrong with CTA? Why not just use CTA? Why are we even interested, in RTA? Dr. Kath Straub: Well, like you said at the beginning, if I ask you to stop and tell me what you are thinking about all the time now I am changing the path of what you would normally be doing so now I am beginning to worry if I am interfering with you by being there and watching and asking you questions. Dr. Susan Weinschenk: Right. Dr. Kath Straub: So this is probably why most of us hold back on asking proofs until a person is finished with doing a particular task. Dr. Susan Weinschenk: Right. So this might help, and I don't know of a lot of studies but I think most people are still doing CTA. Dr. Kath Straub: Right. Dr. Susan Weinschenk: But using RTA might help with this whole criticism of "Oh it is not realistic." Dr. Kath Straub: Right. Dr. Susan Weinschenk: Right. Dr. Kath Straub: This is one the things that we really need to be worried about how much do we interfere with the process by being in the process the first place? Dr. Susan Weinschenk: Right. So that takes us to... Dr. Kath Straub: The eye-tracking piece. Dr. Susan Weinschenk: The eye-tracking piece. Dr. Kath Straub: So the eye-tracking piece is offering us a new way of looking at data in a very refined fashion. What we are looking at is not only what you are doing but also what you are seeing on the screen that's grabbing your attention and this was really I think, interesting to us because it's allowing us to start talking about the intersection between usability and marketing. We know that graphics are important and we know that they feed into the process but we don't know, really how they, or we can't really capture how they capture your attention on the screen. Dr. Susan Weinschenk: But you don't use eye-tracking just for graphics? Dr. Kath Straub: No, you can use it for text and reading and all kinds of things but at the first impression, this is going to be a really interesting look at. Dr. Susan Weinschenk: Look at. Dr. Kath Straub: And if we bring this in by the way in conjunction with other kinds of information that we have so – click stream analysis data – we can take what people are doing in the lab and confirm it with what people are doing in their homes based on regular Linux. Dr. Susan Weinschenk: Wow, so give us some examples here of eye-tracking and for what it's used and what kind of data you can get out of it. Dr. Kath Straub: Well one of the most interesting things we see in the eye-tracking space is this notion of a heat map. The heat map is really a bit like the hurricane map or the weather map that you get on TV and it's just showing you, aggregated over different kinds of slices of the data, where people are looking and in some cases, where they are not. So if you are having people doing a task and this turns out to be important by the way, if you're going to have a heat map and if you're going to have an eye-tracking set up for this, you don't want to just let people surf, right? Dr. Susan Weinschenk: Right. Dr. Kath Straub: We know most people go to the webs to do something. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: At least for the tasks that we typically test. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: There are entertainment sites which are maybe a little bit different and we need to make distinctions about those. Before a task that we are going to look at, we are going to ask you to do something specific and we designed the site so that you are going to look at a particular place and you are going to find a particular link or a particular button and you are going to do something. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: So what I can see with eye-tracking is where you're looking on the way to that button and how often you look and how long you linger. For instance, if I have you come to particular spot and you linger a long time and then look away, what have I learned? Dr. Susan Weinschenk: I don't know. What have you learned? Dr. Kath Straub: I've learned that somehow the link name isn't as sufficient as I want it to be. Dr. Susan Weinschenk: Oh, because you spend a lot of time in there but then you didn't do anything. Dr. Kath Straub: It didn't capture me. Dr. Susan Weinschenk: So when we're looking at a heat map like this, what do the different areas tell us? Dr. Kath Straub: What are the different areas? Dr. Susan Weinschenk: The different colors... Dr. Kath Straub: Ah! So again, it's like a weather map. The hot spots, the red spots, the spots that people were looking at longer or were looking at more frequently, depending on how you do the analysis. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: And one of the really important things in eye-tracking is that the analysis is not trivial. So you get a lot of data, it is new data. Dr. Susan Weinschenk: Oh, so does that how do you come through all that? Dr. Kath Straub: Well, you have to, as we said earlier, you have to be knowing what your questions really are, you have to think ahead about what kind of questions you're going to ask and the data that will be formed with those questions and when you're looking at your data, you have to be sure that you are looking at the right data. Dr. Susan Weinschenk: But that is similar to regular usability testing? Dr. Kath Straub: Sure. Dr. Susan Weinschenk: But a little bit more complicated. Dr. Kath Straub: You're going to have a little bit more data to look at. Dr. Susan Weinschenk: Okay. So usually it will show you a little ‘x' where they clicked. Dr. Kath Straub: Right, so it depends on what you're doing with your eye-tracking software. In some cases, it will show you where you been and stayed. Dr. Susan Weinschenk: So you can set the different parameters. Dr. Kath Straub: Sure, absolutely. So in this case, the red is where they clicked. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: In this case, this is just a public site that we asked the user specifically how to start their own company and you can see the hot spots of where they were looking across the page. Dr. Susan Weinschenk: Okay, and how about on this screen? Dr. Kath Straub: Basically what we are looking at is where the person is seeking to find the link that they need to get started on how you start your own company if you happen to live in the city of San Diego. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: So what we saw is that this particular user was a bit obscured by the bullet, by the bubble that is here but the user started looking at the primary navigation and then if you actually look at the site they looked longer at those first three but they looked less at the next two. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: So they start to do the shadow splicing thing, are you going to tell us what satisficing is? Dr. Susan Weinschenk: No, you're going to tell us what satisficing is. Dr. Kath Straub: I see. Okay, so we're going to look just as far as we need. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: It's the lazy user model, right? Dr. Susan Weinschenk: Now let me ask you when we are looking at a heat map like this, is this from one person? Or is this from all the users? Dr. Kath Straub: That's a good question. That's another one of the things that you have to decide when you are eye-tracking data. How are you going to handle the data? In this particular case, the heat map that you are viewing is one user only. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: But actually to be useful, we wanted to find out over the set of users, where people are looking. Dr. Susan Weinschenk: Right. Dr. Kath Straub: We don't want to have just one person looking at one place. There are ways to aggregate this data. Dr. Susan Weinschenk: Aggregate the data, yeah, you can do that. Dr. Kath Straub: With all the users so you can find out where people are looking. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: So in this case again, what you're just seeing is more about how these people or this one individual looked at a particular page so they were looking to the portal style of navigation to figure out where more details are so I am looking at the title and I am using the title to drive me into the links below. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: In this case, this is actually a fairly effective title, right? Dr. Susan Weinschenk: Yes. Dr. Kath Straub: Because I want to look below it. Dr. Susan Weinschenk: Right, it makes you want to look at it. Dr. Kath Straub: And I am getting validation that that's the case. Unfortunately, we also see cases on this page, where people are drawn to the wrong point. Dr. Susan Weinschenk: They're looking and there is a lot of activity in a place that is useful. Dr. Kath Straub: So something catches their eye and they get stuck because they are thinking hard. Dr. Susan Weinschenk: Now what about here? Dr. Kath Straub: This turns out to be a real big problem for the city and also shows how we can use eye-tracking in a way because now we are looking at cold spots. Dr. Susan Weinschenk: So cold spots mean? Dr. Kath Straub: Cold spots mean that you never look there. Dr. Susan Weinschenk: So nothing shot up and when I look at my data, there is no color, there are no bubbles, there are no heat spots. This person, in aggregate data, didn't look at the page at all. Dr. Kath Straub: That's right and so all these ads which are on the right-hand side aren't really ads because this is a government site. Dr. Susan Weinschenk: But they look like it. Dr. Kath Straub: They do look like ads and people ignore them. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: So we want to be very careful because we now can show that we've had this myth of banner buying and people worry about it. Dr. Susan Weinschenk: Right. Dr. Kath Straub: Here we can show it is not so. Dr. Susan Weinschenk: It's not a myth, right? Dr. Kath Straub: It really exists, right. So it's not just hot spots and cold spots, there is this aggregation also of what I am doing, how long do I look? How long do I linger? Do I just jump in and make that click? The eye-tracking data is going to give you a richer source of data that you will be able to evaluate. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: With that note, on this page, it's actually interesting. It's a home page and we still think that people don't like to scroll on a home page. Dr. Susan Weinschenk: Right. Dr. Kath Straub: In this case, for this particular user, it is absolutely true. Dr. Susan Weinschenk: They never scrolled. Dr. Kath Straub: They never scrolled. Dr. Susan Weinschenk: They never went below. Dr. Kath Straub: He saw nothing below the scroll. Here we have a kind of... Dr. Susan Weinschenk: This looks very different. Dr. Kath Straub: This looks very different but in this case what we are doing is we have the same user doing the same task and where he looked and in what order. Dr. Susan Weinschenk: Oh, okay. So that would just be another, it's another way of looking at the data. Dr. Kath Straub: Right, and in fact, this was a little bit more complicated. This speaks that it needs to be typically I'd want to see where you looked for the first or ten seconds. Dr. Susan Weinschenk: You don't need to see it all. Dr. Kath Straub: I don't need to see it all but in this case, it is interesting too because what we see is a similar representation in that you notice that the bubbles are bigger in some places than in others. Dr. Susan Weinschenk: Yes. Dr. Kath Straub: The big bubbles are where the person looked longer. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: Right? So I am getting a similar kind of data in some ways. Dr. Susan Weinschenk: But it looks like it is in order now. Dr. Kath Straub: Right, we have the order now so I know what order you looked at it and know that the first person at was the mayor. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: Interesting part. Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: People draw people's attention. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: So even though they ignored all the ads, they did look at the mayor. Dr. Susan Weinschenk: They did look at the mayor. Dr. Kath Straub: So the mayor's happy. Dr. Susan Weinschenk: Maybe the mayor, we should put the mayor's face down where we have the ads. Dr. Kath Straub: We could try that and see how that works. Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: Okay, but actually people do scan through the site. You see that people do a quick scan and then just decide based on the labels. Dr. Susan Weinschenk: Yeah then they decide where to go. Dr. Kath Straub: So yeah, the work we've done is that labeling is one of the most important things is confirmed again with the eye-tracking data. Dr. Susan Weinschenk: And then it looks like they also zeroed in onto the links. Dr. Kath Straub: In fact... Dr. Susan Weinschenk: They do, they do look for "what's the link?" Dr. Kath Straub: Right. In this particular case, this user was very link-oriented so you will see saccades through; he never stops to look at the text, to read through it but he only looks from link to link to link to link. So see if he wants to satisfy the cannon and jump in. Dr. Susan Weinschenk: Yes. Dr. Kath Straub: So he is looking at various points of entry too. If we look over at an aggregate of this particular user's tasks, we can see his pattern over tasks shows that he has an approach of scanning pages that says look over the primary places where global and local navigation typically live and if you don't find what you want, look at the other links on the page. Dr. Susan Weinschenk: So you really like using eye-tracking, you're starting to use eye-tracking more and more and more in your studies. Dr. Kath Straub: I think it's giving us a lot more data to work with so that we are asking more sophisticated questions and we are getting more sophisticated answers by doing that. Dr. Susan Weinschenk: Is it hard to do? I mean, if someone says, I want to try eye-tracking, is this very difficult? Is it expensive? Dr. Kath Straub: Well, that's a complicated question. I don't know that you want to buy an eye-tracker and set it up in your garage per se. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: But, you and I might. Dr. Susan Weinschenk: (Laughs) Yeah, we probably would. And it probably would be in your garage or mine. Dr. Kath Straub: Well, I don't know. I'd like to have my own. Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: The idea is that it's a different approach and it's a different set of data and different assumptions and if you are not using your data correctly then just as with any other technique, it would not help you. Dr. Susan Weinschenk: But if you wanted to try it, there are locations? Dr. Kath Straub: Absolutely. Dr. Susan Weinschenk: I mean that there are labs. Dr. Kath Straub: In fact we have a lab in Baltimore. Dr. Susan Weinschenk: Yeah, we have a lab in Baltimore. You could use a lab that had an eye-tracker and try it out. Dr. Kath Straub: Sure and it's fascinating to watch because you're getting a different kind of data. Dr. Susan Weinschenk: It's really fun, I have to say. It's very fun to do. Now I have, let's see what's on the next slide because I have the question, it's something that has always been bothering me about eye-tracking and it actually goes back to 1967 before eye-tracking was all that popular, you know, for usability testing. Dr. Kath Straub: Absolutely. Dr. Susan Weinschenk: And the point is I think we forget sometimes, that where people look and their eye movement depends on what you've asked them so that's important in usability testing. Dr. Kath Straub: Right, it becomes even more important I think because the way that you interact with it during the usability testing... Dr. Susan Weinschenk: Is going to affect it, right? Dr. Kath Straub: Exactly. Dr. Susan Weinschenk: That's why you need that RTA. Dr. Kath Straub: RTA – In fact, one of the things that they are doing, that we tend to do as people and you're watching us now right? We're looking at each other. Dr. Susan Weinschenk: Right. Dr. Kath Straub: If you ask me a question or a proof, I'm going to tend to look back at you and that means I'm not getting a clear track on where you would normally follow on the screen. Dr. Susan Weinschenk: Oh, interesting, okay. So let's look at this study done in 1967, what they did is a, this is not a usability study per se, but they had a picture of characters in a room and then they asked different questions to see where do people look and you'll see on the picture of the, that's to the right, that's where they-the people looked when they weren't given any instruction and all. They were just shown the picture and they were you know, told, "Look at the picture, look at the pattern there and compare it with the bottom left." They were asked to estimate the economic level of people in the picture and in the bottom-right, they were asked to judge the ages of the people and I have to turn the page one more time here, top-left, guess what they have been doing before they arrived, top-right, remember their clothing etc., so you can see based on the question, they had a very, very different pattern. So you really have to pay attention to that. But you know, all of this about eye-tracking and that question we had before, remember I said that we were going to come back to that? The question I had was – are people really, really conscious of what they are doing? I am fascinated with this, I mean I've worried recently about the think-aloud technique because of this issue – do people really know? The book that I love you know, and I talk about it all the time and it's called as "Strangers to Ourselves – The Adaptive Unconscious" and it's a book about the fact that there is so much of our behavior and our decisions that are, we do that are not available to our conscious mind. We are taking actions, we are making decisions and we can't possibly, really discuss them. I don't think we can and actually the researcher shows that people will make up what sounds like very good reasons for what they did and they are making it up, right? So how do we, what do we do with that? Dr. Kath Straub: Well, it is important to make sense right? So it is important for me to feel like I make sense so whatever I did, I have to describe it in a way that is consistent with what I think I mean. Dr. Susan Weinschenk: Right, but that can't really be accurate? Dr. Kath Straub: Well that's why cognition is an interesting thing to study, right? Dr. Susan Weinschenk: Right. Dr. Kath Straub: So we are really trying to understand the processes and it turns out that it is a fairly complicated thing. Dr. Susan Weinschenk: Right. Dr. Kath Straub: Thinking isn't as easy as it seems. Dr. Susan Weinschenk: Right and it might not be what is actually going on under the surface. Dr. Kath Straub: Right. So, how do we look at that? Dr. Susan Weinschenk: I don't know. I mean obviously the eye-tracking starts to get us into the realm of making measurements that are not relying upon conscious processes. Dr. Kath Straub: Right, but even this is difficult, right? Even then we are starting to make assumptions that say where the eye goes gives me some window into what the mind is doing then. Dr. Susan Weinschenk: And maybe it doesn't right? But it would give you, certainly, a glimpse into actual behavior. I mean you know where they are looking. Dr. Kath Straub: Right. So at least I can say something about the concrete, the person took this much longer and made a decision or not about that. Dr. Susan Weinschenk: But you know what I want, of course? Dr. Kath Straub: Yeah. Dr. Susan Weinschenk: I want, I'm looking for and for those of you who have a background in psychology and psychology research might get this, but I want someone to invent a galvanic skin-response mouse, okay? So you're using the mouse and you're looking at the website and the mouse would actually be measuring the electrical conduction. Dr. Kath Straub: So I would say that this is a call to action for everyone who listening. Dr. Susan Weinschenk: Yeah because then we would know what got, you know which page got their attention rather than them saying it, we'd measure their attention. Dr. Kath Straub: Well, tell me more about what you mean by attention? Dr. Susan Weinschenk: Attention in the sense, well I don't know, you know, you'll just have to do some research on this. Dr. Kath Straub: I see. Dr. Susan Weinschenk: Then you can report it on the future website but I don't think the galvanic skin-response mouse actually... Dr. Kath Straub: Exists so far? Dr. Susan Weinschenk: No, not yet. Dr. Kath Straub: Hopefully we will get one. Dr. Susan Weinschenk: If you guys have one there, let me know. Alright, well that's the end of our presentation. So we're now going to have our Q&A time and we've had some questions come in. I want to remind you, if you haven't submitted your question and you're interested in having us answer one, you can still submit it. Use the link on the lower right-hand corner of the screen and I should also mention before we are done, we do have a white paper, actually not on this exact topic, it's on usability testing. Dr. Kath Straub: But it is on a related topic and actually an important one – the usability testing and the white paper is about how to get the right participants for the usability testing. Dr. Susan Weinschenk: Right. Dr. Kath Straub: So the right participants are important because if you're testing people who are not representative, I'm not sure there is much value to the test. Dr. Susan Weinschenk: To your test at all, yeah. Dr. Kath Straub: So testing your fellow developers or the people down the hall, unless it's an intranet and they are going to use this site, might not be the right thing to do. Dr. Susan Weinschenk: Right. So you can download that from our website but let's see, we've got some questions coming in. Dr. Kath Straub: We have some important questions coming in. The first one is the one you will like. QUESTION AND ANSWERS SESSION: Dr. Kath Straub: Will the presentation be available for later review because I'm watching during work? Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: Oh. Well, if we had the eye-tracker on them, we'd know that they weren't working, they were watching their presentation. (Laughter) Dr. Kath Straub: Anyway, they want to be able to watch it again and concentrate, so yes, the answer is yes. Dr. Susan Weinschenk: The first week of October it will be on the site. Dr. Kath Straub: Oh, the first week of October. So we have these, these webcasts are archived on our website and you can go there and download and watch again but it's going to be the first week in October before this one's available. Dr. Susan Weinschenk: That was an easy one. Dr. Kath Straub: Okay, well I have a harder one now. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: So in a test where they were comparing multiple designs did the order of the designs that were shown, and I don't know if you remember this detail, have any effect on the responses or did they randomize the order. Dr. Susan Weinschenk: Yeah, they randomized the order. Dr. Kath Straub: Okay, so that was an important design factor. Dr. Susan Weinschenk: Right. Dr. Kath Straub: Well, that's two easy ones. Dr. Susan Weinschenk: That was an easy one. This is good. Dr. Kath Straub: Now a harder one. Dr. Susan Weinschenk: No, more easy. Dr. Kath Straub: Do you think that they would have similar results for formative testing as opposed to summative testing (when comparing multiple designs)? Dr. Susan Weinschenk: Which one are we talking about? Dr. Kath Straub: I think we are still talking about the prototype where they put side-by-sides. Dr. Susan Weinschenk: Do you think that they would have gotten similar tests for formative as well as summative? Dr. Kath Straub: Maybe we should talk about the formative and summative tests. Dr. Susan Weinschenk: Why don't we do that? Yeah, do you want to define that? Dr. Kath Straub: No. Dr. Susan Weinschenk: But you've defined them all, you even defined satisficing. Dr. Kath Straub: Formative testing is the proof that we have been testing, right? The testing that I am doing as I'm designing the process... Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: Helping me to guide and refine and tweak up the process where it really is. Dr. Susan Weinschenk: Right. Dr. Kath Straub: And then summative testing is – it's the summary. Dr. Susan Weinschenk: Yeah, but you know so many times people think that summative testing is after the product is all programmed so it's not actually but you can do summative testing on an interface. Dr. Kath Straub: Sure. Dr. Susan Weinschenk: But it just means that you are done as far as you know, you've designed the interface, and you want to make sure that you've got it right before it gets turned over for implementation. So what do you think, though? Do you think that it would make a difference? Do you think that they would have gotten some other results on that one for formative as opposed to summative? Dr. Kath Straub: Well one of the things that I think that is really interesting is that is that this acknowledged anecdotally, we see that the lower fidelity in terms of visual presentation... Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: That your prototype is, more people are going to break at it. Dr. Susan Weinschenk: Yes. Dr. Kath Straub: More people would break it down so it would be an interesting test to run. Dr. Susan Weinschenk: Yeah, I don't, we're going to guess that you would see a difference, is that what you're saying? Dr. Kath Straub: Yeah, but not necessarily because of the 3 versus 1 but because it... Dr. Susan Weinschenk: It's a prototype. Dr. Kath Straub: Yeah because it is a prototype issue. So here's another question, My company is currently discussing the fidelity issue on our prototypes. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: How much does one rely, how much does one rely on the user? Dr. Susan Weinschenk: On the user? Dr. Kath Straub: How can a user decide this? Dr. Susan Weinschenk: I'm confused by that question. Dr. Kath Straub: I'm confused by that question as well. Dr. Susan Weinschenk: Because you wouldn't let the user decide on that. Dr. Kath Straub: The whole idea of usability testing is that you let me watch you and draw conclusions while you're doing that. Dr. Susan Weinschenk: Yeah. We're always in control. We'd never want the user to decide on the fidelity. (Laughter) Dr. Susan Weinschenk: There are two things but yeah that's up to the researcher to control. Dr. Kath Straub: Okay. The next one, Could you point to some good usability frameworks or guidelines? I have been using the Microsoft usability guidelines from MSR. Is there something that's more recent or build on that in a meaningful way? Dr. Susan Weinschenk: This doesn't really have to do with usability testing. Dr. Kath Straub: But it is an important question about resources. Dr. Susan Weinschenk: Okay, so I mean, guidelines, do you want to tell them about guidelines? There has been a lot out there and we have of course, our product, Usability Central, which has usability guidelines. If you think, and I will just answer this quickly since it is off the topic, if you think of our conversation over the day, I think it's most, what most people are doing their own custom guidelines and standards. So if you have, for instance, basic templates that's for instance in our product, Usability Central, what's really important is that you customize them. I think it's very hard to use generic guidelines within an organization. Dr. Kath Straub: Okay, here's another more complicated question. Dr. Susan Weinschenk: This is a, there is a lot of writing on that. Dr. Kath Straub: There's a lot of writing. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: Okay so we're going to go back to the question about testing in an automated way... Dr. Susan Weinschenk: Okay. Dr. Kath Straub: And the value of the data and the difference between the qualitative and quantitative data. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: Is there data on the cost-benefit comparison of traditional observer testing vs. automated testing? If the majority of primary usability issues are found through automated testing, what is the quantifiable value in dollars of the secondary issues found by the observer vs. the additional cost of those studies? Dr. Susan Weinschenk: Yeah, so? Dr. Kath Straub: So, I think I'd like to say that the usability expertise is brought to bare when I have an observer of the usability tester is actually going to help me refine the usability of the interface in a significant way essentially when you are doing usability test with an observer, you're sort of getting an expert review on this that's not for free. Dr. Susan Weinschenk: Well, you know one thing that I think that is really important to remember is that the usability expert is not just taking down the data. In an automated test, the person is typing in and responding and that data is collected but there is not an interpretation and how many times have you seen someone do something in a usability test and say, "Oh, it's too many clicks" when really, the usability expert can see that by a combination of clicks they have made, it is not – the problem is not too many clicks, the problem is that they don't understand what they had on the screen before that's something that will never get picked up by an automated test. Dr. Kath Straub: So this goes back to your comment about what we think we do is not what we do. Dr. Susan Weinschenk: Right. So in an automated test, you are relying solely on you know, what the user types in and what they click and you don't have that interpretation but I also want to make another point here because automated testing is actually, I mean the main reason that we suggest automated testing is when you want to test large numbers of people. So sometimes, you need a very, very large sample size. You're asking very specific questions about a very particular part of a product and you need for reasons that we can go onto in another webcast, you know you need a large sample size then that is, you know very cost-effective. Dr. Kath Straub: But to me that sounds more like a survey than a usability test. Dr. Susan Weinschenk: But doing automated testing, you really don't like automated testing? Dr. Kath Straub: I have no reservations. Dr. Susan Weinschenk: So we disagree, we'll agree to disagree on this. I think automated testing has a useful purpose but I do not think it replaces traditional testing. Dr. Kath Straub: Well, then we agree. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: That's disappointing. Dr. Susan Weinschenk: Alright. Well, one of these questions will have more controversy. Dr. Kath Straub: Okay. How about this one, Does the use of color help people locate web controls such as drop-down lists and navigation cues such as hyperlinks? If so, could this cause you to run into testing difficulties when your wireframe is in grayscale? Dr. Susan Weinschenk: Yes. (Laughter) Dr. Susan Weinschenk: That's answered. Well, what do you think? Dr. Kath Straub: I think that that's absolutely the case, yes. The answer is yes. Dr. Susan Weinschenk: So you have to weigh what's more important. Dr. Kath Straub: Right, you have to weigh what's more important and we know that color is going to draw your attention away anyway. Dr. Susan Weinschenk: Right. Dr. Kath Straub: So, no matter what, color is going to draw the attention of the user's eyes but I think what we are trying to do by putting it in grayscale is not to underlie the questions but to actually make sure that people don't get wrapped up in the emotion and the graphics of the page rather than doing the test. Dr. Susan Weinschenk: Right, so that's interesting observation that you had to make. Dr. Susan Weinschenk: We're getting a lot of questions there. So we're going to try and go through these a little faster. Dr. Kath Straub: Okay. Well how about this one? This one is on a little bit different topic. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: What about automated testing for accessibility? How effective are current tools at assessing Section 508 compliance, any recommendations for tools? Dr. Susan Weinschenk: You know there is a study of it that we uncovered today. We talked about I think this year's "Putting Research in the Practice" class. I believe it is this year's class. A study about different methods of testing, accessibility and doing it with an accessibility expert versus the automated tools or programmers and teaching programmers to do it and what they found was that you will need, if you have an accessibility expert looking at it, you will catch 50% of more problems. So you will catch some stuff with automated tools but it's not going to be completely comprehensive. Dr. Kath Straub: So the automated tools are just looking at the production code and not at the interactions? Dr. Susan Weinschenk: Yeah, there is only so much that automated tools can do. Dr. Kath Straub: Sure. Well, here's a quick one – Can I use mouse-capture software to complement an eye tracking software? Dr. Kath Straub: Absolutely. Dr. Susan Weinschenk: Yeah. So explain what mouse-capture software means. Dr. Kath Straub: Mouse-capture software maybe something more of it – what I am doing is actually capturing how long and where you click. Where you click through the path you click of doing something and in fact, the more data streams you have in these kinds of environments, the better off you are. Now the flip side of that is that the more date you have, the more alarmed you are. Dr. Susan Weinschenk: (Laughs) Right. I realize that but you can use both together. Dr. Kath Straub: Right so what you really want to be doing is thinking about the kinds of questions you have to ask because more may not be better if additional data isn't going to answer your questions that you are really trying to ask. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: Eye tracking seems to be high-tech in the field of usability. Is this a method affordable for a small company? Is this a service or do we need to buy equipment? Dr. Susan Weinschenk: Well, I think we kind of touched on that. I mean you know, how small is small? About how much does it cost to buy eye tracking and about how much does it cost, the additional cost to rent you know, to use it in a lab like our lab in Baltimore, versus a lab that doesn't have eye tracking. Dr. Kath Straub: In fact our lab in Baltimore has eye tracking for the same cost that you would get if you had any public lab that you work with. Dr. Susan Weinschenk: So that's a really good way to go. Dr. Kath Straub: Yes. Dr. Susan Weinschenk: If you're only going to be a lot of it, then you may consider buying one of your own. Dr. Kath Straub: Right, but then you'd need to be thinking of also ramping up on your team on understanding all these additional assumptions. One of the scariest things for someone like me to watch someone to be brought into a new technology like eye tracking which is really kind of sexy, heat maps are fun. Dr. Susan Weinschenk: Right. Dr. Kath Straub: But there are a lot of assumptions in what you are doing in your data analysis and there are a lot of dangerous patterns if you don't understand ... Dr. Susan Weinschenk: So it's not just the cost of the equipment. Dr. Kath Straub: Right. Dr. Susan Weinschenk: But how much does the equipment cost, do you know? I don't know that off the top of my head. Dr. Kath Straub: Well, the equipment is coming down in price so you're talking about $50,000 to $80,000 for a reasonable piece but again you know, what you're doing is going to be very different based on the amount of data that you wanted to have so do a spend a lot time figuring out what the data is. The cost of the usability test is going to be higher because you're going to spend more time in that analysis piece. Dr. Susan Weinschenk: Right. Dr. Kath Straub: So that analysis piece can be a little overwhelming. Dr. Susan Weinschenk: Right. Dr. Kath Straub: With remote testing, do you get a good sense of the user's body language compared to the body language you may pick while sitting next to a person? Dr. Susan Weinschenk: And? Dr. Kath Straub: I would have to say no. Dr. Susan Weinschenk: So you do miss a number of nuances. Dr. Kath Straub: You know interestingly, for the people who follow the cue studies series... Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: The cue studies series looks at different agencies and different usability groups and how their reports and they have some interesting feedback on what an effective usability testing report might be for instance and the recent as yet unpublished studies that we participated in was a study about usability testing, our team did the test remote and when we turned in our testing report, the authors of the cue studies actually said had they not known that we were doing usability testing remotely, they would not have used it. Dr. Susan Weinschenk: They would have failed it. Dr. Kath Straub: Right. So this may be something that has to do with usability testing that they need to see for themselves you know, they may have to see for themselves how to pick up some of these finer details with practice in remote testing. There is a lot of information in the tone of your voice. Dr. Susan Weinschenk: Interesting. Dr. Kath Straub: Do you recommend using eye tracking and think-aloud methods together in usability testing? This is a careful question. Dr. Susan Weinschenk: Yes. Dr. Kath Straub: I mean I think we spoke about this a little earlier. Dr. Susan Weinschenk: So, if you are going to do eye tracking, should you use CTA or use RTA? Dr. Kath Straub: I can't remember what they mean anymore. Dr. Susan Weinschenk: CTA is current...or perhaps not. (Laughter) Dr. Kath Straub: We should use RTA. Dr. Susan Weinschenk: RTA. Dr. Kath Straub: Because there is a social contact for you and I having a conversation. We will look away from each other periodically and if I look away from the screen, I'm upset that I can't track it. Dr. Susan Weinschenk: So RTA with eye tracking. Dr. Kath Straub: You want to be careful about doing that. I assume that in order to do an eye tracking, it must be an in-person evaluation rather than a remote evaluation. Is that correct? Well, that's an interesting question to be asking because one of the things that we are looking at right now is piping eye tracking out to our usability testing in our other offices. So the way we have set up our Baltimore office and you may not even know about this. Dr. Susan Weinschenk: No, I was going to say that I don't know this. Dr. Kath Straub: Maybe I should tell you this off-air then. Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: The way that we are setting up our Baltimore office and this is going to mean is that you have to have your tracking in your labs in one place, is that we are going to set up so that you can have a person coming into the Baltimore office for us to do eye tracking and we are going to have agreed on a facilitator for this and so forth. Dr. Susan Weinschenk: Yes. Dr. Kath Straub: But we are going to have data then piped through some of our offices in India and have people either facilitating or watching from there at one point and the interesting thing about this is I don't even have to be next to you to be able to say something like, "You know I see that you are looking at that box in the right-hand screen." Dr. Susan Weinschenk: No. So with the eye tracking data, you don't have to be sitting there with the machine? Dr. Kath Straub: Right. So there are these technical problems of getting from one place to another but since we have solved those problems, it means that I can do much better remote testing. Dr. Susan Weinschenk: That's wonderful, that's really interesting. Definitely that's very interesting, I didn't know that. Dr. Kath Straub: Okay. What is your experience with user testing in context? In other words, you do the testing in the context that the user experiences so you can better understand the environment's relationship? Dr. Susan Weinschenk: I think what they mean is you go to a user's home and they are using the internet, you know? Dr. Kath Straub: Right. Dr. Susan Weinschenk: Or going to their office to see how they use a piece of software. Dr. Kath Straub: So this might be a bit more ethnographic. Dr. Susan Weinschenk: Yeah, right. So what do you think about that? Should testing be done, is it better to do it ethnographically when you are at the user's site or is okay to be bringing them into you know, essentially an artificial environment? Dr. Kath Straub: Right, well I think that there are benefits to both. I have some control in the lab that I may not have in the home environment. Dr. Susan Weinschenk: Yes. Dr. Kath Straub: So I can't tell you that the phone's ringing a bit loud. Dr. Susan Weinschenk: Right. Dr. Kath Straub: But it may be important for me to know that the phone is going to ring. So maybe I want to do a combination of both of these or some of the tools that we are talking about like Moray allow us to some really interesting data capture in the field. So we've done some of these tests where we go to Starbucks and we just ask someone to do a test with us. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: And then we can even go to someone's home. Dr. Susan Weinschenk: So I guess you have to decide how important is the context? Dr. Kath Straub: Right. So this is a bit like balancing the fidelity of things. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: Okay, let's see. I've done remote usability testing but I've not been able to use time metrics such as "time to complete" due to latency in the network. How do you overcome this problem? Dr. Susan Weinschenk: Yeah, do you know? Dr. Kath Straub: So far, we haven't, to be realistic. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: But in many cases, we are not looking at "time to task". Dr. Susan Weinschenk: So I think that is an important distinction. If you think you really need you know, exact time metrics, then you might not want to use remote testing. I think what we do, you know the advantage that remote testing has, is that you just kind of just let go of some of those firm time metrics and use more of the qualitative metrics. Dr. Kath Straub: I'm not so sure I'm worried about time metrics at least for the work we are doing on the web. Time is a perceptual thing, right? So if you don't think it's too long as a user, I won't even worry about it. Dr. Susan Weinschenk: How long is too long, right? Dr. Kath Straub: Right. Dr. Susan Weinschenk: But it is a problem then. Dr. Kath Straub: Right, so if I am doing a heads down data entry test it may be a problem but if you are just looking for information that is relevant to you... Dr. Susan Weinschenk: And comparative, right do these tests take longer or do these tests take longer than normal? Dr. Kath Straub: Oh, so here's a good one that we get in our classes. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: I have a few questions about testing. When you're conducting several rounds of testing and you're tracking – when do you stop? What would be considered an acceptable usability level? Is there a percentage of success that you seek, like 75% of users successfully completed a task, or is that insufficient? Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: Well, what do you think about that? Dr. Susan Weinschenk: When we do a usability testing, the kind that we usually do, we have very small sample sizes. So I think to use this "75% of the people completed", I don't know how meaningful that is and I think it is much better to, I mean I actually I tend to use scales like "all", "some" or "none" and I just don't know if it is that important to get that much fidelity on those measurements, not when you're talking about testing 6 or 7 people. Dr. Kath Straub: Here's another point and I am going to ask you a hard one. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: Can you repeat the name of the book that you talked about? Dr. Susan Weinschenk: (Laughs) That is an easy one. It's called "Strangers to ourselves", that's the main title and then underneath it is "The Adaptive Unconscious" and now you're going to ask me who the author is. Dr. Kath Straub: I am. Dr. Susan Weinschenk: Oh, no. I'll see if I can remember it. Actually I was going to bring it. Dr. Kath Straub: We can probably post that on the website. Dr. Susan Weinschenk: Yeah, we will put that on the website, how's that? Dr. Kath Straub: Well, one of the other books that people have read recently quite a bit is "Blink", right? Dr. Susan Weinschenk: Yes, and "Blink" actually, the book "Blink" led me to the book "Strangers to Ourselves" because in the book "Blink", this book "Strangers to Ourselves" is a reference. Dr. Kath Straub: I'm only mentioned that because I can remember that Malcolm Gladwell wrote that one. Dr. Susan Weinschenk: Thank you very much. Dr. Kath Straub: What – this is the hard one now. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: What criteria do you use to determine what level of prototype you need to use? Dr. Susan Weinschenk: Oh, you mean those five we have? Dr. Kath Straub: Right. Dr. Susan Weinschenk: I don't remember that we've worked enough with this to say. I mean I think, that's actually a good question and we want work on that more and maybe we come back and do that on another one of our webcasts because I haven't played around yet with those five different enough. That's new to me. Dr. Kath Straub: Okay. Do you think that one conclusion from reading these articles as a group is that relying on one technique to determine how usable your design is would be less valuable than making sure that you use several techniques? Dr. Susan Weinschenk: What do you, what do you think they mean by techniques? Dr. Kath Straub: I'm not sure. Dr. Susan Weinschenk: You mean like remote versus not remote and CTA and RTA? Dr. Kath Straub: Or perhaps remote or usability tests versus analytics plus ... Dr. Susan Weinschenk: Oh. Dr. Kath Straub: The different kinds of methodologies. Dr. Susan Weinschenk: Well, it would be wonderful to do multiple techniques. Ii guess it's a question of how much time do you have and how much budget do you have? But we do sometimes combine them. We've done you know, automated testing as well as smaller usability testing. I think we tend to do you know, one technique. We pick you know, what's the best technique given the time, budget and situation? Dr. Kath Straub: But once in a while we do these techniques where we test a small number of people with more fidelity and more intensity and cast a wider net with a survey or some other low testing or automated testing. Dr. Susan Weinschenk: Automated testing, yeah. Dr. Kath Straub: Or surveys. Dr. Susan Weinschenk: Automated testing. Dr. Kath Straub: Do you tell people when you're doing eye tracking, and if so does that skew the data? That's a good question. Dr. Susan Weinschenk: Yeah, do you? Dr. Kath Straub: Well, actually it's the same idea as do you tell people when you are recording. Dr. Susan Weinschenk: Do you tell people that there are people on the other side of the mirror? Dr. Kath Straub: Do you tell people those people are listening? Right and we do, we always do. We don't observers to be there and participants not to know and we actually even know that the new eye trackers don't require any of the old stuff. Do you know any of the old eye trackers? Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: We used to bite bars and all that. Dr. Susan Weinschenk: Right. Dr. Kath Straub: Now you can just sit down in front of the screen. Dr. Susan Weinschenk: Right. Dr. Kath Straub: And there's a little dot that's actually tracking your eyes and you don't even know or you might not even know that we're observing but we will tell people. Dr. Susan Weinschenk: Yeah. Do you think it affects the data? Dr. Kath Straub: You know what? I think that usability testing is an interesting thing, people really get into it. I mean the idea is you're paid good money to break somebody's website. Dr. Susan Weinschenk: Right. Dr. Kath Straub: It's kind of fun. So I think they forget actually. Often, they forget we're there even looking over their shoulder because they're enthused about the fact that somebody is listening to them. Dr. Susan Weinschenk: Right. Dr. Kath Straub: And so, I don't think that that necessarily affects the data. Dr. Susan Weinschenk: You don't think it affects that much. Dr. Kath Straub: No. Suppose you tested multiple participants using an eye tracking method. Couldn't you view all the results and determine how the task questions affected the response (i.e. if the question was clear or not clear) and if the expected path was met by a majority of participants? Yes but that would sound like master pieces. Dr. Susan Weinschenk: Yeah, I was going to say it sounds like fun but it might not the answer to a particular question you have. Dr. Kath Straub: So this is the kind of question that I would ask and you would sort of tell me to go back to my work. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: (Laughs) Pretty much. Dr. Susan Weinschenk: So if anyone would like to fund Kath to do this study, you know they write a check, right? Dr. Kath Straub: Right, exactly. Dr. Susan Weinschenk: And send it to HFI. (Laughter) Dr. Kath Straub: This is a very good question. But I think what you're asking here then is was the question you asked, the right question? Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: And can we do that is a slightly less high-tech way by pre-testing our usability testing? Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: Questions? Dr. Susan Weinschenk: So we don't need to fund your research. We'll just do a pilot. Dr. Kath Straub: Well, let me retract that then. (Laughter) Dr. Kath Straub: How do you prevent data overload when using eye tracking? With eye tracking videos and oodles of logging notes, where does one begin? That's one of the reasons why you would want to let somebody who has done it before at least help you. Dr. Susan Weinschenk: Where would you begin? You've got your usual kinds of data that you would get and now you have all the eye tracking data, what would you tell your team when they get down to analysis, what would you tell them to do first? Dr. Kath Straub: Well, this is a trick question, right, because it depends on the question that we are asking. Dr. Susan Weinschenk: Well, give us an example. Dr. Kath Straub: Well, I think we might want to start looking at for instance where people look first again it depends on the question that I am asking. If I am looking at a high fidelity prototype that has graphics on it and I'm working on the "what do you see?" question. Dr. Susan Weinschenk: Okay, what you see. Dr. Kath Straub: But then I might want to look at the scan path that are timed and see what do people eyes go to first? Where do they see and long are they there? Dr. Susan Weinschenk: So really what you're saying is make a list of the important questions, I mean you should have that before you start the test. Dr. Kath Straub: Right. Dr. Susan Weinschenk: And then go back to that list and then say, okay, the important question number one, where are people looking? And then decide which data to attack first. Dr. Kath Straub: Right. Dr. Susan Weinschenk: Then go to the next question and do it that way. Dr. Kath Straub: Right. In fact, when you are going to do anything as data intensive as eye tracking and I would have to say when you do any kind of data analysis or data collection, anyway, you should have your data analysis path using your strategy for analyzing your data. Dr. Susan Weinschenk: Then you're informed when you start. Dr. Kath Straub: Because otherwise, you're essentially on a fishing expedition. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: And if you don't know what to do with your data, then you have to be worried. Dr. Susan Weinschenk: You get lost. Dr. Kath Straub: Right, exactly. Great, let's see. Have you had experience testing software applications with a tracker? I'm interested in whether it can be used to determine how intuitive an application is. Dr. Susan Weinschenk: Like ok a web page? Dr. Kath Straub: Exactly. Sure, yes and it's again the question you're asking the data to look for and so it can be useful in understanding how people move through an interface that are at a more detailed level and that's what we're really getting out it. The eye tracking data is more than the talk-aloud, more than the think-aloud, we're getting more brand new data of where I look and how long I look and we can go back retrospectively. Dr. Susan Weinschenk: So there's no reason why we can't do it with a software application. Dr. Kath Straub: Here's one. I've dabbled in a remote usability test for a company intranet as you mentioned. The two things I've noticed are that we've seen outrageous "no-show" rate and that we have much less control over the testing environment. One participant took his cell phone call during the test and another sat down in a noisy server room while his buddies kept passing through and joking that he was failing the test. Dr. Susan Weinschenk: (Laughs) Dr. Kath Straub: Any insights on how to control this? (Laughter) Dr. Susan Weinschenk: Yeah. So that, so you, that's interesting, so what are you going to do with that? Dr. Kath Straub: Well, we are actually fairly, I don't want to say "rigid" but fairly careful in what we do in this respect of recruiting and conveying our expectations for the test well ahead of time. Dr. Susan Weinschenk: Ahead of time, right. Dr. Kath Straub: Right so we have voice contact and we have explanations about what we are doing and why this is important and we will ask people to turn off their cells or turn off their phones. Dr. Susan Weinschenk: So you're here while the test is going on and you can talk to them. Dr. Kath Straub: Right. Dr. Susan Weinschenk: So, but part of it then would be to make sure that whoever you were working with that was helping you set up the test would make, you know help you get a good environment and have a quiet room and that someone is talking to the participants ahead of time and letting them know you know what's acceptable and what's not "not acceptable" in a strict way, we're just saying "Here's what the test is going to be like, here's what we need you to do you know, please turn off your cell phones, etc." Dr. Kath Straub: Well here I have a quick question for you then. Dr. Susan Weinschenk: What? Dr. Kath Straub: When do you throw out data? Dr. Susan Weinschenk: Oh, that is always a great question. When do you throw out data in a test? I think you should probably think you should talk about deciding the time and then you should try and decide ahead of time when you're going to throw out data and when you're going to keep it in. I have a tendency to leave the data in and then, but just make notes in my report that you know, we have to realize that you know, the following happened during the test but I tend to leave the data in. Dr. Kath Straub: We never do the same thing and we're both from a science background right? Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: Okay, here's one for you. We have a site that's been in existence for two years and never been tested. Is it worth testing now? Dr. Susan Weinschenk: (Laughs) Yeah, you better hurry up. (Laughter) Dr. Susan Weinschenk: Why wouldn't you test? Dr. Kath Straub: How would you approach a new team for a usability test for an existing site versus a new site in the design phase? Dr. Susan Weinschenk: Oh, we test existing sites a lot. Dr. Kath Straub: I think people have this sense that usability testing has to be very formal. Dr. Susan Weinschenk: Yeah, well you could do a formal test on an existing site too. Dr. Kath Straub: But it doesn't have to be? Dr. Susan Weinschenk: No, absolutely not but yeah, what's the benefit of doing it for an existing site? Well, first of all, if you don't, I mean if you have a site that's two years old and you don't want to make any changes and you are not at all interested in it then don't test it. Dr. Kath Straub: But I think that they are suggesting that they are in the design phase so they know that they don't like their current site ... Dr. Susan Weinschenk: Oh, I missed that part. Dr. Kath Straub: And you're designing a new one, is there any value in knowing how the old one works? Dr. Susan Weinschenk: Oh, they're designing a new one now and they are testing the old one. I would say only if you don't have any data, so if you're going to re-design because you know, you're switching platforms or someone said, "We should re-design for you know, a new look and feel" and you don't have any data on the design and on the usability ... Dr. Kath Straub: Of the existing design. Dr. Susan Weinschenk: Of the existing design, I think you should but you might have lots of data ready, not from the usability test. Dr. Kath Straub: I see. Dr. Susan Weinschenk: You might have comments and feedback from people. Dr. Kath Straub: Well, I think you might also want to test it to get a baseline to get a value of what you are doing. Dr. Susan Weinschenk: Right, to get a value, right. Dr. Kath Straub: So, it's a way to show that... Dr. Susan Weinschenk: It's not necessary that we do that lot. Dr. Kath Straub: When performing automated testing, this one must be for you then... Dr. Susan Weinschenk: Uh-oh. Dr. Kath Straub: How do you account for possible usability issues with the automated testing tools versus usability issues with the product or prototype? Dr. Susan Weinschenk: No, there are never any issues with usability with the work we do, right? Dr. Kath Straub: Right. Dr. Susan Weinschenk: So first of all, the tool you pick is important and then secondly, you have to pilot. You absolutely have to pilot and then you have to figure out I mean, we often change the protocol and the test questions to find a way to work around it but you have to allow for that and you have to know how much that might affect and you could do that with a pilot then you just get to it. Dr. Kath Straub: Here's an interesting methodological question. Dr. Susan Weinschenk: Methodological question, okay. Dr. Kath Straub: The focus here has been on testing software, specifically web interfaces. What for you see as the future of usability testing for consumer products, medical devices and things like that? Dr. Susan Weinschenk: Well, I don't, you know certainly eye tracking is focusing on web sites and on software but a lot of the other things that we are talking about did not necessarily apply to that you know? I mean you still have the CTA, Current Think-Aloud versus Retrospective Think-Aloud if you were testing a remote control or a medical device. So the future, if we just focus for a second there Kath, on the future of usability testing for devices, what would you say? The future is issues are there. Dr. Kath Straub: I don't know, I'm going to sound a bit of a boring person here but there is a way of using tracking to show that people are doing things with things other than screens. Dr. Susan Weinschenk: There is? See? I'm glad I brought her on today so I can say that I know about eye tracking. So how can we do that? So we have a remote control or a cell phone, we have a cell phone on the table, how can you see where I am looking on the cell phone? Dr. Kath Straub: Well, there is a little bit more technology involved and there is ... Dr. Susan Weinschenk: But it is possible? Dr. Kath Straub: It is possible, sure. Dr. Susan Weinschenk: That's great. I think tracking; I think we are all going to be a lot more eye tracking, don't you? I just think it is fascinating. Dr. Kath Straub: Do you know of any data comparing the same visual design on paper versus an interactive prototype? In particular, I'm interested in wizards. Dr. Susan Weinschenk: Research on that, huh? I don't, there was a, I'm going to say that I read some research on that and it was a long, long time ago I don't know have any recent data, do you? Dr. Kath Straub: I don't but I think that one of the challenges is that our tracking is relatively new technique for practical usability. Dr. Susan Weinschenk: Yeah. Dr. Kath Straub: That we are not going to have access to these studies that practitioners really want. Dr. Susan Weinschenk: Yeah. I don't have any data on that. Dr. Kath Straub: So we've done a lot of usability testing to look at things like language comprehension and I know this may not be important as to what it used to be in usability but they will come. Dr. Susan Weinschenk: Right. Dr. Kath Straub: Users can give a lot of feedback on the products that they are testing. Is it difficult to consider all their recommended changes and make a new product? If so, how do we make a decision about what is good feedback and what is useless feedback? Dr. Susan Weinschenk: Yeah, I mean you get a lot of feedback when you are doing a usability test. I think it goes back to deciding what the questions are that you need to get answered by a particular test and so when we are analyzing data, we tend to have a part of the report, the final report that addresses the particular questions and then there was other feedback that was interesting and important, maybe shouldn't form the design but it's not what we're looking for so that goes into a separate section. I think some of the most powerful data you can get from usability testing is often the other stuff that you didn't even think, I mean you're tried it right? Anytime you do research of any kind, with usability testing or academic research, usually the most interesting findings are things you were not even asking. Dr. Kath Straub: Right. Dr. Susan Weinschenk: And then that leads you to you know, a re-design, a next test or more research. Dr. Kath Straub: Well, I think that is relating to this next question. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: Please explain why you think that the user who is not an "expert" in the software area that you have developed does not have useful feedback for usability testing? Dr. Susan Weinschenk: Wait, why do you think, if the user isn't an expert, would they have useful feedback? Well, that, well, doesn't that go back to the white paper? This is all about making, I mean you have to sure you know who you want to test, right? Dr. Kath Straub: Well, I think that also they are asking that if you want to test people who would give respect, design recommendations or representative users. Dr. Susan Weinschenk: Yes. Dr. Kath Straub: Right, so you're right, it goes back to the white paper. Dr. Susan Weinschenk: Well, yeah, I mean usually when we are doing testing, we are testing representative users and we actually want to get data from them but we don't want them to do the re-design. Dr. Kath Straub: Right, so we are looking at their behavior rather than... Dr. Susan Weinschenk: Yeah, you've got be very careful in doing usability testing. We say we want feedback from users, we say we want users involved but the reality of it is that if the user is using a piece of software and says, "Oh, you shouldn't do that, you should put this over here" it may or may not be true. You know, users are not interface designers. We need listen to their feedback and then interpret their feedback but then we don't just go design. I mean they are not interface designers so we need to take their comments into account but we don't necessarily do what they say. Dr. Kath Straub: So I think that's the value of having usability facilitators knowing which comments to take. Dr. Susan Weinschenk: Very important. Dr. Kath Straub: And knowing what to ignore. Here's one that you actually asked earlier and I am going to ask you to answer your own question. Dr. Susan Weinschenk: (Laughs) Oh no! That's not fair. Dr. Kath Straub: What are your thoughts regarding the facilitator being in the room with the participant rather than being in an observation room? Dr. Susan Weinschenk: Yeah, this is very interesting you know, there is no agreement on this in the field. Dr. Kath Straub: Not so much. Dr. Susan Weinschenk: Not so much among the usability people because first of all, there is a really practical answer when you're doing an informal test and you don't have testing facility and you don't have an observation room, you're in the room with them and that's fine. You know, I am actually kind of a stickler for the protocol and when I did, I don't do a lot of usability testing myself any more but when I did, I always did when we were not in the room. Dr. Kath Straub: Really? Dr. Susan Weinschenk: What? Dr. Kath Straub: Really? Dr. Susan Weinschenk: Yeah. I don't know that it makes a big difference as long as you have a very well trained facilitator – now that's the issue because obviously there are lots of things that the facilitator can do to bias the results. So they have to be extremely well-trained. Dr. Kath Straub: But this is one the hardest parts right, learning how to be neutral? Dr. Susan Weinschenk: Absolutely, it's amazing you know, you can't nod your head in a certain way, you can't ... I mean some of fun have a lot of fun mentoring people in the usability testing and we do a test, we set up a test, we are not actually collecting data, it is just that they can learn and you know, the tapping of the foot, or the user does something and they instantly start writing whatever on the pad and there are all kinds of things that you do that bias the test or throw it off or make the participant uncomfortable so you have to be trained. Dr. Kath Straub: So here's another of those methodological questions. When you give a series of tasks to perform on a usability test, do you have the participants do them in the same order? Why or why not? Dr. Susan Weinschenk: Well, it depends. It depends on the test, right and what you are trying to find out. You want to talk on that. Dr. Kath Straub: Well, I think this is an interesting question because it has to do with how good you are with facilitating your tests too so often times it would be a case where participants are being tested, facilitators feel that they need to do things in the same order every single time or they won't have a valid test and I think that's the difference between the academic research and the pragmatic research that we really want to do. Dr. Susan Weinschenk: Right. So if it's academic... Dr. Kath Straub: Then it would have to be in the same order. Dr. Susan Weinschenk: In the same order because if they did one thing first and then another thing... Dr. Kath Straub: That's a tough one. You have to do it in the same order but you have to vary the order so you have different conditions. Dr. Susan Weinschenk: What do you think about the practical usability role? Dr. Kath Straub: The practical usability role that we tend to do and again, this speaks to the facilitator who knows what they are doing and is comfortable in this environment is to know your script enough to be able to move the person through all of it and the tasks that you need and one of the methodologies that we didn't talk about is the kind of methodology where we start someone out perhaps on the task but actually watch them and see what they would do on their own at the site. So giving them some intuition to what's there but then taking them... Dr. Susan Weinschenk: Letting them go. Dr. Kath Straub: Letting them do what they really want to do. Dr. Susan Weinschenk: But I think the important point too is that usability testing in a way that we talk about practitioners doing it is not really what you and I would call research. Dr. Kath Straub: No, it's not but it's a different goal too. We are not really understanding a "how come" issue... Dr. Susan Weinschenk: It's kind of a fine line there because we actually are sticklers for protocol and about facilitators and yet we are not sticklers about other things, you know research role would be, well, critical. Dr. Kath Straub: Right. We're not writing for publication to defend. If we will, we will change our appointments. Okay, is there any form or template available to collect usability test results online? Dr. Susan Weinschenk: There is, I mean there are all kinds of things. There is – we have our own tools that we use, typically spreadsheets and some of the software tools that are there for doing the log in and log out, what are some the tools that you would advise to someone who did not have a particular form or a template like we are using? Dr. Kath Straub: It's usually the way I work on my own but I tend to have a format that I use that is an Excel spreadsheet. Dr. Susan Weinschenk: Yeah. We use it a lot. It's very easy. Dr. Kath Straub: And it is set up to collect the data I want in the way I want it and it allows me to my analysis in a very quick way but, I mean the important thing is it is set up ahead of the test. Dr. Susan Weinschenk: Right. You know if the time matches the scenarios so you know what data you get. Dr. Kath Straub: And again, that's going to require me to be a bit more able to juggle if I am going to let the user move through the scenarios in a different order, where to look etc. Dr. Susan Weinschenk: And by the way, we talk about this when we show this is in our usability testing class and we show a sample spreadsheet that is set up that we make available. Dr. Kath Straub: Well, this also speaks to the question that we often get too about whether you should have one usability tester or two usability testers doing the test? Dr. Susan Weinschenk: Yes, and? Dr. Kath Straub: Typically, what we would do is have two because that allows them one person who is facilitating side-by-side to keep facilitating. Dr. Susan Weinschenk: To keep facilitating, yes. It's very hard to facilitate and be typing away on the spreadsheet. Dr. Kath Straub: And it's distracting too for the participant. Dr. Susan Weinschenk: Right. Dr. Kath Straub: And we don't want to have that kind of problem. Let's see. When using RTA, do you walk the test subject back through their process and have them explain their experience, or do they do it away from the computer? Dr. Susan Weinschenk: Yeah, and? Dr. Kath Straub: A little of both, I think it depends on the situation we are in. We definitely want to let them be able to point to things. Dr. Susan Weinschenk: Right. Dr. Kath Straub: I'd be worried about asking them to reconstruct from memory. Dr. Susan Weinschenk: Right. Saying, "let's go and do it again" and then they are doing it again. Dr. Kath Straub: Right. Dr. Susan Weinschenk: And they actually what you are getting is a current think-aloud. Dr. Kath Straub: Right. Dr. Susan Weinschenk: And the second time through. So they might be able to point to something but they won't actually be able to walk them through the whole process again. Dr. Kath Straub: Let's see. Here's another question that goes back to the same thing. People argue that what their learning, what this question calls a "learning lab" don't rely on pre-defined tasks as a better approach because it most closely mimics the user experience. Dr. Susan Weinschenk: Yes and no, right? I think their idea that "here's a website, go use it" but that's very different, I mean most of the time as you said, people go, they are using the website, they are using software and they have a task in mind. Dr. Kath Straub: Right. Dr. Susan Weinschenk: And in a lab, I mean it's a lab, it's still artificial. You say, "Here's the website, go at it" but you know they're sitting there, "Oh, what am I supposed to do, what is my task?" Dr. Kath Straub: Right. Dr. Susan Weinschenk: So I think you have to ask again, what are you trying to find out? If indeed the question is what people are, where do people go in the site if they didn't have a thing in mind, right? Dr. Kath Straub: Absolutely. Dr. Susan Weinschenk: And let them at it. I think you need to think about what the question is. Dr. Kath Straub: Right. In cases where we are asking them that in cases where we are expected to do that, we ask the participant in recruiting if he has gone to a similar site or looked for similar information on the web. Dr. Susan Weinschenk: Right. Dr. Kath Straub: So now we have at least established that they are interested in what we're talking, they're not just seeing the site for the first time. Dr. Susan Weinschenk: Right. Dr. Kath Straub: Do all commercially available eye tracking devices provide similar charts or output? That's an easy one, the answer is no. You have to pay attention just like any other software, pay attention to the interface and the way that the eye tracking device gives you the data may be very different depending on the tracker that you are looking at. Dr. Susan Weinschenk: Okay. Dr. Kath Straub: So, it's not a plug-and-play purchase. Dr. Susan Weinschenk: Oh, okay. Dr. Kath Straub: And again, that's why you may not to just run out, let's see. Regarding the heat map, how can you tell where a person clicked? This is a good question because it speaks to the multiplicity of ways to look at the data. The heat maps that we showed had one instance of a click, it was ‘x' if you remember? Dr. Susan Weinschenk: Right. Dr. Kath Straub: And there are just a lot of ways to map the data that you get. So you have to decide, again, on what questions you are going to ask and then you have to decide how you are going to portray the data to feed into answering that question. Dr. Susan Weinschenk: So, do you mean, you tell the software "I want to see on the heat map where people clicked" and you just say that "I want to see it as a red ‘x'?" Dr. Kath Straub: I don't know if you can really ... Dr. Susan Weinschenk: That's just an example. Dr. Kath Straub: Right. Dr. Susan Weinschenk: But you can say I would like to see it. Dr. Kath Straub: Right, you can shape your data so you're actually getting some pretty good data back. Dr. Susan Weinschenk: You're choosing that data. Dr. Kath Straub: Okay, let's see. Do you create a new heat map in eye tracking for each task you request the user to do, or is it cumulative over the test? Dr. Susan Weinschenk: I mean so you give him one page, right? Dr. Kath Straub: Right. Dr. Susan Weinschenk: And you say, "Where would you go to do this or that?" Dr. Kath Straub: Well, let's take a step back. You don't give him one page, you give the entire site. Dr. Susan Weinschenk: Right. Dr. Kath Straub: So it's not just, "What do you look at on this page?" but we can follow you through the entire site. Dr. Susan Weinschenk: Right and then you'd have a heat map. Dr. Kath Straub: Yeah. Dr. Susan Weinschenk: But then, you give him another task to do, what are you going to do? Dr. Kath Straub: I'm not sure that I would be very interested in seeing what people looked the course over all of these. Dr. Susan Weinschenk: You'd really want it by test, not as a heat map. Dr. Kath Straub: It's not got to do with it much and again, you know, you may have a reason for doing that we are not thinking of it here... Dr. Susan Weinschenk: Right. Dr. Kath Straub: But we can't think of it. (Laughter) Dr. Kath Straub: When stressing the immediate information in the results section about retroactive think-aloud, there was an acronym AOI... Dr. Susan Weinschenk: I should have known that someone was going to ask. You asked me this question. Dr. Kath Straub: I did ask you this question and I thought this was going to be a softball. Dr. Susan Weinschenk: And I'm going to have to be just brutally honest and say, I'm not sure. It had to do with the scan path and the heat map. I'm sorry, I put the AOI there and I don't remember so I'm going to have to look that up and maybe we can mail, we can see a way to answer that on the website. Dr. Kath Straub: Okay, here is the last question actually. Will your slides be available for download? And the answer is, the slides themselves, no but the webcast itself, yes. Dr. Susan Weinschenk: Right. Dr. Kath Straub: The slides in the context of the webcast. The white paper's appendix lists the research papers that we referred to and has a link to a newsletter discussing these topics. Dr. Susan Weinschenk: And I think that, did I mention that maybe my book, that book is in there too. Dr. Kath Straub: And the material is definitely covered in depth in the PRP papers. Dr. Susan Weinschenk: Well let's mention that. The PRP is "Putting research into Practice", this material is covered in it is actually covered in next year's version. Dr. Kath Straub: Oh so this is ... Dr. Susan Weinschenk: This was a teaser. Plus you know Kath Straub is a scientist who has done how many annual reviews? Dr. Kath Straub: Too many. Dr. Susan Weinschenk: Too many. It must have been at least 1000 I think. Dr. Kath Straub: Something like that. Dr. Susan Weinschenk: Right, for "Putting Research into Practice" then I go through them and pick out what I want in the course and these studies, they were just so interesting and fun and I couldn't resist, so it's a sneak peek. So they will be in the 2007 version of the "Putting Research into Practice". Dr. Kath Straub: But this is just a small part of it. Dr. Susan Weinschenk: Oh, this is a tiny bit though that we are dealing with here. So I want to thank everyone for joining us. We've reminded you here about the white paper and – about downloading the white paper and also view the webcast. I hope you enjoyed it and I hope you will join us for our next one. |
| © 1996-2012 Human Factors International, Inc. All rights reserved | Privacy Policy | Follow us: |
|
