What will we test?
We have 3 core functionalities in our application: mining and rating articles, adding friends and getting statistics of a user compared to his friends. This is why we made 3 special tasks to test the usability of these features.
We also wonder if the detailed statistics can be found by the users. The screen is not accessible in a single click, so we added a fourth task to test if our users can find it.
In addition to these detailed information, we also want to know the overall feeling our users have of the application.
How will we test it?
The users will perform these tasks in the order below. They get the next task as soon as the previous one is finished.
We will measure both the time it takes to perform a task, as the amount of clicks needed to perform it.
- add a friend
- compare your statistics to those of your friend "Drill Bill"
- mine and rate an article
- find the detailed statistics of the category "sport".
We have chosen to use the CSUQ test. This seems the best one to represent what we want to know. Other tests are more focussed on the job of a person or the errors the application gives. We also don't want a too short question list, as that doesn't provide enough information. The 19 questions of CSUQ seemed like the best way to go.
Who will we test on?
As our application is designed for adolescents, we will try to test this on adolescents. This means people between 14 and 22 years old, both boys and girls. It is best if these people don't know too much of informatics. Everyone in our group will test 2 people, so we have 8 testpersons in total. For the ease of finding people, we will ask people in our direct surroudings during the weekend. In Leuven we don't have access to many people in our testing range.
Good description, although I miss what could go wrong during your test due to external factors like technical problems, ...
ReplyDeleteYou might want to rethink the name of your imaginary friend (consult http://www.imdb.com/title/tt0856192/ for an explanation on why).
ReplyDeleteBack on topic:
I have a few questions regarding your choice of a questionnare. A lot of question have to do with "work" and "productivity" how will you translate this to a game-context?
I do not really see how users can be "productive" when playing a mining game.
""It is best if these people don't know too much of informatics.""
ReplyDeleteAre people with little computer knowledge part of your aimed at user base, or do you choose these people to see if even those can use your application?
If they are (part of) your user base, what about other people (same age range) but with more computer knowledge?
"We also wonder if the detailed statistics can be found by the users."
ReplyDeleteIt sounds like you hit it somewhere. Couldn't it be placed on a more obvious place then, or am I just wrong and is it not hidden that far?
I also think the skills of your test persons aren't that important. Both people with only basic skills and people with advanced skills will be able to use this application. Maybe you say this because you already tested CS students in your previous iterations?
This test seems way to wide in scope to me: from the four tasks that you will ask your users to carry out, it seems to me that only number 3 ("mine and rate an article") is really related to the core scenario of use?
ReplyDeleteI think there are still some doubts about this basic concept, so I would really focus on that.
Moreover, time and number of clicks may be easy to measure, but will they really help you to figure out whether people understand the concept and are interested to apply it?
I think it might be usefull for your application to keep an eye on the number of wrong clicks. When doing your paper prototype, is was hard to see what was clickable and what wasn't. If this also occurs in the digital version, you'll have a problem.
ReplyDeleteMaybe we should throw the concept away and start over yet again. Because the only one still believing in this application is my little brother (Jan Gerard, 26/12/1996, no computer skills besides facebook and gaming) who really want to use it in practice.
ReplyDeleteIs there somebody out there who magically has an idea that will be accepted by everyone on first glance and could borrow it to us?
I don't think you should throw away your concept as long you still have one believer :). Maybe more could arise during the next testing process...
ReplyDeleteLike erikduval, I also think you should mainly test and evaluate number 3 of your presented tasks. If you evaluate this, you could get to know if the basic concept (if number 3 is the basic concept) of your application is attractive to the adolescents and thus get to know if you should keep your concept or throw it away and find a new one.
I would choose for a short test, focusing on the basic concept and a lot of test persons, not only from your direct surroundings, to get a more general idea about the attractiveness of your application to other people.
The problem with testing the "main" feature of the application is that it is supposed to be highly interactive. We tried to use a paper prototype for this in the first iteration, but because a human computer can't nearly reach the kind of fluency and interactivity that we want to achieve, we pretty much had to leave out the mining from our paper prototypes. What we want to test most for now are the fluency of navigation between the different sections of our application.
ReplyDeleteAs for the target groups, the most important aspect is that we'll be testing with younger people, since they form the our target audience (which does not mean that no one else could use it, of course). The fact that we'll be testing it with people without much CS-background is indeed mostly because we have already performed numerous tests with CS people, and the application should be useable by anyone.
I don't agree with you on the last comment. Yes, a human computer isn't fluent, but you can at least tell the test participant that everything will be a bit slower. Almost everything can be tested with a paper prototype, even hovering. We also thought it would be hard to do things like that, but it is possible, with patience.
ReplyDeleteFurthermore I don't think you should throw away your idea, yet again. Your idea hasn't changed that much from the last time you "reinvented it". It merely became more attractive to more people. Accept the fact that some people don't like some things about your application and just adapt. This doesn't mean it isn't right sometimes to drop your idea. But I'm sceptical about doing that again and again...
You could clarify CSUQ by telling people that, since amusement/information is the end goal instead of production, they should read 'complete my work' as 'read news' or 'amuse myself' and 'productive' as 'informed' or 'amused'. Otherwise people would see these questions as irrelevant, as chileaders also implicated.
ReplyDeleteOn your idea, I would suggest improving on it but not throwing the baby out with the bathwater. It's a very refreshing concept that might or might not work, but it's hard to know beforehand unless your concept is a really bad one, which is not the case.
I don't think you will need to drop your idea. I like the main concept but when I saw your third paper prototype last time, I wasn't sure if these statistics would be used so often. I understand you want to have a task comparing statistics with your friends, since you would like to know whether you have the same interests as your friends. Although it seems weird to me that you want to have a task to find your own statistics of a specific category. It seems like this task isn't used very often and isn't very important, especially because users won't be able to reach it in a single click, so I think it shouldn't be a main focus for testing.
ReplyDeleteTeam SachiniNews
You could make a pretty big map and say to the user that he can go anywhere on the map, but he needs to say where he want to go and which buttons he would click to get there. (click on a rock with your mouse, run against it or stand next to it and press space)
ReplyDelete