Further testing

For the next evaluation of our application, we will use the paper prototype for the last time. We will now describe how this testing will happen.


What will we test?
We have 3 core functionalities in our application: mining and rating articles, adding friends and getting statistics of a user compared to his friends. This is why we made 3 special tasks to test the usability of these features.
We also wonder if the detailed statistics can be found by the users. The screen is not accessible in a single click, so we added a fourth task to test if our users can find it.
In addition to these detailed information, we also want to know the overall feeling our users have of the application.

How will we test it?
The users will perform these tasks in the order below. They get the next task as soon as the previous one is finished.
We will measure both the time it takes to perform a task, as the amount of clicks needed to perform it.

1. add a friend
2. compare your statistics to those of your friend "Drill Bill"
3. mine and rate an article
4. find the detailed statistics of the category "sport".

Once the users are finished with the prototype, we will ask them to fill in an additional questionnaire. This should help us measuring the overall usability of the application and the feeling users have of the application. We want to keep the amount of questions lower than 20. Otherwise our users would get bored with the questions and might give the application a worse rating, just because they are getting bored.
We have chosen to use the CSUQ test. This seems the best one to represent what we want to know. Other tests are more focussed on the job of a person or the errors the application gives. We also don't want a too short question list, as that doesn't provide enough information. The 19 questions of CSUQ seemed like the best way to go.

Who will we test on?
As our application is designed for adolescents, we will try to test this on adolescents. This means people between 14 and 22 years old, both boys and girls. It is best if these people don't know too much of informatics. Everyone in our group will test 2 people, so we have 8 testpersons in total. For the ease of finding people, we will ask people in our direct surroudings during the weekend. In Leuven we don't have access to many people in our testing range.

Report 1

The following link contains our first report for the course of human-computer interaction. In this report we explain what we did during the first step of our project. We go into detail about our concept with the aid of a storyboard and a screen-transition diagram. Then we continue with explaining our paper prototype and what conclusions we took from these first tests of our application. Finally we also make some conclusions about this iteration and plan on what we will do in the next iteration.

We wrote the report in English, so our non-dutch-speaking readers can also read the report.

http://www.mediafire.com/?cb3824a2ffh1bww/

Paper prototype - 3rd version

We tested this third prototype with housemates who are not computer scientists. This way we could get some feedback from people who have a different view on computer programs.
The feedback we got from this was generally positive. Although some testers still had their doubts about the concept, the interface seemed to be doing its job very nicely. These are the relevant remarks:

• The text in the indroduction screen is not clear enough.


• The button to indicate which friends are taken into account for the value of articles - a green/red “stoplight” - is confusing. People don’t know what it’s for until it has been explained explicitely.


• The button to rate an article neutral was unclear.


• Using the terms Like/Dislike for rating an article causes wrong usage: interesting articles about bad events were rated negative although for our system they should be rated positive.


• When looking at the statistics, some people wanted more specific information, such as a further refinement of the categories.


• One of the test persons thought the button to close a window would close the entire application, and didn’t manage to return to the mining game.


• We had a facebook like-button to the starting screen of the application. However, this was only reachable when starting up the application while it should be accessible anywhere.

Because there were still quite a number of (small) remarks, we are planning to do one more paper prototype iteration. We will make the following changes:

• The “stoplights” in the friends list will be replaced with a checkbox. The color of the box of the friend will also help indicate whether or not they are included in the calculations.


• The rating buttons will have a visible name in addition to the icons, clarifying the purpose.


• The introduction message will be slightly more elaborate to clearly specify the goal of the game.


• It will be possible to view the statistics of categories in general, as well as more specific stats within a category.


• The facebook like-button will be visible everywhere, although we haven’t figured out exactly where on the screen it will be.

We decided not to change the close button, since we believe that when the application appears in the context of a webbrowser it will be clear that it doesn’t close the entire application.

Paper prototype

Since we had to re-do a lot of the work of the previous sessions, we are running a little behind on schedule. We started with paper-prototype tests of our application on Wednesday. We started with a prototype to test on our own. Not all popups and mouse-over effects were there, as it was just to see if we didn't forget some major functions. In the picture below, you can see how the prototype looked like. We already kept in mind some of the things we learned from the paper prototype tests of our previous concept. Users want a close-button for the menu items, and don't want to click the menu on the left. They are lost if we just give them a menu, they want close buttons. We decided having both the menus and close buttons available would be best.

Once we had a good feeling about it, we made a second prototype. This time we tested it on people sitting in the hall of the computer-science department. We noticed people didn't know where the mining button was for. They kept clicking on it, while they were already on that screen. This button remained there from the time we didn't have the close buttons. In the next prototype we will remove this button.

The users also wanted to be able to select which friends are being used to determine what articles they are mining. To achieve this, we will add an extra button in the friends list. By clicking this, a friend can be enabled or disabled.

We are now testing our new version of the paper prototype with new users and hope the interface will be better with our new improvements.

Working out our new concept

We decided to work out the idea for mining and rating news. We will first explain how this new concept will work, and then explain what we think the benefits are compared to our old concept. However, if users prefer the old concept we will go back to the previous one.

A user will have the possibility to mine freely in the underground. In this underground, he will find different minerals that represent articles. When his friends haven't rated the category of the article or the total ratings aren't positive or negative, it will be a rock. A more valuable mineral will represent an article category his friends liked. The more they liked it, the more valuable mineral will represent it. A diamond will represent the most valuable article categories. A non-valuable icon, like talc will represent an article category that was rated negatively by his friends.

The user can earn badges with the actions he does in the game. There will be badges for reading articles attached to a specific mineral. For example "diamond lover" if the user read 100 diamond articles or "persevering" if he read a lot of articles rated as talc. The user is still allowed to choose like or dislike after reading the article without consequences for the badges. The rating system is to help his friends, so the reward for pressing the right button is that the user will get more relevant news later on, depending on his actions. We will only attach badges to how often the user uses the application, not how he uses it.

Since we have a clear view of what categories of articles a group of friends like, we can create graphs to indicate the common interests of the friend group. This way a new user can find out what his friends like, just by adding them. Without even reading a single article. The graph could also show the difference between what the user likes and what his friends like, allowing him to remove friends that have totally different interests. If you just want the articles you like, just don't invite other friends.

For finding friends easily and making it easy for our application to spread, we would make this as a facebook application. This way users don't have to register and they can just select their friends in a list to play together. It is also possible to give users the option to post badges on their wall when they earned them, allowing to attract more new players in our application.

So what are the advantages of this new concept?

• The users have a clear motivation for playing the game. If they rate articles, they will get news focussed on what they like. If his friends rate a specific category, this will also give the current user more news from that category.
• The application still has badges, but now the badges also make it possible to attract new players. In the old application badges only made the current users attached to the application.
• We think the application should be simpler and the concept more clear. We dropped the highscores and the selling system, keeping the application focussed on 1 thing.
• Users can now indicate if they didn't like a specific article. This will make sure the current category will be shown less often. They won't feel we are bullying them with articles they don't like, we are just trying to learn what they like.

We will create a storyboard for this new concept in the following week and start to test it with a paper version once that is finished.

Back to the drawing board?

After the reviews we got from the paper version, we now have a better look at our concept. It seems the concept isn't very clear for the users. Not all the reviewers seem to understand the point of the game.
This leaves us with a tough choice. Do we have to keep to our current concept of mining with a tree-structure or do we have to change to yet another idea?

What are the pros and the cons of our current idea?

• We saw the tree-structure as a way to unlock better news and more refined categories later in the game. This can however also be seen as annoying the user with stuff they don't want to read in the beginning. Only later on in the game they will be able to get the refined news they are looking for. The user will never get there, as we scare them away in the beginning of the game. For us it was clear that the user can work towards the news he likes, but apparently not all users see it this way.
• We had money in the game, but the money had no use except the highscores. This was very confusing for the users, because they expect to be able to buy something with the money.
• It seems that the rollovers were not clear enough in our paper version. Users got lost in the UI and didn't know what to do, because the information was in the rollovers. Some users also didn't know how to get out of certain menus, as there was no cross to close it. Both these things can be related to the fact that it is only a paper version, where a rollover is hard to simulate and that a paper isn't always the same as a window. We don't know this for sure, the only way to find out is by testing it in a real version. 

If we want to continue with our current idea, we should find solutions for these problems.
Now, why would we stick to our current concept?

• Some of the test-users liked the concept, even though it can use some refinement.
• We already worked out this concept. Although this should not make our decision, unwillingly it will play a role.
• The idea of badges and highscores will make the players come back to our application.

What next?

One of our test subjects (E.D.) suggested to give the articles a value based on the opinions of friends on that article. In this vision, you will rate articles which will increase their value. This could be globally or for your friends. We should decide what the meaning of the depth is. This could be refinement of categories, value of the articles found or even nothing. We had the feeling we did this, but apparently it wasn't clear enough.

We thought we could give more articles of a specific category based on what the user and his friends liked. So every category starts with the same likelihood to be digged at the surface, but the deeper you mine, the more you get articles that match the interests of your friend group. You are also able to go back up to get some more varied news. If you want to read something totally unrelated to what your friends like, you won't find it if you mine too deep. However, articles that match the interests of your friends will be worth more. To keep this system working, a user would be obligated to press like or dislike. That way no news stays unrated.

Similar to this idea, and merged with an idea of E.D., we can make a user walk around underground, indicating where articles are. These articles would be represented by mud, rocks, diamonds,... The value of the mineral indicates how much the article was rated by the friends of the user. If it receives a negative rating, its mud, a neutral or no rating makes it a stone and a positive rating makes it gold or diamonds. If a user mines a rock, and finds it iteresting he gives it a good rating and the article raises in value. If afterwards the value rises, the user gets extra points. If however others dislike it and the value decreases, the user that gave it a good rating gets penalized. In this solution the mining depth would have no use or value. The value will depend on the icon representing the article.

The problem with these new ideas is that we don't know if they will solve the problems we have with our current design. It could be that our users think they are even worse than our original idea.

We are leaning towards the last idea we described here. We will work this out in more detail and try if users prefer the paper version of that idea. Depending on the comments we get from the users, we will decide with what idea we should continue. We won't make a final decision at this moment.

Any thoughts on the new concept we chose to work out, the other concepts we described or new concepts are welcome.

Related applications

HearSay

http://www.newsgaming.de/2010/10/a-social-news-game/

This game is still being developped. The main features of it are a lot like one of the first ideas that we posted: you get a list of articles liked by your frieds, and you can read/rate articles of your own. The game aspect of it, is that it uses badges and such to keep people entertained and interested in using the app. This also overlaps with what we plan to do to attract users.

What we learn from this is that people are definitely interested in what their friends read. Although in our concept the actual concept is hidden until it is mined, the principle remains the same. We also learn from this, as well as from a ton of other applications, that badges, achievements and similar things give users a big incentive to keep using the application.

Dig Dug

http://en.wikipedia.org/wiki/Dig_Dug

Dig Dug is a very old game using essentially the way we would use mining. Here the goal is to move around in the ground and kill all the monsters to move to the next level. This game was quite popular in the eighties. Although the concept of the game is completely different, the controls/interaction of the game are similar to what we intend to use.

From this game, we can learn that moving around underground can be exciting. Although fighting is probably more exciting than moving around to find gems, we think that the additional functionality of the game should compensate for this.

Dig It

http://www.appchatter.com/2009/07/app-review-i-dig-it-by-inmotion-software-llc/

http://appadvice.com/appnn/2009/08/review-i-dig-it/
In this game, the goal is to mine as much as possible as fast a s possible, before running out of fuel. With the money from what you mined, you can buy upgrades, reload fuel, and also reach the final goal of the game: pay off your mortgage.
Once again the mining interface of this game is very similar to what we intend to use. The goal is different again: earn money, mine faster and so on. However, in contrary to Dig Dug, this game is actually about mining.

The lessons learnt are similar to those mentioned for Dig Dug. In addition, it shows that associating a value to what is being mined creates a stimulation to dig towards targets.
These two applications focus more on mining as a goal, while we intend to use mining as an entertaining way of discovering new things.

Refining our idea

Because of the negative comments on the auction system, we decided to get rid of the auction system.

We don't want to offer the application as just a click game where the news is used as in-game currency. This is why we thought of a new feature for the application.

When a user mines an article, he will get the choice of 3 possible buyers. One of them will be related to the article and the other two will just be random people. Only the person related to the article will offer a lot of money for the article, while others will offer almost nothing. There would then be a highscore of the players with the most amount of money. Those who sell the articles to the correct persons will get money faster. This way there is a reason for actually reading the articles. To implement this, we can use opencalais or something similar to get names out of the articles.

Now that we have a final concept, we will explain everything in more detail with the help of a storyboard.

Storyboard

Again the application starts with some kind of start screen.

The user start with no depth and no money in the dig tab.

All bullets in the picture are clickable.

When going over a certain bullet, a roll over menu is shown.

This menu indicates which type of article will be mined and from which newspaper.

Also an indication of progress is given.

Every time the user clicks on the "Random Story" bullet, a random new story article will be shown. More about this in just a few moments.

When a certain node is fully dug out, the user will be able to go to that point.

Also new categories will be available. New categories are refinements of the earlier category or random articles from a total new newspaper.

Earlier bullets will still be available to dig by going to that point and clicking on it.

The screen will always be centered on the user.

When the user goes deeper he will discover new layers with more interesting contents.

When a user has dug up an article, he will be able to see the abstract of the article.

When interested, he will be able to read the entire article by clicking on the read more button.

To proceed, the user has to sell the article to a random character.

1 character will reward the user generously. The other two will give consolation credits.

In this way, we do not force the user to read the article, but it is more advantageous if he would.

Highscores!

Achievements, trophies, badges, ...

Some kind of special reward for players that fulfill a certain requirement.

Users can invite friends and watch their progress, badges and trophies.

Why do we want to make this application?
Our goal is to get younger people in touch with news. Most kids and teens aren't interested in the news. By adding a playful element to the news, they might read more of the news and get interested in it. They will learn while playing the game.
How will we achieve our goals?

• We will add a competitive element to make sure users keep playing the game. The competitive element will consist of badges that users can show to others and two kinds of highscores: mining depth and coins gathered.
• We will make sure that users read the news by rewarding them for reading it. When they sell the article to the correct person, they get more money.

Similar existing applications
We found a related idea about social news gaming. http://www.newsgaming.de/2010/10/a-social-news-game/

Their concept is very similar to ours. They also want a rating system with badges and stress the competitive element while retaining the social news aspect.

They do have some variations. They let the users be reporters, while we let them be miners. They are also talking about selling things to their users and companies, while ours will be free.

They say they will be the first app in this genre. Our goal is now to try to beat them by finishing our application before them.

What we learned from the reactions

After reading all the comments on our concept, we learned that the connection between the data miner and its auction house is not clear enough.
Originally we only had the dataminer, but this concept on its own can not make users come back. Once they have mined all corridors, they will just quit playing. This is why we wanted to add some kind of living economy system. The most basic one seemed to be an auction house, as this is used in most multiplayer games and the users will be familiar with the concept.
It is true that the link between the mining and the auction system is rather vague, but we couldn't find any economy system with a better link to the dataminer. This leaves us with three options.

• We could just drop the auction system, but that would cause problems with players leaving the game due to a lack of content. This would also reduce the value of the news to nothing. It would just be a replacement of coins.
• We could replace the auction system with an other type of economy. Unfortunately we don't really have good ideas of what could come instead. The economy has to be related to the value of the news in the articles, to make sure users also read the articles. We don't want to have articles just as a replacement of coins.
• We could further explain the link between the two and keep the concept as it is, while just explaining why it is this way. In this system the users indicate how much some news is worth to them by setting a price for it in the auction house, meaning they have read the articles.

Choosing between these three options is hard, as none of them is an optimal solution. 

Once we have chosen what we will do, we will work out the concept in more detail, adding a storyboard for further explanation.