What I'm Excited About...

This is an attempt to articulate the basic motivation behind my passion for games, the drive underlying all my explanations of Why I Care About Games. What is it that excites me, at this very basic level?

In the abstract, the things that get me excited are figuring out how to expand the limits of what people think is possible, and helping other people appreciate the things that I find interesting or beautiful. As an example of the former, a big part of the appeal of calculator game programming for me was being able to do things that people would have never thought possible with the horribly inefficient language TI-BASIC, like make a rudimentary raycasting engine for a simple FPS, or an action game with actual pixel art graphics. It was really fun to explore the possibility space of the platform, researching the most obscure functions to see if they could somehow be put to use in making games just that much faster, or establishing a standard set of ugly but beautiful shortcuts and optimizations just to shave off a few bytes or milliseconds. And of course to share and show off these tricks with other calculator programmers online.

Now I direct this creative energy toward exploring new forms for Games (with a capital G) as a whole, expanding on what people think of and what they think is possible with something that is a "game". Not an easy task. The thing that excites me the most, and what I consider to be my life's work, is to help the medium of Games grow into its own as a powerful vehicle for understanding, expressing, and sharing views of the world and dreams for the future. I believe that Games have the potential to be as significant a new medium as the printed word ever was, perhaps more, and as powerful a force for change, hopefully beneficial. I will do what I can to make sure it happens, and happens in the best way possible.

One way I think this will happen, and where I am most interested at the moment, is through transforming education. I am very passionate about improving the quality of education in terms of its effectiveness and potential to benefit all life and living, particularly by using games to teach - teaching through games. As a student, I have experienced, and continue to experience, my fair share of less-than-optimal education. I'm convinced that such instances could have resulted in much more effective and enjoyable learning if only they had been conceived and structured as games. Computer and video games have much to offer to education, not only in game design theory but directly, using actual game software to enhance traditional schooling. It is an area to which I've directed a significant portion of my game design energy.

And as I mentioned earlier, what motivates me from project to project is in connecting those fields that I find interesting and sharing my appreciation with others, whether that is in Persian classical music, the martial art and philosophy Aikido, the beauty of plants, or Northwest Coast native art, among many others. Games simply happen to be the default mode of expression I turn to when imagining how to accomplish this.

Death, Patronage, Mythic Games

I just played through Today I Die and Don't Look Back, two art games, short and sweet. Like Passage. Worth playing.

Today I Die, aside from its artistic merit, represents the start of an experiment in a new monetization model for art games, beyond advertising or microtransactions. That is, patronage, or enhanced donations. Further discussion here.

Don't Look Back is a retelling of the myth of Orpheus and Eurydice, in interactive form. There are no words, other than in the title and instructions. To my mind, it really demonstrates the similarities between myth and games. I think there is a lot to explore here.

Fourfold Formline Folly

Again, I offer no apologies for the title of this blog post.

I've just posted four drawings to my deviantART gallery, my four best pieces from a class I took about a year ago, taught by artist Marvin Oliver. No, I didn't get his autograph, but I should have. :p

They are done in the formline style, developed by many native groups along the Northwest Coast of North America, such as the Tlingit and the Haida. A formline is a line that changes in width whenever it changes direction, and when you put a bunch of formlines together in certain ways, you get art that looks kind of like this:

Leaping Velociraptor, a naturalistic design. The silhouette more or less matches that of the actual animal. Not traditional at all, perhaps to the point of being offensive to some.

Baby Turtle, a configurative design. The silhouette is recognizable as that of the animal, but the parts are distorted stylistically. Fairly traditional.

Future of Frog, an expansive design. The parts of the animal are distorted to fill a geometric shape, but remain recognizable. Inspired by the current amphibian extinction crisis.

Salmon Box, a distributed design. The parts of the animal are rearranged to fill a geometric shape, ignoring anatomical relationships. This is a common approach in traditional art.

If you like this artwork, and you want to learn more about the cultural context in which the style was developed, please see the excellent book A Story as Sharp as a Knife: The Classical Haida Mythtellers and Their World by Robert Bringhurst. I guarantee that it will get you excited about Haida culture and myth, especially if you're into games or art. Be sure to check out the review I wrote for it if you need further convincing! ;)

Designing and Evaluating Player Learning

Guess what? I'm done with school! This was my senior thesis, based on the work I did on Foldit, the protein folding game. Enjoy!

Designing and Evaluating Player Learning
in Human Computation Games

Alex Cho Snyder
March 31, 2009

Abstract

We present a new approach to the design of effective training systems for participants in human computation projects. When the computation involved is not only too complex for computers to solve, but too specialized even for an untrained human to handle, there must be an effective and enjoyable way for would-be participants to learn the skills necessary to contribute meaningful results to the project. Our work on the protein structure prediction game Foldit indicates that player learning is most strongly affected by a small set of key design components, whose effect on player understanding and ability can be evaluated through both formal and informal data collection.

1 Introduction

One new and barely explored area of computer science is the development of games that harness human brainpower to solve problems that are too complex for computers to handle. While some of these games, such as the Google Image Labeler, can be played without any special knowledge, the biochemistry game Foldit requires players to master a highly unique and specialized skill set - protein folding - before their playtime can be of any use to scientific research. Any game that expects to make use of unpaid human problem-solving ability within such a specialized domain must find a way to reliably turn new players into capable experts, and let them have fun while learning, or else no one will want to play.

Though previous human computation projects have succeeded in using game design to motivate the participation of thousands of volunteers and have labeled millions of images, none except Foldit have had the challenge of teaching an entirely new domain as complex as that of protein structure prediction. As a result, there has been no systematic attempt to identify the design principles that could make such teaching both effective and enjoyable for the participants who would like to contribute to the computational task.

Until now.

In this paper we identify four game design components that have the greatest effect on a player's conceptual understanding and technical ability: feedback, progression, usability, and interaction. We also describe several methods for evaluating the effectiveness of these components in order to guide and inform the design process. The utility of this approach is demonstrated by both the mistakes and successes of our work in improving the introductory levels of Foldit.

2 Related Work

The ESP Game, created by Luis von Ahn in 2003, was the first human computation project to successfully disguise itself as a fun game and thereby attract a large and stable population of participants [13]. Now known as the Google Image Labeler, this game has players assign useful labels to images on the web as a side effect of a simple two-player guessing game. While assigning labels to images is a trivial task for most humans, it is nearly impossible for computers. The ESP Game's pioneering accomplishment is in making this trivial task enjoyable and automatically harvesting the results, which has led to the labeling of over 50 million images [12].

Luis von Ahn has proposed a general process for turning a given computation problem into a game, which he has applied to the creation of several similar projects such as Peekaboom, Phetch, and Verbosity [12]. However, for more complex domains, such as protein structure prediction, creating the game is only half the story. If the task is nontrivial even for humans, as is the case with Foldit, then the other side of the story is in teaching people how to play at all.

In von Ahn's approach, game design is used to improve the enjoyability of these basic computation tasks. But there is another area of game design that deals with learning to accomplish these arbitrarily complex tasks in the first place. Many videogame theorists, including Raph Koster in his book A Theory of Fun for Game Design, have argued that the fun of games is fundamentally about the brain's addiction to solving patterns [8]. People like figuring things out. This applies to learning new patterns and domains just as well as it does to solving simple puzzles like those presented by the ESP Game.

Daniel Cook, game designer and author of one of the most widely read game design blogs on the web, has developed a conceptual framework for analyzing the learning in games, which forms the theoretical basis of our approach. In his article "The Chemistry of Game Design", he uses the concept of a "skill atom" to describe the process of learning a new skill, and describes the process of mastering a game as a progression through a network of skill atoms, a "skill chain" [4]. Enjoyment is defined as the result of mastering a given skill atom. "When you learn something new, when you understand it so fully you can use that knowledge to manipulate your environment for the better, you experience joy." [4]

While the skill chain model is obviously simplistic, it provides a useful framework for evaluating the learning in games. Any game that is found to fail by the skill chain model is unlikely to provide an effective learning experience in practice. With this in mind, we describe the four design components that are most crucial in facilitating game-based learning according to the skill chain model. To help put this theory into practice, we suggest a number of ways to collect data on each component's effectiveness and how to use this to inform the design process.

3 Problem

Foldit is a human computation game where players are presented with a virtual protein molecule and are asked to fold it into the shape with the lowest energy cost [6]. This is a very difficult computational task that has previously been approached through distributed computing projects such as Rosetta@home [11] and Folding@home [5]. But even with the combined power of hundreds of thousands of home computers, there still exist problems that such automated approaches fail to solve [7]. The idea with Foldit is that human ingenuity and intuition could succeed where raw computational power has failed.

Like the ESP Game, Foldit is presented as a game to motivate people to participate. Players compete to get the highest score on each protein, where score reflects the energy cost of the folded protein such that the more accurate solutions score higher. Players can also collaborate with each other by forming teams and sharing solutions and hints in the hopes of earning a higher score than competing teams.

The problem is that the rules of protein folding and the tools available to Foldit players are both complicated enough that without prior training, anyone given a typical protein to solve would have no idea what to do, or how to do it. Even a biochemist who studies proteins for a living would hardly be any better prepared to play Foldit [7].

When first introduced, Foldit had no way of teaching beginners other than a short tutorial video prepared by the game's creators. Then a sequence of introductory puzzles was added to the game. These puzzles were simpler than the big proteins given out for competition, usually requiring just one or two moves to solve. Instead of competing to get the highest score on each of these introductory puzzles, a player would only have to get past a certain score to progress to the next puzzle. This provided some assurance that the player had mastered the skills required by the previous puzzles before moving on to the next challenge.

While this sounds good in theory, the approach has its shortcomings. Even if a puzzle is designed to require the use of some new skill or concept, there is no guarantee that the player will ever learn the required skill in the first place. It may be possible to guarantee that a player who has already completed all the introductory puzzles is prepared to begin playing the real game, but there is no assurance that a significant percentage of players will actually make it through the entire sequence. And the more hints given to players to 'help' them complete each puzzle, the less assurance there is that they have actually learned the skills being tested, rather than simply having followed instructions without understanding.

The purpose of such an introductory sequence is to ensure that each player gains a certain conceptual understanding of the problem, along with the technical ability to carry out a solution within the game. In order to design an introductory sequence that successfully accomplishes this goal, we'd like to pinpoint the design considerations that are most relevant to producing conceptual understanding and technical ability, and define methods for evaluating their effect.

4 Conceptual Understanding

What we want to evaluate is how well a player has achieved mastery of a game's problem domain. To make this easier we distinguish between conceptual understanding and technical ability and consider each separately. In reality, the two are inextricably linked because, generally speaking, learning in games is learning by doing. A player learns a concept through exercising abilities in the game and observing the results. But as a basis for thinking and communicating about a game's design, or measuring its effectiveness, the distinction remains useful.

Conceptual understanding, in the context of a human computation game, refers to how well a player understands what characterizes a good solution, and how well that player can invent specific solutions that optimize such characteristics.

4.1 Understanding Goals

The first part of conceptual understanding is understanding the goals of the game. Every game has a goal. Most games have many goals. In the context of a human computation game, this means knowing what characterizes a good solution. When players understand their goals, they can choose to perform actions that will bring them closer to their goals, and avoid actions that will hinder their progress. Without a clear understanding of a game's goals, players are liable to become bored or frustrated.

4.1.1 Designing for Feedback

Feedback is the design component that has the most impact on a player's understanding of a game's goals. Simply put, feedback is how a game responds to player action, how a game speaks to the player. Whenever a player performs an action, the game immediately communicates the result of that action back to the player. That's feedback. And the most important information that can be communicated to a player through feedback is whether the action moved the player closer to a goal, or further away.

This is essential because games teach through exploratory learning [2]. The player is given a goal and a set of actions. Instead of being told how to reach the goal, the player is expected to experiment, to try out different actions in different ways. Then for each action, the game gives feedback to the player, if the goal is now closer or further, if the action was good or bad. Through sampling a variety of actions in a variety of contexts and observing the feedback that results, the player builds a mental model of how the game works, how the goal may be reached. And the player experiences satisfaction and delight when this mental model is confirmed by the game's feedback, especially when that feedback announces that the goal has been reached. Without appropriate and informative feedback, players are denied the opportunity to figure things out on their own, the learning is much less effective and the fun disappears.

Feedback in Foldit.

In Foldit, there are several independent goals that often come into conflict. A single move might be beneficial from the point of view of one goal, but harmful from the point of view of another. Successful play requires balancing between these goals for the best outcome overall. Feedback in Foldit originally consisted of a single score to show progress toward all goals simultaneously. This prevented players from seeing which goals were affected by any given action, instead seeing only the aggregate effect on all goals. In order to make it easier for players to distinguish between competing goals and balance between them, we added a bit of text that would appear after each score change, indicating which goals were affected and whether they were improved or worsened by the player's action.

While this certainly represents some improvement, the information content of the feedback remains much lower than that of the action it attempts to guide. Foldit players must manipulate a protein in three dimensions, and to guide their action appropriately we would like to provide feedback in three dimensions as well. For each major goal in Foldit there exists a way to pinpoint the location of problem areas and display these to the player. However, few of these visualizations were actually utilized in the original introductory puzzles. What we found is that players would tend to give much higher priority to the few goals that were represented by these local problem indicators, and would often become confused when expected to optimize their solution toward a goal not associated with such spatial feedback. In order to encourage a more balanced understanding of the goals involved, we modified one of the unused visualizations such that it could be inserted into the existing puzzles easily.

4.1.2 Evaluating with Playtests

We use playtests to evaluate the effectiveness of feedback on player understanding. For each set of changes we'd like to test, we find two or three people who have never played Foldit before and observe each of them playing the game. Evaluating feedback requires looking very closely at specific interactions and reactions between a player and the game, so we find that such informal methods end up being the most useful. The purpose is not to get statistically valid data, but to see where feedback works and where it doesn't, and to get a feel for why and how players get stuck or succeed. After a couple playtest sessions, we invariably end up with plenty of problems to fix, and more often than not a number of good ideas for how to fix them.

In accordance with "Ron's Rules for Playtesting" [1], we note that it is essential to use only new players to evaluate player learning, as anyone who has learned to play once will not be able to learn it again. We also avoid making comments or giving hints to the player, to ensure that we test the game's effectiveness rather than our own teaching ability. The only exception is that when the player is hopelessly stuck, we may try to test whether a certain change in the feedback would help lead the player back onto the right track, and describe the hypothetical new feedback to see if the player can get going again based on that one change. We also find it helpful to encourage the player to "think aloud" and verbalize thoughts, questions, and in particular, reactions to the feedback encountered in the game.

4.2 Conceptualizing Solutions

The next part of conceptual understanding is the ability to conceptualize a specific solution that satisfies the goals of the game. This solution could be a particular configuration for a protein molecule, or a battle plan in a war game. This is problem solving. This is the ability that makes humans so useful for difficult computational tasks. While a computer program may be able to 'understand' in some sense the goals of a particular task, or the criteria for an optimal solution, for complex problems there is often no efficient algorithmic way to generate solutions that satisfy these criteria. Human participants are useful to the extent that they develop this ability.

4.2.1 Designing for Progression

Progression is the most important design consideration when it comes to developing a player's problem-solving ability. Progression deals with the way new skills and concepts are introduced and how the player learns based on previous knowledge. A player must become competent with a large number of skills and concepts before achieving mastery. Every new player starts out with few basic concepts and through the course of playing the game, develops a nuanced understanding of the problem domain. Progression is what facilitates this process.

Games that facilitate effective learning experiences for their players share several important characteristics [3]. First of all, mastery of such games involves a few complex skills that are decomposable into many simple skills and concepts that are more easily acquired. These simple skills are not separate and unrelated but rather combine to make up more complex skills, which themselves can be combined into yet more complex skills, until mastery is attained. There also exists, for effective games, a smooth sequence for learning all the relevant skills, starting with the most simple and building up to the most complex. Simple skills are reused in the service of more complex skills, and fun comes from the process of finding ways to use existing skills and knowledge for new purposes or problems. Progression ensures that this process remains smooth and enjoyable.

Progression in Foldit.

In Foldit, as with most games, progression is made manageable by dividing up the learning experience into a sequence of introductory puzzles. Each puzzle is a scenario in which the player may exercise some new skill, and when the game determines that the player has demonstrated sufficient competence with this new skill or understanding of this new concept, the next puzzle is made available. These gated scenarios are often referred to as "levels" and even when this pattern is not explicitly used in a game, there is almost always some purposeful engineering of the game environment to encourage practice of particular skills or familiarizing with a certain concept.

4.2.2 Evaluating with Statistics

We record how many new players complete each introductory puzzle and use these statistics to evaluate progression in Foldit. What we end up with is a graph for each day showing the gradual attrition of players, starting strong and dwindling toward the end of the introductory sequence as players become bored or frustrated and stop playing. What we hope to see is a shallow and smooth decline, with the majority of new players making it to the end of the sequence. A large number of factors must contribute to this basic decline, including how long the puzzles take to play, and how engaging the game is overall. What we are concerned with in terms of evaluating progression, however, are the steep slopes between puzzles where many players quit. Such decreases often indicate puzzles for which players are insufficiently prepared, whether because the challenge itself is too difficult or the preceding puzzles fail to teach the required concepts. Once problem areas are discovered, we turn to playtests to discover the specific ways that players get stuck and how the progression might be improved.

5 Technical Ability

Technical ability in a human computation game refers to how well a player can make use of the tools available to turn an imagined solution into an actual one in the game. This tends to be a simpler problem than that of conceptual understanding, but remains an important consideration in all games.

5.1 Recognizing Abilities

The first part of technical ability is recognizing what actions are available. For simple games this may be trivial, but for complex games it can be quite difficult, especially as new features are added to an evolving project. Any player who does not understand how to use the basic abilities of a game will end up frustrated when pushed toward a goal that requires the use of such abilities.

5.1.1 Designing for Usability

Usability is the design consideration with the most impact on how well a player will recognize the abilities that are available in any given situation. This generally translates into optimizing the spatial and temporal organization of a game's visual display for clarity, and making control schemes easy to use and easy to understand. The key concern for usability as it pertains to technical ability is to make sure players know what they can do, and how to do it.

Usability is a broad and well studied topic outside of games, even outside of software, and it is the subject of many books, such as the web usability classic Don't Make Me Think! by Steve Krug [9]. One important concept for usability is visual hierarchy, the use of graphic design techniques to imply the relative importance of interface elements as well as the grouping of related elements. Visual hierarchy is what directs the player's flow of attention across the screen, where some elements may draw attention more strongly, or hold attention for longer, or direct attention to other elements based on factors such as size, color, or position. This is important because a new player looking at a game's interface for the first time does not see it in its entirety. Instead, a player sees the most prominent visual element and begins scanning from there, and may fail to ever examine elements assumed to be unimportant based on their visual characteristics and the feedback given by the game. Difficulty in recognizing options or accurately interpreting feedback may result if the visual hierarchy of the interface does not reflect the game's rules.

Usability in Foldit.

In Foldit we made two major changes for the sake of usability. One change was to center the score display on the top of the screen to make it more prominent, and in general to distribute various interface elements around the screen where they could each be seen more clearly. The second change was to move the instructions for each puzzle from a single block of text into a sequence of text bubbles that could be attached to any interface element or part of the protein. This made it easy to point to a specific button when introducing a new tool, or in the case of tools not activated by buttons, to direct the player's attention to the part of the protein that must be manipulated. The text bubbles also provided a convenient way to draw attention to a particular interface element that might otherwise be hard to find on the screen.

5.1.2 Evaluating with Playtests

We evaluate usability through playtesting, similarly to the way we evaluate feedback. Usability is another component where observing details and specific interactions is important, so again we rely on informal methods. We are interested in finding problems and possible solutions, not statistically valid data. As mentioned before, we make sure to use new players for every test and say as little as possible during the session, so as not to taint the results with prior experience or expectations. In the case of usability tests, it is not particularly effective to suggest hypothetical changes to the game in order to see how the player responds, as usability tends to depend on very specific details of the interface that cannot be conveyed through verbal description or simulated in the imagination.

5.2 Applying Techniques

The next part of technical ability is actually applying techniques, using several actions in combination or in different situations. It's one thing for a player to know what actions are available in a given situation, but the real concern is how well the player can make use of those actions to accomplish a specific task. If players are able to see what's required and conceptualize a possible solution, all that remains for them is to put their plans into action, and in the process, produce harvestable results.

5.2.1 Designing for Interaction

Interaction is the most important design component for ensuring that a player will develop techniques for accomplishing a game's goals. Progression is also important, but good progression may still fail when paired with poor interaction. If gameplay can be thought of as a conversation, interaction describes how well a game listens to the player. Strong interaction occurs when players are provided with a rich set of skills to exercise and plentiful opportunities in which to use them. As the saying goes, practice makes perfect. When players are able to use their abilities many times in a wide variety of contexts, and are encouraged to do so by the game, they are much more likely to extract general strategies and techniques from their experience.

One way to think about this is in terms of the information density of player action, how much information a player communicates through the controls of the game. As game design blogger Krystian Majewski puts it, "20 years ago, in a NES game, players used a 4-way d-pad and two buttons. That's 6 bit of information 30 times a second. That's 648.000 bits of information per hour. THIS is the kind of interactivity and freedom players are used to." [10] Of course, if a game does not make use of all this information then the benefit is lost. What is in question here is not simply how much information the player can pump into the game, but how much information the game makes use of in a meaningful way. If a particular scenario rewards skilled use of a deep input channel, this represents strong interaction, but if the scenario is set up in such a way that only a small subset of the possible input has any meaningful effect and the rest is ignored or punished, then the interaction is weak.

Interaction in Foldit.

In Foldit, there were originally several 'puzzles' in which all that was required was to press a single button. Furthermore, attempting to experiment with any other action would often make the puzzle more or less impossible to complete without resetting it back to its original state. This obviously represents an extreme example of poor interaction, as well as poor progression, as no prior skills were involved in the completion of such trivial puzzles. In order to improve the degree of interaction involved for these particular puzzles, we redesigned each one such that a player would have to apply a previous skill to bring the protein to such a state where a single button press would complete the puzzle.

5.2.2 Evaluating with Chat

As technical ability is closely tied to progression as well as interaction, looking at attrition statistics may help point out places where players lack the technique required to complete a puzzle. However, we find that testing with live players through chat provides a more direct way to determine what players are able to accomplish. Evaluating technical ability does not require so much attention to detail as does evaluating the effects of usability or feedback.

With an in-game textual chat system set up and an easy way to share screen shots through the chat, we easily ask players to perform certain tasks and judge their ability by looking at screen shots of their result. New players who get stuck will often ask for help in the chat, and such players provide the perfect opportunity to get a feel for how well techniques are being learned at various stages in the game. Upon noticing a request for help by a new player, we ask for screen shots to help determine the issue that has the player stuck, and proceed to give verbal instructions through the chat and examine the results through screen shots. For a game with a fairly consistent population of players, these chat sessions are much faster and easier to set up than playtests are, and provide a similar measure of player ability.

6 Results

To evaluate the overall effect of our changes to Foldit, we started recording statistics on December 2, 2008 to keep track of how many new players complete each introductory puzzle. As we mentioned earlier, looking at these statistics can provide useful information about a game's progression. But we'd also like to find out whether all this redesigning and adjusting, both of our own process and the game, has resulted in any concrete benefits to the project. These statistics help us determine whether the many changes introduced have actually increased the number of new players who make it through the training sequence.

We released five major updates to Foldit since we started collecting this data. What we would hope to see after each update is an increase in the number of players who complete each puzzle. However, as the following graphs will show, this is often not the case.

Each graph shows the average player attrition before an update and after, such that the effect of the update on player attrition is apparent. The puzzles are arranged along the horizontal axis, in the order in which they are presented in the game. The vertical axis shows the percentage of new players who complete each puzzle, starting with one hundred percent on the far left, and dwindling toward the final puzzle on the far right. The red line shows the average percentage after the update, while black is the average before the update. Here the average is calculated by taking the percentage of players for each day that the update is in effect, adding these together, and dividing by the number of days. The transparent pink and gray areas show the standard deviation after and before the update, respectively. Unconnected points indicate that a puzzle was moved, added, or deleted in the update.

Figure 1: Player attrition before and after the December 08, 2008 update.

December 08, 2008 was the date of the first new release since the start of the statistics. The red line shows the player attrition after this update, while the black line shows the statistics collected from the previous update. Interestingly enough, the attrition seems to have gotten worse with this new update. However, the difficult last puzzle was also removed, meaning that the new update actually had a higher percentage of completion at the end. Despite this small triumph, we note that simply cutting out the difficult puzzles is hardly a sustainable approach if the real goal is effective learning rather than maximizing statistics.

Other than the removal of the last puzzle, which had been prompted by numerous complaints of its difficulty, the only other changes in this update were small adjustments to the text of the puzzles and perhaps some slight difficulty tweaks here and there. As we can see this had very little effect on player attrition, and what little effect it did have was apparently detrimental.

Perhaps one lesson that we might draw from this example is that small tweaks based on uninformed guesswork are a waste of time. At the time of this update, we did not recognize the importance of playtests in pointing out real rather than imagined problems. As a result, we failed to focus on real problems and wasted our time fixing problems that did not exist.

Figure 2: Player attrition before and after the December 18, 2008 update.

December 18, 2008 was the date of the second new release. The last puzzle reappeared again in this update, with its difficulty somewhat reduced. However, several other puzzles were removed on the grounds that they seemed redundant, and that perhaps by removing them players could progress to more interesting puzzles more quickly. This was another example of a guess uninformed by playtests or any other empirical method of evaluation.

Judging by these graphs, whatever immediate gain resulted all but disappeared as players encountered more difficult puzzles that might have been more manageable had they had more opportunity to practice in the 'redundant' levels. This could be a problem of interaction, where puzzles that may seem redundant are actually essential in providing sufficient opportunity for players to practice their new abilities.

Interaction may have been improved in another area however. One puzzle which previously had required a single button press was redesigned to require the use of another skill as well. But the effect of this change was hard to discern because so many more drastic changes were also introduced in the same update. A lesson we might extract from this experience is that updates with fewer changes are easier to evaluate, and should be preferred over large updates.

Figure 3: Player attrition before and after the February 10, 2009 update.

February 10, 2009 was the date of the third release. The most important change in this update was the addition of an entirely new puzzle toward the end of the sequence, intended to introduce a new tool that had been added to the game. This does bring up the question of whether a single puzzle can ever be sufficient to familiarize players with a new tool, though at the time of this update we gave little thought to the issue of interaction.

We also removed the last two puzzles in this update, again because they seemed redundant and didn't make effective use of skills from previous puzzles. Whether or not this was a good idea is difficult to tell, without also tracking statistics on player progress beyond the introductory sequence. This is something that has yet to be done in Foldit. Regardless, removing puzzles without following up by reanalyzing and redesigning the progression as necessary seems unlikely to be a particularly effective strategy, in the long run.

Like the others, this update had its share of small tweaks and fixes for supposed problems. One seems to have increased the percentage of players completing at least a few of the puzzles, but this success is short-lived. Upon reaching a more difficult puzzle, the line plunges down to a level below the previous update. Once again, these tweaks prove to be ineffective.

Figure 4: Player attrition before and after the February 19, 2009 update.

February 19, 2009 was the date of the fourth release. The biggest change for this update was in usability, where all the instructional text was moved into a sequence of text bubbles that would point to specific parts of the protein or the interface, instead of having all the text appear at once in one area of the screen. Though the effect on the player experience was significant, the change hardly registers in the statistics. Perhaps this lends support to the notion that these statistics are not particularly useful in evaluating usability.

Another usability-related change was showing a puzzle selection screen upon puzzle completion where players could choose which puzzle to play next instead of going directly to the next puzzle. This is another example where it would have been preferable to release a separate update for each modification. From the graph it is impossible to tell what effect, if any, the selection screen had on player attrition. We do not know whether players are more likely to quit when presented with a selection screen between puzzles, or less likely.

The one big drop in this graph has a known cause, however. After our first playtest, we decided to increase the difficulty of one puzzle such it would require mastery of a certain concept we wanted to teach. As it happened, this difficulty increase may have been premature.

Figure 5: Player attrition before and after the March 21, 2009 update.

March 21, 2009 was the date of the fifth and latest release. This was a large update, featuring a new system of event triggers, where the puzzle could respond with text if the player got above or below a certain score, or performed a certain action, for example. We began doing playtests with this update, and the event system made it possible to respond to the tests with meaningful tweaks, to better handle the common cases where players would get stuck.

This was also the first update to introduce significant changes to feedback. Text explaining each change in score, as well as a new variety of local problem indicator was introduced. However, we did not have time in this update to redesign the puzzles based on these changes.

The one difficulty increase from the previous update was reduced here, but a new drop was inadvertently created when an early puzzle, thought to be unnecessary, was removed. Even though playtests had been done with this change, we did not find a case where this was a problem. Perhaps this serves as a useful reminder that playtests are not statistically valid, especially for evaluating progression, whose specific effect is highly variable from player to player. The best that can be done is to cautiously introduce progression-related changes, and be prepared to revert them quickly if the statistics reveal them to be mistakes.

7 Conclusion

Teaching complex skills is itself a complex skill. And the same design components that help a player master a game can help a game designer create an effective learning experience. Usability means know what changes are possible. Start with a clear picture of what needs to be taught and what software changes are feasible. Interaction means test often, in many contexts. Don't just try one option, try several variations and see which one works best. Feedback means acquire appropriate data. Collect data to evaluate and inform each important dimension of the design. Progression means build on a solid base. Get a minimal design working and gradually add pieces to it, testing and adjusting all the way.

Through our own experience in trying to improve Foldit's training sequence, we've been able to discover a few guidelines for using various evaluative techniques to effectively guide the design process. For the rough experimental stage of development, focus on playtests, as they are quick and will provide enough information to guide a rapidly evolving design. Then test these early assumptions with statistics once the game goes out to a steady pool of players. Statistics are more reliable than playtests but provide very little information. In order to maximize the guidance that can be extracted from statistics, release small updates with only a few changes at a time, such that resulting changes in statistics can be interpreted as the effects of specific changes in the design. And if the statistics indicate a new problem, run playtests to get an idea of what component of the design is failing and how to fix it. As a general rule, use informal evaluation like playtests to guide intuition about the design, and use formal statistics to guide interpretation of the informal evaluation.

The Foldit project is still in its early stages. We expect that much more will be learned along the way and many more improvements will be continue to be made to enhance the learning experience for players. With the techniques and concepts we have begun to develop, it is conceivable that any arbitrarily complex computational task may be made accessible to a wide pool of potential participants. We hope that this possibility may inspire the creation of many new and ambitious human computation projects, and that the work presented here will help to provide a solid starting point for the development of such projects.

8 References

[1] Carmel, Ron. "Ron's Rules for Playtesting," 2D Boy. November 2007.

[2] Cook, Daniel. "The Princess Rescuing Application: Slides," Lost Garden. October 2008.

[3] Cook, Daniel. "What Activities Can Be Turned into Games?" Lost Garden. June 2008.

[4] Cook, Daniel. "The Chemistry of Game Design," Gamasutra. July 2007.

[5] Folding@home.

[6] Foldit.

[7] Hickey, Hannah. "Computer game's high score could earn the Nobel Prize in medicine," University of Washington News. May 2008.

[8] Koster, Raph. A Theory of Fun for Game Design. Paraglyph Press, Scottsdale AZ, 2005.

[9] Krug, Steve. Don't Make Me Think! A Common Sense Approach to Web Usability, Second Edition. New Riders Publishing, Berkeley CA, 2006.

[10] Majewski, Krystian. "BORT Digest," Game Design Scrapbook. February 2009.

[11] Rosetta@home.

[12] Von Ahn, Luis, and Laura Dabbish. "General Techniques for Designing Games with a Purpose," Communications of the ACM, p.58-67, August 2008.

[13] Von Ahn, Luis, and Laura Dabbish. "Labeling Images with a Computer Game," Proceedings of the SIGCHI conference on Human Factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria.

The Path

The Path is out.

I've bought it. I haven't played it yet. I haven't even watched the trailer. I haven't read any reviews because I don't want to risk reading any spoilers.

This is the new game by Tale of Tales, the indie game development studio that brought you The Endless Forest and The Graveyard. Neither of which I have really played. But I've been a follower of their blog for years now.

The Path looks poised to be their breakthrough game. It's been getting a surprising amount of critical attention. And it may be the first real example of the kind of game they've been talking about in their Realtime Art Manifesto.

"The Path isn’t much of a game. But whatever it is, it’s one hell of one of those." (Fidgit)

Try it out.

Google Is the Future of Games

It seems to me that the future of game development, in terms of minimizing the barrier to entry so anyone can easily make a game, would look a lot more like Google than like Photoshop. And it wouldn't look at all like Visual Studio. ;)

I can imagine a knowledge network of algorithms and components and behaviors, built up by people searching a space, as in Electric Sheep, and then organized and filtered by people participating in some kind of social network metagame.

If I am putting together an environment sketch and I'm looking for some procedural water ripples for a fountain, I'd be able to search for these procedural components as easily as I would for a web page. I'd be able to navigate through the space of algorithms at a finer level, too, like Biomorphs, to tweak an existing component without ever touching any code. And if I wanted to, I could modify the code directly.

Whatever social rewards I'd gather through my creation would automatically trickle down to those who created the components I used to make it. In this way, there would be an ecosystem of people creating, evolving, filtering, and combining this procedural material from which games are made and recycled back into.

I'm not saying it will happen, but something like that will have to happen before creating games becomes a mainstream activity. It's a tough problem, but I'd love to see it solved. How do you turn software engineering into an art form?

*image from one of my favorite flash animations, Pencilmation*

Instant AI - Just Add People

Even today, in the age of the Wii, video games still have a predominantly male image. To explain this historic lack of games that appeal to girls, many people point to the extremely high complexity of social or emotional systems compared to the movement of objects and projectiles. Making Space Invaders is a lot easier than making an interactive adaptation of Pride and Prejudice.

This may be true, but this complexity gap need not stop us any longer. Why bother simulating any social or emotional nuances when real live humans are so easy to co-opt into willing participants in a social space facilitated by web technology? Don't waste your time writing advanced AI, just use people.

Say you have some item, like a virtual item in a game. If you are making a social game, you might be tempted to try directly specifying its effects on a player's social network. This would be hard to do effectively. But if instead you structure your players' social interaction and communication to meaningfully involve and be affected by this virtual item, these players will construct a much more nuanced meaning for it themselves. There is no need to abstract it away.

Provide opportunities for the players to create significance themselves out of the elements you provide. Come to think of it, this approach can even apply outside of social games, sort of like how Tale of Tales would like to see more games as instruments.

Now I think that abstracted formal simulations of social dynamics have their place, particularly for educational purposes or to illustrate a certain point. But I wonder if either of these goals might still be compatible with the structured socialization approach, using real players. It's an interesting idea. How far can you go in simplifying these interactions and dynamics before people are unable or unwilling to play in your world?

*image from the excessively cute and catchy Less Than Three*

Planet Earth Is Awesome

And I mean that literally. It inspires awe. Have you had the chance to see any episodes of Planet Earth yet? Amazing. I really want to start watching those again.

The Earth is a beautiful place. I never realized how out of touch I was with the rest of life on this planet until this film brought it home to me. The sheer scale of this world, and what it means for all these millions and billions of species of plants and animals to exist. They're not just pictures in a book. Each one of them must do, and be, in a different way, every day.

There is life being lived beyond the human world and even if all humanity faces extinction, I am content that life continues to live, fragile but vigorous.

I should make a game about that. ;)

Game Idea Giveaway - Pique

...continued from The Game Idea Giveaway Thread

Request by Incrue:

• what sort of game idea you're looking for
  A puzzle that dont looks like a tetris variation, something like gimme friction
  

• what your goals are in making this game
  To make cool game ;)
  I mean, it would be a puzzle very very simple to learn but the things that are going make the game more complex came from the movements of the player, not from a level pre designed
  

• what games you've made already
  Only one point n click
  

• your favorite Flash games
  samorost,gimme friction, sylvaniah, heli attack3
  

• your abilities in game design, programming, art, and sound
  This game is not suposed to have sounds, i dont like sounds.
  Also, im not good with phisics engines, so no advanced phisics or falling blocks
  

• your preferences in game design, programming, art, and sound
  Anything but sounds.

The idea: Pique

In short, this idea is a Minesweeper-like puzzle game about the processes of evolutionary search. Your goal in each level is to find the highest peak of an invisible height map, by taking peeks at one spot at a time.

a height map...

At the beginning of each randomly generated level, you are presented with a blank and unknown map, a grid waiting to be filled in. It is like Minesweeper, but instead of randomly placed mines, there is a hidden landscape of peaks and valleys to chart and uncover. Your goal is to locate the highest Peak. You start by clicking a spot on the map to place a Peek, revealing the height of the landscape at that point. Placing several Peeks would allow you to have a rough idea of what the landscape looks like. The challenge is essentially in inferring the shape of the terrain based on limited information, as fast as possible and with as few moves as possible.

This would be a good opportunity for a simple 3D display.

might look a bit like this...

Now this could be the entire game. In fact, I'd encourage you to start by just getting this basic system up and running, and see if it's fun. I don't know. It might be fun, or it might not. I suspect a lot of the fun at this stage would depend on the particular algorithm you use to randomly generate the height map.

There's a bit more to the game though. This is supposed to be about evolutionary search, remember? Here, the player takes the role of evolution in the search for the high peak in a fitness landscape. Each point in the landscape represents a simple genome, and the height represents its fitness. Let's give the player a few extra tools to help them out. The two most basic tools used by any evolutionary algorithm are mutation and crossover. So, let the player use them.

Here's how it would work. You start off with a limited number of Peeks at your disposal. When you place a Peek, in addition to revealing the height on the map, you get some amount of money proportional to the height you just found. You can then spend that money to use your special abilities. One ability would be Mutation. You select the Mutation ability and click on a Peek that's already on the map. Then a handful of new Peeks pop up around it, scattered randomly within a short radius like mushroom spores. Money from these new Peeks would be added to your total, as usual. Some testing would be necessary to ensure that the cost of a mutation is balanced with the likely payoff.

Crossover is the other basic action. You'd select two existing Peeks on the map, and several new Peeks would pop up somewhere in the line between the two. Just like Mutation, it would cost some amount to use, and each of the Peeks created by it would earn you money based on their height.

The controls for this could actually be very minimal. Click an empty space to place a new Peek. Click an existing Peek for a mutation, and drag from one Peek to another for a crossover. Though you'd have to be sure that you give plenty of information by the cursor icon and text. If you mouse over an existing Peek, it should tell you clearly that this is a Mutation, and how much it will cost, and in what radius the new Peeks will be distributed. If you start dragging, it should tell you that you are starting a Crossover, and how much it will cost. You'll need to introduce each new tool through the introductory levels. It might be better to have a toolbar on the side where you can select Peek, Mutation, or Crossover explicitly, with some explanatory text that comes up when you mouse over them. Otherwise the players might not realize what options are available to them.

Once the player finds the highest Peak, the level is won. It would be nice to then reveal the landscape with some sort of nice 3D rendering, perhaps using ray casting.

Let me know what you think. If you're unfamiliar with the idea of evolutionary search, I'd recommend that you read this interesting excerpt from the book Out of Control.

Want an idea? Make a request on The Game Idea Giveaway Thread!

Game Idea Giveaway - Bacon Frenzy

...continued from The Game Idea Giveaway Thread

Request by Mini Chris:

• what sort of game idea you're looking for
  A platformer that has a different element then the normal platformer or a creative twist on current platformer elements
  

• what your goals are in making this game
  To create my first great game, as so far all my ideas have been not so good and I end up stop making them before they are even finished.
  

The normal idea: Swan Song

The weird idea: Bacon Frenzy


In short, this idea is physics-based platformer (like Gish) with graphics consisting of photographs of food from restaurant advertisements (kind of like And Yet It Moves except with food). That's right. I'll start with the first part.

First of all, you must play Gish if you haven't already. There's a demo. Play it. If for some reason you cannot download and install the game right now, at least watch this excellent design tour video to get an idea of what it's about.

So, now that you understand why it would be desirable to make a physics-based platformer, like Gish, in Flash, we may turn our attention to how. The blob of tar that stars as your avatar in Gish is a mass-spring system consisting of a large number of particles, too many for our poor, overworked Flash Player to reasonably handle. So, for our character, we will use the smallest mass-spring system possible - that is, three particles connected into a triangle. That, as luck would have it, is precisely the shape of a corn chip (or if you prefer, tortilla chip). Thus, our character is a corn chip. With a face, of course.

Here is how our corn chip moves: rotate left and right with the arrow keys (to roll along the ground) and compress into a very short triangle (and make a cute scrunched-up face) when the space bar is held down. Release the space bar and the corn chip jumps. If you want to get a feel for how this would work, try this suspiciously similar physics demo and hold down the left and right arrow keys at the same time, then release them. To get a feel for how the rolling along the ground movement would work, try this other physics demo. Not quite the same, but enough to show that it works.

Here is how our corn chip fights: (there's gotta be fighting, right?) corners are deadly, edges are vulnerable. Pretty simple. Mario can land on a Goomba anywhere as long as he's going down, but our corn chip has to land on a corner. Perhaps some corners are more powerful than others. Anyway, here you've got the basics for a vaguely Gish-ish platformer, where our triangular hero may jump and tumble around a playground of platforms, massacre entire species of innocent food items, collect salsa or something and rescue his girlfriend. Whatever. You are free to flesh out the story and actual gameplay as your inspiration takes you.

Here is how our corn chip is rendered: three lineTo() commands and a beginBitmapFill() with a photographic corn chip texture. You could even crop a rectangular piece out of this image and just use that for the texture bitmap. Anyway, the point is that everything in this game is made of food (or rather, images of food). And not just any food, but that hyper-real, airbrushed food that seems only to exist in advertisements for big restaurant chains. You want the player to be drowning in saliva after just a few minutes of playing this game. Frolicking among rolling hills of crispy bacon glistening with cholesterol, leaping from atop gloriously shiny apples to land on a golden-crusted loaf of artisan bread, obliterating a hostile crowd of sentient jello cubes... There are so many possibilities.

so many possibilities...

Want an idea? Make a request on The Game Idea Giveaway Thread!

Game Idea Giveaway - Swan Song

...continued from The Game Idea Giveaway Thread

Request by Mini Chris:

• what sort of game idea you're looking for
  A platformer that has a different element then the normal platformer or a creative twist on current platformer elements
  

• what your goals are in making this game
  To create my first great game, as so far all my ideas have been not so good and I end up stop making them before they are even finished.
  

• what games you've made already
  A racing game, fighting game, text based RPG, 3x Unfinished platformers, Destructible Landscape engine, a few avoiders.
  

• your favorite Flash games
  I love RPG games and platformers
  

• your abilities in game design, programming, art, and sound
  I have trouble designing games thats why I need your help =), programming i'm getting alright at now (waye sent me sourcecode of his to study :P). Art I am not really sure because I leave it till last and they are usually rushed , and sound I either make my own or buy a license.
  

• your preferences in game design, programming, art, and sound
  programming

The weird idea: Bacon Frenzy

The normal idea: Swan Song

This idea came about when I watched the trailer for The Unfinished Swan and thought, "Hmm, I should try doing that in 2D." In short the idea is a 2D maze game in the form of a platformer beat-em-up (like Rage 3). Here's the twist: like in The Unfinished Swan, the playing field is completely and undifferentiatedly white, glowing like the Matrix with full-spectrum light, infinite possibilities. Where are the walls and ceilings and floors? How do you find the right path? Here's how: the blood of your enemies. A continual stream of baddies comes in through the maze to attack you, and the resulting bloodstains (theirs, of course) outline the topography of the level. You want to coordinate your perhaps physics-based attacks to direct the particle sprays of blood into the most illuminating positions. It would look remarkably similar to this animation.

The goal could be to get to the end of the maze, as quickly as possible. Extra points through skillful combat, collectable powerups, and acquirable upgrades could certainly come into play. If you could come up with a way to procedurally generate the mazes that would be cool too. I imagine you'd want to have a scrolling tilemap with support for angled surfaces like in N, but the idea would still work with just a single-screen map and only square tiles. Likewise, it would not be necessary for the combat to be complex or physics-based, but some way to direct the sprays of blood would be necessary (at the least with a forward, upward, and downward attack). I'd be glad to advise you on the art direction or design of the game. Let me know. :)

Want an idea? Make a request on The Game Idea Giveaway Thread!

Game Idea Giveaway - Paint Physics

...continued from The Game Idea Giveaway Thread

Request by kcaz_rel:

• what sort of game idea you're looking for
  I'm not quite sure, something simple but with a twist an idea I had was a sort of physics RPG.
  

• what your goals are in making this game
  Again a new twist that is completely different from what you would expect.
  

The normal idea: Monk Tactics

The weird idea: Paint Physics

This is a game born of hatred. Or at least disdain, for the legions of Crayon Physics clones that have so shamelessly made their success, most notably Magic Pen. So, the player starts up Paint Physics, anticipating perhaps some delightful fluid dynamics and colorful creative fun with paint. They are soon dismayed to see that this apparent Crayon Physics clone does not take fingerpaint as its nostalgia-inducing motif of choice, but that the Paint here is in fact MS Paint, the maligned instrument of art noobs and of course, Satan. The word for today is anti-aesthetic. Clicking the inexpertly scrawled play button, they are greeted with the sight of a familiar blue ball and yellow star, rendered in MS Paint's distinctive style. Oh, how cute - the mouse cursor looks like the pencil tool from MS Paint. A line of text at the top, artfully displayed in an aliased Times New Roman font, suggests to the player that they use their mouse to trace around the dashed outline of a square placed above the blue ball.

Here is where the fun begins. The player dutifully proceeds to trace a line around the shape, and there it is - a messy but passable square hovering in the air above the blue ball. Now, will this shape suddenly solidify and fall to the ground with a graceful crash, nudging the blue ball delicately toward the waiting yellow star? The player releases the mouse button, expectantly. Nothing at first, but then, slowly, this squiggly outline begins to peel away like a wet noodle thrown on a wall, until the whole thing collapses into a limp, tangled heap on top of the ball. Oh dear. The player mouses over to investigate. Oh look, the cursor changes to a drag icon when the mouse is over the blue ball. Can you drag the ball? The player drags the ball around, and shakes off the ropes. Drag to yellow star - level complete!

Now, if this were the whole game, it might be somewhat amusing as a parody but not really worth all the work necessary to develop it. Fortunately, that's not the game. It's just the intro. The true game would be a physics-based mouse avoider combined with a physics construction toy. Given that the basic action is to drag a blue ball all the way to a yellow star in order to beat each level, you've got a lot of options for mouse avoider obstacles and puzzles right there. And given that you'd be using a physics engine like Box2DFlash or similar, there are lots of creative physics-based ways you could take this without having to simply imitate an existing avoider game. But anyway, let's just say that certain shapes, as well as perhaps more actively swarming enemies will result in you losing the level if the blue ball touches them. Let's say that these are all the red shapes. Well, how might the player ease their way through the hazards that lie in wait, using the humble limp noodle pencil tool that is their sole accessory?

Having the ability to create flexible, perhaps stretchy, ropes is more useful than you might think. If you draw a rope with either end inside an existing object. like a floating platform or a box sitting on the ground, the end of the rope will stay attached to it. If you draw a rope from the blue ball to, say, a heavy black box, then all of a sudden you've got a weapon against your reddish attackers - pick up the ball and starting bashing them away with the heavy black box, a la Hammerfall. Basically, you use the stretchy ropes to connect the objects that already exist in the level, rather than drawing your own. I haven't seen this mechanic used anywhere yet. But if you really feel like the player must be able to add shapes of their own, I guess you could also make use of MS Paint's circle and rectangle tools, or even the others if you can figure out how. ;) Either way, I imagine that level design and testing would take up most of the development time of this game, at least if you're using an existing physics engine like you should. The success of Paint Physics will hinge entirely on its level design.

Want an idea? Make a request on The Game Idea Giveaway Thread!