Showing posts with label learning. Show all posts
Showing posts with label learning. Show all posts

2013/06/17

Interview with a Software Engineer - Part 2

A couple years ago, my cousin interviewed me for a school project, which I posted as Interview with a Game Programmer. Recently, a number of students have found my blog and asked to interview me for their own school projects! Here for Part 2 is the second interview, by Anthony Hammen.

Needless to say, the views expressed here are my own and do not necessarily reflect the views of Linden Research, Inc.


What hours do you typically work?

I usually get to work anywhere between 9am and 10:30am, and leave between 5pm and 7pm. This is pretty typical of my coworkers as well. I'm trying to get myself into a stricter 9am to 5pm schedule though!

Are your hours constant?

No, as you can tell from my previous answer, there's no real incentive to be at work at exactly the same time every day, though it might make for a more efficient routine if I did! As a programmer, it's hard to know when I'll be leaving work any given day, as I don't like to leave until I've gotten to a good stopping point so I don't forget what I was doing when I come back the next day! Though I have heard some people say that it's better to stop in the middle of a task, so you won't procrastinate on starting up the next one when you get going again. I guess it depends on how good you are at remembering!

Who do you report to?

The way my team is structured at Linden Lab, I report to both a manager and a product owner for the project I'm working on. The manager keeps track of the productivity of myself and the other developers on my team, estimates how long it will take to finish the project based on our previous progress, and helps negotiate when other teams want us to help them with stuff. The product owner comes up with the goals that we are actually working toward, basically taking the perspective of a customer and describing the product that we developers are supposed to actually build.

What education or training do you have?

I have a Bachelor's degree in Computer Science from the University of Washington (BS with College Honors). I also had a couple internships with software companies while I was in college, which taught me a lot. Even more so, the jobs I've had working full-time as a programmer after graduating have really developed my skills over the last few years. And of course, I've taught myself a ton of stuff from free resources online, like how to program in ActionScript for Flash games, or how to make ragdoll physics engines.

Are your tasks scheduled? Who handles that?

Our work is divided into two-week periods called "sprints". At the end of each sprint, we decide what we will take on for the next sprint, with the aim of planning out exactly the amount of work that we can finish completely in two weeks. Our manager and product owner oversee this planning process, but they don't directly break our work into tasks for us or tell us what we need to do when - that's our job, as developers. Since we are the ones who actually know how to do the work, we have to estimate how long things will take and commit to an appropriate amount.

What languages do you currently code in?

I've been mostly coding in C++ for the last few years, both at Linden Lab where I work now and at my last job at Fugazo. I've recently started coding in Python as well. For my side projects, I use ActionScript 3.0 for Flash.

What's your work environment like?

I work in a building that originally was a brewery, and now is an office building with three floors and lots of open space. It's pretty cool. Desks are clumped together, and spread out in clusters around an open office plan - no cubicles, though a few people work in separate offices. Most of the separate rooms are used for meetings though - there are lots of those scattered around. Of course, everyone has a computer or two on their desk. I use an exercise ball for a chair, though most people use normal office chairs. There's a fair amount of natural light, though there's a lot of fluorescent light too, unfortunately. I was able to get the lights above my desk turned off, at least, so they're not in my eyes while I work.

What is your actual job title?

I'm a Software Engineer at Linden Lab.

How do you progress in your current workplace?

My understanding is that if you're a Software Engineer, you can become a Senior Software Engineer if you become awesome and experienced. From there, you can continue along that track to be a Senior Architect, or switch over to a management role as a Dev Manager. There's not a huge ladder to climb, though. Most people will just be Software Engineers, which is totally fine. Of course, there are other tracks at the company in addition to the Software Engineer track, but I don't really know how those work.

What development processes are used at your company?

As you may have guessed from my description, we use Agile, or Scrum. This is actually a relatively new process at the company, as originally it was just a bunch of programmers working on whatever they felt like, more or less. Now we have dev managers and product owners and daily standup meetings and two-week sprints.

If you could change something about your job, what would it be?

I'd much rather be working on experimental new games (or "shared creative spaces" at Linden Lab) instead of just adding more stuff to the giant machine that is Second Life. I like working on smaller projects where I can build from the ground up, and I have lots of ideas for cool projects I'd like to try, so I'm really hoping I can switch over to that.

What do you like about your job? Dislike?

I like that I'm constantly learning new things and improving my skills. And I like the people that I work with, and the respect and autonomy I have in general. What I don't like is that I'm not working on what feels most meaningful to me right now, and that I struggle to find time and energy for my own projects outside of work.

What do others think about what you do?

I don't know, I guess I've never really asked! Well, sure, my parents are proud of me. ;) Generally, within the industry, programmers have a lot of respect - we make everything actually happen, and we deal with arcane, technical stuff that other people get confused just looking at, so they generally assume we must be really smart! Maybe people in the outside world think we're weird or nerdy or geeky, but I've never really had to deal with that kind of thing myself, so I don't know. I mean sure, every programmer understands the stereotype, and there's some truth in it, but I've always found that there's a strong undercurrent of respect. I mean, when you're in demand, as programmers generally are these days, people are going to think of you more highly!

Does this job leave you very much free time?

Generally speaking, no. The game industry is notorious for demanding long hours, especially if you're a programmer. The joke is that once you get into the game industry you no longer have time to play games. I've been lucky enough to avoid the worst of that, but even in my last job I spent at least half of each year in a moderate crunch (working long hours for an extended period of time) and then the rest of the year recovering from crunch, so I didn't have as much time or energy as I wanted to pursue other projects and activities outside of work. At Linden Lab things are a lot better, since it's not technically in the game industry - it's more of a typical software company. A lot of people at the company have kids and families, and simply won't tolerate the terrible work-life balance that is stereotypical of the game industry. I have to say, though - even when you are working reasonable hours, programming itself is one of the most mentally exhausting activities you can do, so it can be hard to have enough energy for your own projects after work!

What inspired you to become a video game programmer?

I've always liked making things. When I was a little kid, I was always drawing monsters, building with LEGO, and folding origami. In fourth grade I found out that it's possible to make computer games by this weird thing called "programming", and I loved the idea of making my own little worlds on the computer, so I decided to figure out how to do that! By the time I was in high school, I had been programming tons of games on my TI-83 Plus graphing calculator, and had dreams of creating my own game company.

What kinds of math do you use, and how did you learn it?

I don't often use math directly when I'm programming, but certainly in order to understand programming you have to be adept in a mathematical way of thinking. You have to be very familiar with how algebraic expressions work, and evaluating formulas made up of variables and functions operating on other variables and functions - it should be second nature to you. Or rather, it will become second nature as you get better at programming. So it's not that you have to be good at calculating stuff in your head - that's the computer's job - but you have to be very comfortable with understanding the formulas and expressions themselves.

However, as you get deeper into specific areas of programming, particularly game programming, you may need to use more advanced math. At the most basic level, when you are laying out buttons and text on the screen for the user interface of a game, you'll most likely be dealing with numbers for the sizes and positions of these objects, and doing lots of basic addition and subtraction to put them in the right spot relative to each other, or dividing the widths by two to get the center, for example. If this is not second nature to you, it will quickly become confusing. Or if you are making an action game where objects are moving around and colliding with other objects and shooting and all that, you'll be working with simple Newtonian physics and tweaking numbers relating to velocity, acceleration, friction, and so on, where you must be very familiar with derivatives - not in the sense of doing calculus, but in how things behave when they are connected by the mathematical relationship we call the "derivative". Or if you are dealing with things rotating, and converting angles into vectors and velocities and all that, you'll need to understand how to use basic trigonometric functions to switch between these formats. All of that stuff is pretty easy, not as hard as it sounds. But once you start getting into making your own physics engines you'll have to learn tricky geometric algorithms and maybe even do some vector calculus, which I still have some trouble with. And 3D graphics is a big mess of matrix algebra and quaternions and stuff that I've been able to avoid completely so far.

So really, I started learning programming on my own at the same time that I was learning the relevant mathematical concepts in school. I started reading about programming at the same time that I was just getting introduced to algebra in school. And not too long after I started learning trigonometry in school, I discovered that it was actually really helpful for making a turret rotate and shoot in a game! And when I was learning about physics in school, I started exploring physics in games. Unfortunately, game programming isn't really used as a way to teach math in schools, even though for me it was the best way for me to really understand it. I ended up learning this stuff mostly from online resources and tutorials. If you have a basic foundation in mathematical and algebraic thinking, learning the math you need from online resources is no harder than learning programming the same way. Doesn't make it easy, though! Just start small - only learn as much as you need to do the game you're working on now. As you make more complex games, you can learn more complex math.

And here is Part 1, if you missed it!

2013/06/15

Interview with a Software Engineer - Part 1

A couple years ago, my cousin interviewed me for a school project, which I posted as Interview with a Game Programmer. Recently, a number of students have found my blog and asked to interview me for their own school projects! Here for Part 1 is the first interview, by Yair Granados.

Needless to say, the views expressed here are my own and do not necessarily reflect the views of Linden Research, Inc.

Could you explain a little about what exactly Linden Lab does?

Linden Lab is the company that created Second Life, which is probably the most well-known online virtual world. Second Life launched in 2003 and started getting a lot of media attention around 2006, which was when I first heard about it. The unique thing about Second Life is that anyone can create things in the virtual world - clothes, buildings, artwork - and fully own the rights to their creation, including selling copies of their work in the Second Life marketplace. As a result many people are able to create or participate in a wide variety of virtual experiences that they would not be able to in more gameplay-focused virtual worlds like World of Warcraft, where your only option is to kill monsters and earn gold.

Today, Second Life continues to be the primary source of income for Linden Lab, so most of the company's resources are devoted to maintaining and improving Second Life. And that's what I'm working on - programming new features and fixing bugs in Second Life. But last year the company started branching out, with four new projects - Creatorverse, Patterns, Dio, and Versu - that all fall under the theme of "shared creative spaces" kind of like Second Life. The hope is that as Linden Lab can continue to have some people developing new projects like these, and eventually at least one of them will become big like Second Life has. This is definitely what I'm most excited about, as I really like the idea of building virtual spaces where you can use your creativity and share with other people.

How long have you been programming, and how did you get started?

I've been programming professionally (getting paid for it!) for about three years now, but I first started learning about thirteen years ago now. So it took ten years between my first attempts and when I started making a living from it! Of course, I was about twelve years old when I first started so things were pretty slow at first. I didn't know anyone who could teach me programming, so I started out trying to find books about programming at the library and at bookstores - but it was hard to find books for beginners back then! It wasn't until I got this educational software in fifth grade called "Learn to Program: BASIC" that I actually managed to try out some of the stuff I had been reading about. I remember working on a simple game like Frogger, where you are an ant trying to get to the top of the screen while avoiding slugs and snails that move back and forth, but I never completely finished it. Not too long after that, my parents signed me up for a summer course on Visual Basic at a community college right before I started middle school, and I ended up making my first finished game in Visual Basic as part of a science project that year, in sixth grade. After that I started reading about C++, and tried making an artificial life simulation in C++ for my seventh grade science project, but failed spectacularly as my lack of programming experience finally caught up with my ambitions.

Fortunately, around that time we started using graphing calculators in math class, and so I began learning to program the TI-83 Plus graphing calculator using the language TI-BASIC. This was great because I couldn't get too ambitious - I made tons of little experiments and games, and just ended up doing a lot of coding, instead of spending so much time reading about it! In eighth grade I released my first calculator game on ticalc.org, and also started learning z80 assembly language, which is used to make more sophisticated games on the TI-83 Plus. By that time, programming games on the calculator was my main hobby, and I continued to release more games and grow in my programming skill over the next few years in high school and even college. As I got busier, becoming a university student and taking real computer science classes, I stopped doing much calculator programming, but it wasn't long before I found a new platform for making games - Flash. I taught myself the ActionScript language for making Flash games, and made a number of games in Flash on the side while studying computer science for my university degree. By the time I'd graduated, I'd gotten pretty comfortable with Flash and had done a few internships (some paid, some not) making use of my Flash programming skills.

My career goal has always been to make a living by making my own games, and my parents were nice enough to give me a year to try the whole indie game development thing while living rent-free, so that's what I did. I learned to use the Flixel game engine for making Flash games, and started making little experiments, hoping to come across an idea that would be worth developing into a full game. Unfortunately, I found that my lack of experience was really holding me back, not just in terms of programming but in terms of how to be consistently productive and how to manage my energy and all that, and on top of all that, I realized that I was really lonely trying to spend all day coding in my room. So, when that year was up, I had finished one game, Flydrill, but it wasn't successful and I wasn't able to finish another one, so I realized that I had to get a job.

I was fortunate enough to get a job soon after at Fugazo, a casual game studio, as a C++ game programmer. Despite my initial reservations about becoming an employee and working on other people's games, it was exactly what I needed. I ended up staying there for over two years, learning much more there than I had in any two years of college, leveled up several times in my game programming skill and understanding of the business, and emerged as a very valuable member of the team and a desirable target for recruiters, which brought me to Linden Lab. And I kept making Flash games on the side, of course. My most popular game so far, The Love Letter, was released last year while I was at Fugazo, and I'm looking forward to releasing even more games this year while I'm at Linden Lab!

What exactly does a Software Engineer do?

Generally speaking, the term "software engineer" is pretty much interchangeable with "programmer" or "developer", but it tends to imply a more systematic approach to creating software than just hacking things out in a quick and dirty way. As far as what a software engineer actually does, that of course varies from company to company, and from project to project, but you can be pretty sure that if you're a software engineer you'll be spending most of your time writing code. Programming.

You'll probably also spend a fair amount of time thinking about how you'll program something before you actually do it, and investigating possible ways of doing things, and learning new technologies and tools and programming languages as needed, and debugging and revising old code, and talking with other software engineers on your team about all this. And depending on how big of a company you work at, you might have to spend some time in meetings, but hopefully not too much. ;)

In other words, there will be non-technical people (game designers, producers, product owners) telling you what they want you to make, and then your job is to get with the other software engineers on your team, figure out how to actually make what you're supposed to make, and then make it. You're the maker. You know how to make things happen. Other people tell you what to make, but they don't know how to make it, so it's your job to actually do that. It's not always easy, but that's why there are smart people like you to figure it out.

At first, this was kind of frustrating to me, because I like thinking of my own ideas for what to make instead of just making what other people tell me to. But one thing I discovered is that there is a creative aspect to building software that is entirely invisible to non-technical people, which means that as a software engineer, you can have a lot of creative freedom in designing the structure of the software itself, even if you don't get to decide what it does. Software engineering is largely about this invisible structure, and one of the nice things about having the job title "software engineer" is that you are more likely to have the opportunity to give this structure the attention it deserves, instead of having to do things in an ugly way just to get the job done, as you might find as a mere "programmer" or "developer". But of course it varies a lot from company to company, and project to project.

What skills or training does a student like me need to become a programmer? Especially a game programmer.

Well, as a programmer (or "software engineer") you really need to get good at programming, because that's what you'll be doing! If you are good at programming, it won't be too hard to find a job as a programmer. And if you're not good at programming, why would anyone want to hire you? Of course, when you're just starting out, you don't have to be perfect - there's something to be said for raw potential - but you still need to be good at programming. Kind of like if you want to be a professional athlete - if you want to be paid to play basketball, you better be really good at playing basketball. Except that there are a lot more jobs out there for programmers than for professional athletes!

The point is, you can't just decide to start learning programming when you want to get a job - you need to start programming now. It doesn't really matter what kind of programming - as long as you enjoy it, and can start getting good at it, there's probably going to be some demand for that skill. And even if it's something like TI-BASIC for the graphing calculator, which no one would pay you for, the skills you learn in one programming language can transfer very easily over to other languages and platforms. What really matters are the fundamental skills. So start learning one programming language, and then if you're not too into it just try a different one. There are a ton of languages out there, and a ton of different types of software you can make (or even types of games, for that matter) so keep trying until you find something you really like. And at some point you may want to branch out and you can try something new. The important thing is just to do it. This may take years - it certainly did for me - but so does any real skill, whether it's learning to play the saxophone or getting a black belt in Taekwondo.

Of course, there are easier ways to start and harder ways to start, and it can be tough to know where to look if you've never programmed before. So I can give you some suggestions. First of all, don't start with C++. I know there's some macho thing about C++ being a more "serious" programming language and that you're a noob if you don't know it, but seriously. Don't waste your time learning C++ if you can help it! Until you've had several years of experience with other languages you won't really "get" what C++ is all about even if you technically know everything about the language. That was certainly the case for me in seventh grade. Nowadays I code in C++ for my job, but I still use Flash for my side projects. I don't hate C++, but I would only use it when I absolutely have to. It's powerful, but it's heavy and cumbersome, and unless you're working on a huge project with a bunch of other experienced programmers it's like carrying around a big sledgehammer when you could be using a nice, ergonomically shaped hammer that fits in your pocket instead. No matter what you think, you don't need that extra power. Seriously. Just don't.

Okay, so if not C++, where should you start? Well, there are a lot of possibilities out there, but for general programming your best bet would be Python. Python is just an awesome language overall, whether you're completely new or have years of experience. I use Python just as much as C++ in my current job. But if you try Python and you're not too into it, you could try Processing instead. I don't think anyone would call Processing an "awesome" language, but it really lets you dive into graphics and input right away, so if you're drawn to programming because of games this may be an easier place to start. Then if you try Processing and it's still not game-focused enough for you, you could give Lua a try. In particular, you should try the free Love2D engine, which lets you make games with Lua. If you try all of these and you just can't seem to get the hang of typing in all this code, I'd suggest checking out Stencyl, which is a game-making tool where you can program by dragging code blocks around, which can be a little easier to learn at first. Then once you are comfortable with that you could try learning an actual programming language again.

The important thing to keep in mind is to just make stuff. Don't try to make your big, ambitious dream game - just mess with code and make stuff happen on the screen, and shape that into games. Make small games. Make lots of small games, and finish them. And release them online! As you do this, your abilities will scale up, and your experiments will start to become interesting games in their own right.

If you've made a few games and you're looking for your next challenge, I'd recommend learning ActionScript 3.0 and a free game engine like Flixel or FlashPunk to make Flash games if you want to get further into 2D, or check out the Unity engine with C# or JavaScript if you want to go 3D. That's basically where I'm at now, so don't feel like you have to go any further to be a "real" programmer! But if you have your heart set on C++ for some strange reason, make sure you're very comfortable with making games in Flash or Unity first. And if you want to make iPhone games, Objective-C is hardly better than C++ in my opinion, and you should check out the Corona game engine for iPhone, which uses Lua.

As a game programmer, you also need to be able to collaborate with people in other disciplines, like artists and game designers. Once you've made a number of games on your own and feel ready to tackle something bigger, I highly recommend you seek out other creators online and try to make a game with them. Most collaborations fizzle out or fall apart before finishing a game, so don't get discouraged if that happens, but learn from the experience and try again another time. If you're used to making entire games all by yourself, I'd say it's easiest to collaborate with a music composer, so try that first. Then you could try collaborating with an artist too. Then you can try collaborating with a game designer or level designer if you want. But I think the hardest would be collaborating with another programmer! Because you really have to coordinate when you are both working on the same code. So start simple, and don't be surprised if things don't work out at first.

Lastly, while I have emphasized the importance of actually coding a lot, there's also something to be said for the kind of theoretical knowledge you can gain from a computer science degree (or if you are very self-directed, from learning on your own from the many resources available online). But for the first few years, you're not going to be able to really appreciate most of it, so it's much more important just to start coding a lot, and learning from your own mistakes. Eventually, if you want to get hired as a programmer you'll probably need a computer science degree from a university to even be considered, but really, having a bunch of games that you've made yourself that you can show off is just as important. And if you're going to try to make your own way as an independent game developer, the university degree is only as good as what you make of it. So start by just making a lot of games, and then once you start to get curious you can delve into the theory of computer science and maybe even get a degree in it.

If it isn't too personal, what is the range of income you can make working as a programmer?

According to the Game Developer Salary Survey, the average annual income for game programmers who have been in the industry for three years or less is around $60,000 while the average for game programmers with more than six years of experience is around $100,000. I'd say the actual range would start around $40,000 and then potentially up to the low six figures if you've been in the industry a long time. Outside of the game industry, programmer salaries are actually a bit higher, as I understand it, but the same general range still applies.

What is good about your job, or what do you like about it? And what do you not like about it?

I think one of the best things about being a programmer is that I get to constantly exercise my brain and learn new things. That means that my work can often be frustrating and confusing, but even then I don't feel too bad about it because I know that eventually I'll get through it and I'll have gotten lots of programmer experience points to help me level up my skills!

As far as the actual programming goes, I have always enjoyed building things, whether that's LEGO sets or game engines, and when I understand what I'm doing well enough then programming becomes a very enjoyable process of building, and designing an elegant and aesthetically pleasing conceptual structure for whatever I'm building. I really enjoy that, and at its best programming can give me that feeling better than just about anything else. But programming isn't always like that - you're often running into snags, or encountering problems that you can't solve without delving into some new technology that you don't understand yet, and so you have to be prepared to spend a lot of time puzzling over some tricky problem, or trying to figure out why things aren't working the way you expect them to.

It's also nice that programming is a very meritocratic profession - if you write good code, you will have respect. Because it's about the code that you write, you don't have to dress a certain way or keep a strict schedule of working hours, and while I don't know how it is in every company, I've been lucky enough to work at companies with a very friendly, casual atmosphere.

The biggest problem I have with programming is that I have to sit in front of a computer all day in order to do it. I've learned to take breaks to keep my mental energy and productivity high throughout the day, but it's still a struggle. Because programming is such a mentally taxing activity, managing my energy has been a crucially important skill for being a consistently productive programmer. And that's another drawback - I like to work on my own games on the side, but oftentimes I'll be so worn out at the end of the day that I won't have the mental capacity in the evening to work on my own stuff. However, like any kind of exercise, you do get stronger over time, so I'm still able to get some stuff done. Just not as much as I'd like.

Is there anything else I should know about this job?

Probably. But words can only get you so far. You really just have to try it. And the most important thing to know is that you can just try it. You don't need anyone's permission; you don't need anyone's help. If you're going to be a programmer, you have to be willing to persevere in pursuit of your goals, to fail and fail and keep trying until you figure it out. Because that's what programming feels like, most of the time. Even when you've been at it for thirteen years, like me. ;)

And on to Part 2!

2011/03/06

Interview with a Game Programmer

One of my cousins asked to interview me for a school project, since I'm a real game programmer now! I went all-out on these questions and thought I'd share them here, so if you know anyone who's considering a career in games or programming, feel free to send them over for a bit of behind-the-scenes with a recent college grad who has just broken into the game industry.

Needless to say, the views expressed here are my own and do not necessarily reflect the views of Fugazo, Inc.

What is your name and job title?

My name is Alex Cho Snyder and I work as a Game Programmer at the casual game development studio Fugazo, Inc.

How long have you had this job?

I started this job on August 9th, 2010, so I've been working at Fugazo for about six months.

Please explain what a typical day of work is like for you.

I have breakfast at home, pack a lunch, and take the bus downtown to the office. I usually get there around 9:30am, but time is pretty flexible there, so it's no problem if you get there a bit late. I go to my desk - all our desks are together in the same big open-studio-style room - turn on my computer and wait for it to start up so I can check my work email, download the latest images and files that my teammates have added to the project from the SVN repository, and start up Visual Studio, which is kind of like Microsoft Word for programming code. I also look through my work notebook to refresh my memory about what exactly I've been working on and what I was thinking about the day before.

I work on a team of about three or four other people, making Fugazo's next hidden-object adventure game. There are about two other such teams at Fugazo, each working on a different game. Each team has one designer, one programmer, and two or three artists. I'm a programmer. I like to think of it in comparison to building houses. The designer is the architect, who draws the plans for the house and manages the overall process. The artists are the ones who prepare the pieces of the house - boards, doors, roofs, siding, etc. And the programmer - me - is the contractor putting the pieces together and actually building a functional house. I like building things, so it works out. But I really don't like remodels, where I'm taking an old building that someone else made and have to tear out old parts of the building and fix it up and add new rooms and features. You never know if one wall or pillar is okay to take out or if the whole building will collapse when you do. That's the annoying part.

In more specific terms, my designer will come to me with a feature or puzzle or something that he wants me to add to the game, and describe exactly what he wants while I ask questions until I feel like I understand completely. I usually like to write out the details of the task in my notebook by hand, because it helps me wrap my brain around the problem. Then I'll either start writing ideas for possible solutions, or look through the existing code of the game to get a better understanding of how the solution will have to fit in. Then I start actually writing code, and testing it out. Most of my time is spent going back and forth between thinking about the problem, reading code, and writing code. If I get stuck I'll ask the designer for clarification, or talk to our lead programmer to ask how he would recommend solving the problem. Once I've got something actually working on screen, I'll show the designer and ask him for feedback. Either he'll suggest more things to change, or he'll give me the next feature he wants me to build.

At noon we have an hour for lunch, and I often walk down to a nearby alternative school to volunteer-teach game programming to some middle-school kids. After that I'm back to work, until around 5:30pm when everyone leaves and I take the bus home or to martial arts practice for exercise.

What have you enjoyed most about your occupation?

That is a hard question for me to answer. I love thinking of ideas for games, and hashing out designs with teammates, but as a programmer that is something I am rarely involved in directly. In my role as a programmer, my enjoyment comes from exercising my near-magical ability to create imaginary worlds that live inside a computer screen. When I have gained enough understanding and familiarity with my tools that this process is no longer a confusing struggle but a mildly stimulating problem to solve, I find this very enjoyable. This usually happens when I am asked to create a smaller puzzle within a larger adventure game, where I can visualize the end result I want and then figure out how to actually build it with computer code. When things are going smoothly, I can go from start to finish in this process within a day or two.

I like seeing the fruits of my labor up on the computer screen - the quicker and more immediate, the better. I like to take something that works, that I can see and play around with, and tweak it and adjust it until it's just right. It is very rewarding to me to see and appreciate and savor this thing I have created, and when I am deprived of that gratification for too long I can easily become frustrated. I'm quite stubborn, so I don't give up when things take a long time to sort out. That doesn't mean I have to like it, though! :p

Do you believe most of your fellow colleagues or workers enjoy their work? Why?

I think they do. But not necessarily in the way you might assume. Many people mistakenly conflate the fun of playing games with the hard work of making them. I like to compare the process of game development to cooking or baking. Most people love to eat pastries and cookies and donuts, just like most people enjoy playing games. But not everyone likes to bake those cookies, and not everyone likes to make games. If you want to become a baker, it certainly helps to like eating pastries. But it's not enough.

Some game companies are more like donut factories. They are not fun places to work. Fugazo, where I work, is more like a successful neighborhood bakery. The work isn't amazingly fun, but it's usually interesting, and the people are nice, and it's not a bad place to be all day. And at the end of a project, after a few months, you have a finished game that's pretty cool and you can feel proud of it and show your friends and family and all that. And you get to think about games all the time.

If you could change anything about your occupation what would it be?

If I could make games without ever having to sit at a desk and stare at a computer screen, I would. If I could run around in a virtual forest, climb inheritance trees and swing from conditional branches, build towers with blocks of code, and just use my physical body and my stereoscopic vision and my musical ears and my agile primate fingers, I would be so much happier. I love plants; I love being outside. I also love the ability to create virtual worlds that other people can experience, through computers. But every minute that I spend in front of a computer screen is a sacrifice that I make in exchange for this power to shape (virtual) reality.

More realistically, if I never had to renovate another programmer's poorly encapsulated user interface code, that would be awesome. Most frustrating part of my job right there. Also, if I had more teamwork throughout the day, I think I would enjoy that. I love collaboration, and I get just enough of it to avoid feeling unbearably lonely throughout the day, but I really get very little. There's something called pair programming where two people share a computer and take turns writing code and observing, and I've done that a few times in school and I think I'd very much like to do that again. However, it's obviously not as feasible in a small company like Fugazo where each game has only a single programmer to work on it.

Have you gone back for more educational training since you began the job?

Nope. For me, working at this job is all the educational training I need! It has been at least as educational as an equivalent time at school was, in the six months I've been at Fugazo. In computer programming, school can be essential to getting started, but once you're actually working, you should be learning a lot as you gain experience and your company starts branching out to new technologies and platforms. At the least, you can learn a lot from your own side projects as well.

What educational background do you have?

I graduated from the University of Washington with a Bachelor's degree in Computer Science. It took me a little less than five years at the UW to decide on my major, take all my required classes, and graduate. Throughout college, however, I had a summer internship or two working as a programmer and also made quite a few of my own games on the side, which is crucial if you're trying to get a job after graduating. Just a Computer Science degree doesn't cut it. You need to have experience working at a programming job, at least as an intern, and show that you have the passion, skill, and discipline to make your own games in your free time.

If you won the lottery and did not have to work, would you? Explain.

For a while, at least. ;) I decided to look for a job last summer so I could earn money, spend my days working on a team instead of by myself, and get experience learning from people who know what they're doing instead of fumbling around on my own. The lottery would only take care of one of those things. My life's work is to change the world through games. The most important thing I can do right now is to hone my skill at designing and programming games, and become fluent in all other aspects of the process as well. I will work at a company for as long as I need to until I'm ready to strike out on my own and form my own team, and I can't imagine finding a better place for that than Fugazo right now. A lot of game companies are not fun places to work. Fugazo is a notable exception.

What kind of strategies do you use when your job gets stressful?

Programming is often frustrating and can easily get stressful if you are not careful. I find it important to take breaks whenever I am really stuck or not thinking clearly, and I find that taking a walk right before lunch is very helpful in clearing my head. I also have started giving myself permission - with my boss's encouragement - to take short naps when I feel sleepy, especially in the afternoon. When it comes to jobs like programming that consist almost entirely of difficult problem-solving, it is often much more efficient to take a twenty-minute nap and come back refreshed than to fight back sleep deprivation for three hours without getting anything done. As with any sort of activity that involves sitting at a computer and staring at a screen all day, I've also found it very helpful to take a few minutes every so often to get up and move around, do some handstands, stretch, get my blood flowing, and help loosen whatever mental rut I might have gotten into while hunched over at my desk.

In your career field, is it difficult to balance work with home/family or personal life?

The game industry is one of the most notorious when it comes to poor work-life balance. This is a problem that is slowly improving, but it is definitely a real concern. Many companies, because of poor scheduling, can have months of forced "crunch time" at the end of a project, where people are working evenings, even weekends, continuously until the game is finished. Obviously, this is a very bad thing and it is arguably counter-productive in the long run, but many companies still do it. Even at Fugazo, where I work now, there may be a week or two of crunch at the end of a project, where I stay late in the evenings, but I've never worked weekends. Even then, no one is forcing me to stay late - I choose to keep working because I have a long list of things I want to get done, and I tend to be a perfectionist about these things. And I'll usually be the only one working late on most occasions, since no one is forced to crunch.

Outside of extreme circumstances like crunch time, I do often struggle to balance work and personal life. I wouldn't necessarily attribute this to the game industry itself. I think this has more to do with my inexperience (I've been working full-time for less than a year) and my tendency to cram way more things into my life than I could reasonably have time for. Though maybe the game industry tends to attract people like me anyway, so who knows!

What was it like when you went to high school? What kind of career planning did you have in high school?

I started high school about eight years ago. Back then, I wasn't thinking about jobs at all, but I knew that I wanted to do something with games. At the time I was doing a lot of programming on the graphing calculators we used in math class, making little games and releasing them online. I was also starting to read about using games for education and social change and became inspired to make that my life's work. As it turns out, those were the perfect things to be doing if I later wanted to start a career in game development!

What do you feel is most important when choosing a career?

There are a lot of ways to think about this question. One approach that I like is to think of something that you could see yourself getting really good at, if you spent the next ten or twenty years doing it for a living. That doesn't mean something you are already good at, necessarily, but something that you could see yourself learning to get really good at. I started learning to make games about ten years ago. The reason I choose to make games for a living now is not that I'm particularly good at it now, but I could see myself becoming really good at designing and making games over the next ten years. And that has less to do than the skills I currently have, and more to do with my passion and what I'm excited about and what I enjoy doing and learning and thinking about in my free time.

If you really love to do something, chances are that you could get really good at it - that if you kept doing it for a living over the next ten years, you wouldn't get bored and drop out - you'd keep learning and exploring and getting better and better as time goes on. I wouldn't suggest that you think about money right away. The jobs that tend to make a good amount of money are the jobs where you can keep getting better and better, where someone who is really good, with a lot of experience, can be worth ten or a hundred times what someone might be worth when they're just starting out. Like programming. Or game design. And you're going to have a hard time getting better and better unless you really love doing what you're trying to do. So try a bunch of things, find out what you love to do, and then ask yourself, "Which of these things could I really become good at, if I did this for a living for the next ten years?" Start there.

What advice do you have for me as I think about my future education and career plans?

Explore! Try lots of things, read about lots of things, ask lots of people about the things they do, and follow your curiosity when something attracts your interest. I was in fourth grade when I first learned that it was possible to make your own computer games. Before that, I guess I just assumed they spontaneously materialized in the store, fully formed, like action figures or LEGO sets. But once my eyes were opened to this possibility, I started reading every book I could find about how to make games, buying educational software, signing up for programming classes. At the time I had no idea what I was doing so my efforts were surprisingly ineffective, but I was persistent. I just wouldn't give up. And this interest gradually grew from a drip and a trickle here and there into a full-fledged flood of obsession that has significantly shaped my life and taken me to where I am right now.

But game development was not the only field that caught my interest. In high school, for example, I read tons of books on biology, philosophy, and how the mind works, just because I was so fascinated by these subjects. I thought I'd end up studying evolutionary biology or psychology or neuroscience in college, even though I ended up in computer science. And there's nothing wrong with that. In middle school I was big into music, and I started learning to make bamboo flutes in high school. Throughout the last several years I've been doing a lot of martial arts practice. And I'm still working as a programmer. But the key is to keep exploring. Be curious. There are so many things you could do with your life. Your life's work and passion may turn out to be something you've never even heard of yet. Before fourth grade, I'd certainly had no idea that I could make a living making games. If I hadn't been willing to explore, I would never have found this path.

And, if you want to get a job in the game industry, the best thing you can do right now is start making games. I made my first computer game in sixth grade. It wasn't a great game, but it was something, and I learned a lot in making it. And more importantly, after that I made another one. And another one. And another one. If you don't know how to make computer games, go and find out how. Look online, read books, take classes - whatever you can find. There are so many more resources out there now than there were ten years ago when I was starting out. Download a free tool like Game Maker to get started - you don't even need to learn programming. Or make board games, and test them out with your friends - no programming required. Just make games, and keep making games, and by the time you are actually trying to convince someone to hire you, you will be able say, "Look at this. I made this. And this, and this, and this."

If you want to make games, make games.

2010/10/13

Teaching Flixel

A few weeks ago, I started teaching some kids how to make games with Flixel and other free tools. I'm offering my time up as a volunteer, at a progressive school I came across through this inspiring blog. Here is an email I just wrote to my students:

Hi class,

I'm really tired right now. I should be sleeping. But I'm writing these instructions to you now because of something that is more urgent to me than sleep. River sent me an email asking me again to explain how to finish the animation and change where the bullet shoots from. He apparently wanted to figure this out enough that he bothered to remind me about it. He didn't have to do that. I don't have to do anything for you guys. But he wanted to. And because he wanted to, I will do whatever it takes to help him get there, even though I'm really busy, even though I'm tired. Because this is the whole reason that I decided to try teaching a class at PSCS. I came here on the chance that I might be able to help someone get somewhere they really want to go but can't get to by themselves. I'm not an expert in most things, but this opportunity to share what I can with people who want it is very motivating for me.

And I've been waiting for this. And now I see the beginnings of it.

I'm not attached to you wanting to learn game programming, or game design, or whatever. You love playing games, and I think there is a chance that you may come to love creating games as well, once you get more of a taste of it. It is an acquired taste. It might not be your thing. Which is fine with me. I don't really care personally either way. But if it is, and if there is a chance that it might be, I will do whatever I can to catch that spark in you, and help you kindle it into a steady flame, if you so choose. Just let me know.

Yeah. So, those instructions...

You added some code in the beginning of update() that said this:

var moved:Boolean = false;

Then you added code to each if-statement for the arrow keys, to changed moved to true, if you press an arrow key during the update.

moved = true;

So you have a virtual bucket, sitting in your computer's memory, which you call moved, in which you can put either true or false. But it doesn't do anything by itself. It just sits there.

However, it is useful. Because you can look at it. If it has true inside it, you know that an arrow key was pressed, because you put true in it whenever an arrow key is pressed. Right? If it has false inside it, you know that no arrow key was pressed.

Say you wanted to do something only if an arrow key was pressed, and another thing if no arrow key was pressed. Like play a walk animation, or play an idle animation. If you wanted to do that, then it might be helpful to look into this moved bucket, and see what's inside.

You could say...

If moved has true inside of it, then play the walk animation.
Otherwise (if it is false), play the idle animation.


And you could translate that into actual AS3 code, like this:

if (moved)
{
    _box.play("walk");
}
else
{
    _box.play("idle");
}


And you would put that code in update(), after all the arrow keys and such have been checked.

Some notes:

if (moved) is the same as if (moved == true) - because what you put between the parentheses in an if-statement is checked to see if it is equal to true. If it is equal to true, the code in the curly braces is run. Otherwise, it runs the code for else.

Why does if (FlxG.keys.LEFT) work to see if the left arrow key is pressed? Because FlxG.keys.LEFT is a virtual bucket that holds true if the left arrow key is pressed and false otherwise. Get it?

Also, there is no way to just stop an animation. You must have another animation to replace it. For example, you could create an animation, with _box.addAnimation(), that is called "idle" and that has only one frame in its list of frames. Then when you play that animation, the _box would stay on that one frame forever, unless you play a different animation later on.

You might also wonder whether telling the _box to play an animation, telling it again and again every frame, might make it start over from the beginning every frame and not get anywhere. There is a way to do that. But fortunately, Flixel will not restart the animation if it's already playing. So this works.

There you have it - the instructions for how to make an animation only play while moving. It will take a bit of work, a bit of thought, a bit of failure and frustration and trial-and-error in order to get it to work in your particular program. But this is the price we pay.

I do not demand success from you. But I do demand that you try, or I cannot help you. And I suggest that you try more than once, even if you fail at first. You have unlimited lives, when it comes to programming. Remember that.

Next.

Making bullets shoot from somewhere other than the top left corner of the box sprite.

This is easier.

Find the code where you first create the bullet. It looks something like this:

var bullet:FlxSprite = new FlxSprite(_box.x, _box.y);


Remember? That creates a new FlxSprite, and gives it the name bullet, and puts it at a certain spot on the screen. That certain spot on the screen happens to be the upper left corner of the _box sprite.

If the box sprite has its corner at the coordinates (5, 10) and it is 50 pixels wide and 50 pixels high, then where is its center?

I'll tell you the answer. Its center is at the coordinates (5 + 50 / 2, 10 + 50 / 2), which is the same as (5 + 25, 10 + 25), which is (30, 35).

To put it more generally, the box's center is at (_box.x + 25, _box.y + 25). Just add an offset of (25, 25) and you get the center. Right?

What if you wanted to put the bullet at the center of the box?

var bullet:FlxSprite = new FlxSprite(_box.x + 25, _box.y + 25);

Would that work? I suggest you try it out and see.

What if you want it to appear somewhere other than the center? Well, try using some other numbers, instead of 25.

Once you get that working (which I'm sure you will), try making an animation for the bullet. Can you do it?

Show me on Monday.

If you have any questions, let me know before the weekend.

Good luck.

2009/05/01

Designing and Evaluating Player Learning

Guess what? I'm done with school! This was my senior thesis, based on the work I did on Foldit, the protein folding game. Enjoy!


Designing and Evaluating Player Learning
in Human Computation Games

Alex Cho Snyder
March 31, 2009

Abstract

We present a new approach to the design of effective training systems for participants in human computation projects. When the computation involved is not only too complex for computers to solve, but too specialized even for an untrained human to handle, there must be an effective and enjoyable way for would-be participants to learn the skills necessary to contribute meaningful results to the project. Our work on the protein structure prediction game Foldit indicates that player learning is most strongly affected by a small set of key design components, whose effect on player understanding and ability can be evaluated through both formal and informal data collection.

1 Introduction

One new and barely explored area of computer science is the development of games that harness human brainpower to solve problems that are too complex for computers to handle. While some of these games, such as the Google Image Labeler, can be played without any special knowledge, the biochemistry game Foldit requires players to master a highly unique and specialized skill set - protein folding - before their playtime can be of any use to scientific research. Any game that expects to make use of unpaid human problem-solving ability within such a specialized domain must find a way to reliably turn new players into capable experts, and let them have fun while learning, or else no one will want to play.

Though previous human computation projects have succeeded in using game design to motivate the participation of thousands of volunteers and have labeled millions of images, none except Foldit have had the challenge of teaching an entirely new domain as complex as that of protein structure prediction. As a result, there has been no systematic attempt to identify the design principles that could make such teaching both effective and enjoyable for the participants who would like to contribute to the computational task.

Until now.

In this paper we identify four game design components that have the greatest effect on a player's conceptual understanding and technical ability: feedback, progression, usability, and interaction. We also describe several methods for evaluating the effectiveness of these components in order to guide and inform the design process. The utility of this approach is demonstrated by both the mistakes and successes of our work in improving the introductory levels of Foldit.

2 Related Work

The ESP Game, created by Luis von Ahn in 2003, was the first human computation project to successfully disguise itself as a fun game and thereby attract a large and stable population of participants [13]. Now known as the Google Image Labeler, this game has players assign useful labels to images on the web as a side effect of a simple two-player guessing game. While assigning labels to images is a trivial task for most humans, it is nearly impossible for computers. The ESP Game's pioneering accomplishment is in making this trivial task enjoyable and automatically harvesting the results, which has led to the labeling of over 50 million images [12].

Luis von Ahn has proposed a general process for turning a given computation problem into a game, which he has applied to the creation of several similar projects such as Peekaboom, Phetch, and Verbosity [12]. However, for more complex domains, such as protein structure prediction, creating the game is only half the story. If the task is nontrivial even for humans, as is the case with Foldit, then the other side of the story is in teaching people how to play at all.

In von Ahn's approach, game design is used to improve the enjoyability of these basic computation tasks. But there is another area of game design that deals with learning to accomplish these arbitrarily complex tasks in the first place. Many videogame theorists, including Raph Koster in his book A Theory of Fun for Game Design, have argued that the fun of games is fundamentally about the brain's addiction to solving patterns [8]. People like figuring things out. This applies to learning new patterns and domains just as well as it does to solving simple puzzles like those presented by the ESP Game.

Daniel Cook, game designer and author of one of the most widely read game design blogs on the web, has developed a conceptual framework for analyzing the learning in games, which forms the theoretical basis of our approach. In his article "The Chemistry of Game Design", he uses the concept of a "skill atom" to describe the process of learning a new skill, and describes the process of mastering a game as a progression through a network of skill atoms, a "skill chain" [4]. Enjoyment is defined as the result of mastering a given skill atom. "When you learn something new, when you understand it so fully you can use that knowledge to manipulate your environment for the better, you experience joy." [4]

While the skill chain model is obviously simplistic, it provides a useful framework for evaluating the learning in games. Any game that is found to fail by the skill chain model is unlikely to provide an effective learning experience in practice. With this in mind, we describe the four design components that are most crucial in facilitating game-based learning according to the skill chain model. To help put this theory into practice, we suggest a number of ways to collect data on each component's effectiveness and how to use this to inform the design process.

3 Problem

Foldit is a human computation game where players are presented with a virtual protein molecule and are asked to fold it into the shape with the lowest energy cost [6]. This is a very difficult computational task that has previously been approached through distributed computing projects such as Rosetta@home [11] and Folding@home [5]. But even with the combined power of hundreds of thousands of home computers, there still exist problems that such automated approaches fail to solve [7]. The idea with Foldit is that human ingenuity and intuition could succeed where raw computational power has failed.

Like the ESP Game, Foldit is presented as a game to motivate people to participate. Players compete to get the highest score on each protein, where score reflects the energy cost of the folded protein such that the more accurate solutions score higher. Players can also collaborate with each other by forming teams and sharing solutions and hints in the hopes of earning a higher score than competing teams.

The problem is that the rules of protein folding and the tools available to Foldit players are both complicated enough that without prior training, anyone given a typical protein to solve would have no idea what to do, or how to do it. Even a biochemist who studies proteins for a living would hardly be any better prepared to play Foldit [7].

When first introduced, Foldit had no way of teaching beginners other than a short tutorial video prepared by the game's creators. Then a sequence of introductory puzzles was added to the game. These puzzles were simpler than the big proteins given out for competition, usually requiring just one or two moves to solve. Instead of competing to get the highest score on each of these introductory puzzles, a player would only have to get past a certain score to progress to the next puzzle. This provided some assurance that the player had mastered the skills required by the previous puzzles before moving on to the next challenge.

While this sounds good in theory, the approach has its shortcomings. Even if a puzzle is designed to require the use of some new skill or concept, there is no guarantee that the player will ever learn the required skill in the first place. It may be possible to guarantee that a player who has already completed all the introductory puzzles is prepared to begin playing the real game, but there is no assurance that a significant percentage of players will actually make it through the entire sequence. And the more hints given to players to 'help' them complete each puzzle, the less assurance there is that they have actually learned the skills being tested, rather than simply having followed instructions without understanding.

The purpose of such an introductory sequence is to ensure that each player gains a certain conceptual understanding of the problem, along with the technical ability to carry out a solution within the game. In order to design an introductory sequence that successfully accomplishes this goal, we'd like to pinpoint the design considerations that are most relevant to producing conceptual understanding and technical ability, and define methods for evaluating their effect.

4 Conceptual Understanding

What we want to evaluate is how well a player has achieved mastery of a game's problem domain. To make this easier we distinguish between conceptual understanding and technical ability and consider each separately. In reality, the two are inextricably linked because, generally speaking, learning in games is learning by doing. A player learns a concept through exercising abilities in the game and observing the results. But as a basis for thinking and communicating about a game's design, or measuring its effectiveness, the distinction remains useful.

Conceptual understanding, in the context of a human computation game, refers to how well a player understands what characterizes a good solution, and how well that player can invent specific solutions that optimize such characteristics.

4.1 Understanding Goals

The first part of conceptual understanding is understanding the goals of the game. Every game has a goal. Most games have many goals. In the context of a human computation game, this means knowing what characterizes a good solution. When players understand their goals, they can choose to perform actions that will bring them closer to their goals, and avoid actions that will hinder their progress. Without a clear understanding of a game's goals, players are liable to become bored or frustrated.

4.1.1 Designing for Feedback

Feedback is the design component that has the most impact on a player's understanding of a game's goals. Simply put, feedback is how a game responds to player action, how a game speaks to the player. Whenever a player performs an action, the game immediately communicates the result of that action back to the player. That's feedback. And the most important information that can be communicated to a player through feedback is whether the action moved the player closer to a goal, or further away.

This is essential because games teach through exploratory learning [2]. The player is given a goal and a set of actions. Instead of being told how to reach the goal, the player is expected to experiment, to try out different actions in different ways. Then for each action, the game gives feedback to the player, if the goal is now closer or further, if the action was good or bad. Through sampling a variety of actions in a variety of contexts and observing the feedback that results, the player builds a mental model of how the game works, how the goal may be reached. And the player experiences satisfaction and delight when this mental model is confirmed by the game's feedback, especially when that feedback announces that the goal has been reached. Without appropriate and informative feedback, players are denied the opportunity to figure things out on their own, the learning is much less effective and the fun disappears.

Feedback in Foldit.

In Foldit, there are several independent goals that often come into conflict. A single move might be beneficial from the point of view of one goal, but harmful from the point of view of another. Successful play requires balancing between these goals for the best outcome overall. Feedback in Foldit originally consisted of a single score to show progress toward all goals simultaneously. This prevented players from seeing which goals were affected by any given action, instead seeing only the aggregate effect on all goals. In order to make it easier for players to distinguish between competing goals and balance between them, we added a bit of text that would appear after each score change, indicating which goals were affected and whether they were improved or worsened by the player's action.

While this certainly represents some improvement, the information content of the feedback remains much lower than that of the action it attempts to guide. Foldit players must manipulate a protein in three dimensions, and to guide their action appropriately we would like to provide feedback in three dimensions as well. For each major goal in Foldit there exists a way to pinpoint the location of problem areas and display these to the player. However, few of these visualizations were actually utilized in the original introductory puzzles. What we found is that players would tend to give much higher priority to the few goals that were represented by these local problem indicators, and would often become confused when expected to optimize their solution toward a goal not associated with such spatial feedback. In order to encourage a more balanced understanding of the goals involved, we modified one of the unused visualizations such that it could be inserted into the existing puzzles easily.

4.1.2 Evaluating with Playtests

We use playtests to evaluate the effectiveness of feedback on player understanding. For each set of changes we'd like to test, we find two or three people who have never played Foldit before and observe each of them playing the game. Evaluating feedback requires looking very closely at specific interactions and reactions between a player and the game, so we find that such informal methods end up being the most useful. The purpose is not to get statistically valid data, but to see where feedback works and where it doesn't, and to get a feel for why and how players get stuck or succeed. After a couple playtest sessions, we invariably end up with plenty of problems to fix, and more often than not a number of good ideas for how to fix them.

In accordance with "Ron's Rules for Playtesting" [1], we note that it is essential to use only new players to evaluate player learning, as anyone who has learned to play once will not be able to learn it again. We also avoid making comments or giving hints to the player, to ensure that we test the game's effectiveness rather than our own teaching ability. The only exception is that when the player is hopelessly stuck, we may try to test whether a certain change in the feedback would help lead the player back onto the right track, and describe the hypothetical new feedback to see if the player can get going again based on that one change. We also find it helpful to encourage the player to "think aloud" and verbalize thoughts, questions, and in particular, reactions to the feedback encountered in the game.

4.2 Conceptualizing Solutions

The next part of conceptual understanding is the ability to conceptualize a specific solution that satisfies the goals of the game. This solution could be a particular configuration for a protein molecule, or a battle plan in a war game. This is problem solving. This is the ability that makes humans so useful for difficult computational tasks. While a computer program may be able to 'understand' in some sense the goals of a particular task, or the criteria for an optimal solution, for complex problems there is often no efficient algorithmic way to generate solutions that satisfy these criteria. Human participants are useful to the extent that they develop this ability.

4.2.1 Designing for Progression

Progression is the most important design consideration when it comes to developing a player's problem-solving ability. Progression deals with the way new skills and concepts are introduced and how the player learns based on previous knowledge. A player must become competent with a large number of skills and concepts before achieving mastery. Every new player starts out with few basic concepts and through the course of playing the game, develops a nuanced understanding of the problem domain. Progression is what facilitates this process.

Games that facilitate effective learning experiences for their players share several important characteristics [3]. First of all, mastery of such games involves a few complex skills that are decomposable into many simple skills and concepts that are more easily acquired. These simple skills are not separate and unrelated but rather combine to make up more complex skills, which themselves can be combined into yet more complex skills, until mastery is attained. There also exists, for effective games, a smooth sequence for learning all the relevant skills, starting with the most simple and building up to the most complex. Simple skills are reused in the service of more complex skills, and fun comes from the process of finding ways to use existing skills and knowledge for new purposes or problems. Progression ensures that this process remains smooth and enjoyable.

Progression in Foldit.

In Foldit, as with most games, progression is made manageable by dividing up the learning experience into a sequence of introductory puzzles. Each puzzle is a scenario in which the player may exercise some new skill, and when the game determines that the player has demonstrated sufficient competence with this new skill or understanding of this new concept, the next puzzle is made available. These gated scenarios are often referred to as "levels" and even when this pattern is not explicitly used in a game, there is almost always some purposeful engineering of the game environment to encourage practice of particular skills or familiarizing with a certain concept.

4.2.2 Evaluating with Statistics

We record how many new players complete each introductory puzzle and use these statistics to evaluate progression in Foldit. What we end up with is a graph for each day showing the gradual attrition of players, starting strong and dwindling toward the end of the introductory sequence as players become bored or frustrated and stop playing. What we hope to see is a shallow and smooth decline, with the majority of new players making it to the end of the sequence. A large number of factors must contribute to this basic decline, including how long the puzzles take to play, and how engaging the game is overall. What we are concerned with in terms of evaluating progression, however, are the steep slopes between puzzles where many players quit. Such decreases often indicate puzzles for which players are insufficiently prepared, whether because the challenge itself is too difficult or the preceding puzzles fail to teach the required concepts. Once problem areas are discovered, we turn to playtests to discover the specific ways that players get stuck and how the progression might be improved.

5 Technical Ability

Technical ability in a human computation game refers to how well a player can make use of the tools available to turn an imagined solution into an actual one in the game. This tends to be a simpler problem than that of conceptual understanding, but remains an important consideration in all games.

5.1 Recognizing Abilities

The first part of technical ability is recognizing what actions are available. For simple games this may be trivial, but for complex games it can be quite difficult, especially as new features are added to an evolving project. Any player who does not understand how to use the basic abilities of a game will end up frustrated when pushed toward a goal that requires the use of such abilities.

5.1.1 Designing for Usability

Usability is the design consideration with the most impact on how well a player will recognize the abilities that are available in any given situation. This generally translates into optimizing the spatial and temporal organization of a game's visual display for clarity, and making control schemes easy to use and easy to understand. The key concern for usability as it pertains to technical ability is to make sure players know what they can do, and how to do it.

Usability is a broad and well studied topic outside of games, even outside of software, and it is the subject of many books, such as the web usability classic Don't Make Me Think! by Steve Krug [9]. One important concept for usability is visual hierarchy, the use of graphic design techniques to imply the relative importance of interface elements as well as the grouping of related elements. Visual hierarchy is what directs the player's flow of attention across the screen, where some elements may draw attention more strongly, or hold attention for longer, or direct attention to other elements based on factors such as size, color, or position. This is important because a new player looking at a game's interface for the first time does not see it in its entirety. Instead, a player sees the most prominent visual element and begins scanning from there, and may fail to ever examine elements assumed to be unimportant based on their visual characteristics and the feedback given by the game. Difficulty in recognizing options or accurately interpreting feedback may result if the visual hierarchy of the interface does not reflect the game's rules.

Usability in Foldit.

In Foldit we made two major changes for the sake of usability. One change was to center the score display on the top of the screen to make it more prominent, and in general to distribute various interface elements around the screen where they could each be seen more clearly. The second change was to move the instructions for each puzzle from a single block of text into a sequence of text bubbles that could be attached to any interface element or part of the protein. This made it easy to point to a specific button when introducing a new tool, or in the case of tools not activated by buttons, to direct the player's attention to the part of the protein that must be manipulated. The text bubbles also provided a convenient way to draw attention to a particular interface element that might otherwise be hard to find on the screen.

5.1.2 Evaluating with Playtests

We evaluate usability through playtesting, similarly to the way we evaluate feedback. Usability is another component where observing details and specific interactions is important, so again we rely on informal methods. We are interested in finding problems and possible solutions, not statistically valid data. As mentioned before, we make sure to use new players for every test and say as little as possible during the session, so as not to taint the results with prior experience or expectations. In the case of usability tests, it is not particularly effective to suggest hypothetical changes to the game in order to see how the player responds, as usability tends to depend on very specific details of the interface that cannot be conveyed through verbal description or simulated in the imagination.

5.2 Applying Techniques

The next part of technical ability is actually applying techniques, using several actions in combination or in different situations. It's one thing for a player to know what actions are available in a given situation, but the real concern is how well the player can make use of those actions to accomplish a specific task. If players are able to see what's required and conceptualize a possible solution, all that remains for them is to put their plans into action, and in the process, produce harvestable results.

5.2.1 Designing for Interaction

Interaction is the most important design component for ensuring that a player will develop techniques for accomplishing a game's goals. Progression is also important, but good progression may still fail when paired with poor interaction. If gameplay can be thought of as a conversation, interaction describes how well a game listens to the player. Strong interaction occurs when players are provided with a rich set of skills to exercise and plentiful opportunities in which to use them. As the saying goes, practice makes perfect. When players are able to use their abilities many times in a wide variety of contexts, and are encouraged to do so by the game, they are much more likely to extract general strategies and techniques from their experience.

One way to think about this is in terms of the information density of player action, how much information a player communicates through the controls of the game. As game design blogger Krystian Majewski puts it, "20 years ago, in a NES game, players used a 4-way d-pad and two buttons. That's 6 bit of information 30 times a second. That's 648.000 bits of information per hour. THIS is the kind of interactivity and freedom players are used to." [10] Of course, if a game does not make use of all this information then the benefit is lost. What is in question here is not simply how much information the player can pump into the game, but how much information the game makes use of in a meaningful way. If a particular scenario rewards skilled use of a deep input channel, this represents strong interaction, but if the scenario is set up in such a way that only a small subset of the possible input has any meaningful effect and the rest is ignored or punished, then the interaction is weak.

Interaction in Foldit.

In Foldit, there were originally several 'puzzles' in which all that was required was to press a single button. Furthermore, attempting to experiment with any other action would often make the puzzle more or less impossible to complete without resetting it back to its original state. This obviously represents an extreme example of poor interaction, as well as poor progression, as no prior skills were involved in the completion of such trivial puzzles. In order to improve the degree of interaction involved for these particular puzzles, we redesigned each one such that a player would have to apply a previous skill to bring the protein to such a state where a single button press would complete the puzzle.

5.2.2 Evaluating with Chat

As technical ability is closely tied to progression as well as interaction, looking at attrition statistics may help point out places where players lack the technique required to complete a puzzle. However, we find that testing with live players through chat provides a more direct way to determine what players are able to accomplish. Evaluating technical ability does not require so much attention to detail as does evaluating the effects of usability or feedback.

With an in-game textual chat system set up and an easy way to share screen shots through the chat, we easily ask players to perform certain tasks and judge their ability by looking at screen shots of their result. New players who get stuck will often ask for help in the chat, and such players provide the perfect opportunity to get a feel for how well techniques are being learned at various stages in the game. Upon noticing a request for help by a new player, we ask for screen shots to help determine the issue that has the player stuck, and proceed to give verbal instructions through the chat and examine the results through screen shots. For a game with a fairly consistent population of players, these chat sessions are much faster and easier to set up than playtests are, and provide a similar measure of player ability.

6 Results

To evaluate the overall effect of our changes to Foldit, we started recording statistics on December 2, 2008 to keep track of how many new players complete each introductory puzzle. As we mentioned earlier, looking at these statistics can provide useful information about a game's progression. But we'd also like to find out whether all this redesigning and adjusting, both of our own process and the game, has resulted in any concrete benefits to the project. These statistics help us determine whether the many changes introduced have actually increased the number of new players who make it through the training sequence.

We released five major updates to Foldit since we started collecting this data. What we would hope to see after each update is an increase in the number of players who complete each puzzle. However, as the following graphs will show, this is often not the case.

Each graph shows the average player attrition before an update and after, such that the effect of the update on player attrition is apparent. The puzzles are arranged along the horizontal axis, in the order in which they are presented in the game. The vertical axis shows the percentage of new players who complete each puzzle, starting with one hundred percent on the far left, and dwindling toward the final puzzle on the far right. The red line shows the average percentage after the update, while black is the average before the update. Here the average is calculated by taking the percentage of players for each day that the update is in effect, adding these together, and dividing by the number of days. The transparent pink and gray areas show the standard deviation after and before the update, respectively. Unconnected points indicate that a puzzle was moved, added, or deleted in the update.

Figure 1: Player attrition before and after the December 08, 2008 update.

December 08, 2008 was the date of the first new release since the start of the statistics. The red line shows the player attrition after this update, while the black line shows the statistics collected from the previous update. Interestingly enough, the attrition seems to have gotten worse with this new update. However, the difficult last puzzle was also removed, meaning that the new update actually had a higher percentage of completion at the end. Despite this small triumph, we note that simply cutting out the difficult puzzles is hardly a sustainable approach if the real goal is effective learning rather than maximizing statistics.

Other than the removal of the last puzzle, which had been prompted by numerous complaints of its difficulty, the only other changes in this update were small adjustments to the text of the puzzles and perhaps some slight difficulty tweaks here and there. As we can see this had very little effect on player attrition, and what little effect it did have was apparently detrimental.

Perhaps one lesson that we might draw from this example is that small tweaks based on uninformed guesswork are a waste of time. At the time of this update, we did not recognize the importance of playtests in pointing out real rather than imagined problems. As a result, we failed to focus on real problems and wasted our time fixing problems that did not exist.

Figure 2: Player attrition before and after the December 18, 2008 update.

December 18, 2008 was the date of the second new release. The last puzzle reappeared again in this update, with its difficulty somewhat reduced. However, several other puzzles were removed on the grounds that they seemed redundant, and that perhaps by removing them players could progress to more interesting puzzles more quickly. This was another example of a guess uninformed by playtests or any other empirical method of evaluation.

Judging by these graphs, whatever immediate gain resulted all but disappeared as players encountered more difficult puzzles that might have been more manageable had they had more opportunity to practice in the 'redundant' levels. This could be a problem of interaction, where puzzles that may seem redundant are actually essential in providing sufficient opportunity for players to practice their new abilities.

Interaction may have been improved in another area however. One puzzle which previously had required a single button press was redesigned to require the use of another skill as well. But the effect of this change was hard to discern because so many more drastic changes were also introduced in the same update. A lesson we might extract from this experience is that updates with fewer changes are easier to evaluate, and should be preferred over large updates.

Figure 3: Player attrition before and after the February 10, 2009 update.

February 10, 2009 was the date of the third release. The most important change in this update was the addition of an entirely new puzzle toward the end of the sequence, intended to introduce a new tool that had been added to the game. This does bring up the question of whether a single puzzle can ever be sufficient to familiarize players with a new tool, though at the time of this update we gave little thought to the issue of interaction.

We also removed the last two puzzles in this update, again because they seemed redundant and didn't make effective use of skills from previous puzzles. Whether or not this was a good idea is difficult to tell, without also tracking statistics on player progress beyond the introductory sequence. This is something that has yet to be done in Foldit. Regardless, removing puzzles without following up by reanalyzing and redesigning the progression as necessary seems unlikely to be a particularly effective strategy, in the long run.

Like the others, this update had its share of small tweaks and fixes for supposed problems. One seems to have increased the percentage of players completing at least a few of the puzzles, but this success is short-lived. Upon reaching a more difficult puzzle, the line plunges down to a level below the previous update. Once again, these tweaks prove to be ineffective.

Figure 4: Player attrition before and after the February 19, 2009 update.

February 19, 2009 was the date of the fourth release. The biggest change for this update was in usability, where all the instructional text was moved into a sequence of text bubbles that would point to specific parts of the protein or the interface, instead of having all the text appear at once in one area of the screen. Though the effect on the player experience was significant, the change hardly registers in the statistics. Perhaps this lends support to the notion that these statistics are not particularly useful in evaluating usability.

Another usability-related change was showing a puzzle selection screen upon puzzle completion where players could choose which puzzle to play next instead of going directly to the next puzzle. This is another example where it would have been preferable to release a separate update for each modification. From the graph it is impossible to tell what effect, if any, the selection screen had on player attrition. We do not know whether players are more likely to quit when presented with a selection screen between puzzles, or less likely.

The one big drop in this graph has a known cause, however. After our first playtest, we decided to increase the difficulty of one puzzle such it would require mastery of a certain concept we wanted to teach. As it happened, this difficulty increase may have been premature.

Figure 5: Player attrition before and after the March 21, 2009 update.

March 21, 2009 was the date of the fifth and latest release. This was a large update, featuring a new system of event triggers, where the puzzle could respond with text if the player got above or below a certain score, or performed a certain action, for example. We began doing playtests with this update, and the event system made it possible to respond to the tests with meaningful tweaks, to better handle the common cases where players would get stuck.

This was also the first update to introduce significant changes to feedback. Text explaining each change in score, as well as a new variety of local problem indicator was introduced. However, we did not have time in this update to redesign the puzzles based on these changes.

The one difficulty increase from the previous update was reduced here, but a new drop was inadvertently created when an early puzzle, thought to be unnecessary, was removed. Even though playtests had been done with this change, we did not find a case where this was a problem. Perhaps this serves as a useful reminder that playtests are not statistically valid, especially for evaluating progression, whose specific effect is highly variable from player to player. The best that can be done is to cautiously introduce progression-related changes, and be prepared to revert them quickly if the statistics reveal them to be mistakes.

7 Conclusion

Teaching complex skills is itself a complex skill. And the same design components that help a player master a game can help a game designer create an effective learning experience. Usability means know what changes are possible. Start with a clear picture of what needs to be taught and what software changes are feasible. Interaction means test often, in many contexts. Don't just try one option, try several variations and see which one works best. Feedback means acquire appropriate data. Collect data to evaluate and inform each important dimension of the design. Progression means build on a solid base. Get a minimal design working and gradually add pieces to it, testing and adjusting all the way.

Through our own experience in trying to improve Foldit's training sequence, we've been able to discover a few guidelines for using various evaluative techniques to effectively guide the design process. For the rough experimental stage of development, focus on playtests, as they are quick and will provide enough information to guide a rapidly evolving design. Then test these early assumptions with statistics once the game goes out to a steady pool of players. Statistics are more reliable than playtests but provide very little information. In order to maximize the guidance that can be extracted from statistics, release small updates with only a few changes at a time, such that resulting changes in statistics can be interpreted as the effects of specific changes in the design. And if the statistics indicate a new problem, run playtests to get an idea of what component of the design is failing and how to fix it. As a general rule, use informal evaluation like playtests to guide intuition about the design, and use formal statistics to guide interpretation of the informal evaluation.

The Foldit project is still in its early stages. We expect that much more will be learned along the way and many more improvements will be continue to be made to enhance the learning experience for players. With the techniques and concepts we have begun to develop, it is conceivable that any arbitrarily complex computational task may be made accessible to a wide pool of potential participants. We hope that this possibility may inspire the creation of many new and ambitious human computation projects, and that the work presented here will help to provide a solid starting point for the development of such projects.

8 References

[1]
Carmel, Ron. "Ron's Rules for Playtesting," 2D Boy. November 2007.

[2] Cook, Daniel. "The Princess Rescuing Application: Slides," Lost Garden. October 2008.

[3] Cook, Daniel. "What Activities Can Be Turned into Games?" Lost Garden. June 2008.

[4] Cook, Daniel. "The Chemistry of Game Design," Gamasutra. July 2007.

[5] Folding@home.

[6] Foldit.

[7] Hickey, Hannah. "Computer game's high score could earn the Nobel Prize in medicine," University of Washington News. May 2008.

[8] Koster, Raph. A Theory of Fun for Game Design. Paraglyph Press, Scottsdale AZ, 2005.

[9] Krug, Steve. Don't Make Me Think! A Common Sense Approach to Web Usability, Second Edition. New Riders Publishing, Berkeley CA, 2006.

[10] Majewski, Krystian. "BORT Digest," Game Design Scrapbook. February 2009.

[11] Rosetta@home.

[12] Von Ahn, Luis, and Laura Dabbish. "General Techniques for Designing Games with a Purpose," Communications of the ACM, p.58-67, August 2008.

[13] Von Ahn, Luis, and Laura Dabbish. "Labeling Images with a Computer Game," Proceedings of the SIGCHI conference on Human Factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria.