Testing, testing, is anyone out there?

LovePosted by Eskil Mon, June 04, 2012 22:53:15
Ed Catmull, my hero, and president of the Pixar animation studio (as well as the inventor of texture mapping, subdivision surfaces, and much more) once said that there is two ways you can find out that the movie you are making is bad, either the audience tells you its bad after you have released it, or you figure it out yourself while making the movie. The difference is that if you figure it out yourself before you release it, you can still do something about it. At Pixar the goal is always to try to find problems and fix them before the films are released. The last few years i have visited various studios, and a theme that runs threw all the really good ones is how humble and self critical they are. Its not something emotional, its about being methodical. They are constantly worried about missing something, and they seek everybody's opinion in order to make it better.

A few days a go I had a conversation with a few players who complained about the fog in Love, and said it should be reduced. I found this to be very peculiar because these players have played for years and never complained before, despite the fact that I just reduced to noise a few weeks ago. So I started tweaking the numbers and after twenty minutes I gave them a new executable to try. This one they liked much better, so problem was fixed. The thing was that in the new release, I had in fact increased the fogs brightness, not reduced it. What my player were complaining about was that the fog looked bad, so they wanted me to remove it. Once I make it look good, they no longer wanted me to remove it. This is a lesson that I have learned over and over again making this game, whatever your users say, should always be interpreted as "Look at this!". How you go about fixing the issue is the job of the designer, not the player. A player can never be wrong in their opinion, but they are very likely to be wrong when it comes to how to address it.

At this years GDC I went to a full-day tutorial on user testing, but I walked out after 20 minutes for the simple reason that the people giving the tutorial, believed in what the user testing would give them answers as to how to design a game. User testing is very much like economics. No economic theory can ever take in to account every factor that influences the economy, therefore it can never be accurate. It doesn't make economics useless, it just makes it very dangerous to think that it has the answer. Testing tells you how people react to what you are showing them, not why, and not how they will react if you show them anything else. Unless you are testing the final release, the tests you are doing will have very little bearing on how people may perceive the final product. Testing is essential, but the results of testing is often also weirdly useless.

Swedish state television used to have the saying "We want to show our audience what they didn't know they wanted to see". To me this is the essence of progress, to provides something different but also great. Therefor you you cant ask your audience what they want, you can only ask them what they think of what you have already created. In video games many of the things we imagine we would enjoy turns out to be not so much fun, and some very counter intuitive thing turn out to be great, and as designers, we must learn the difference. We need to learn how to read the players feedback without taking them at face value. The more we understand about how the systems we build work, the fewer iterations we need to find the right solutions.

All this makes testing a very precarious affair in general, and if there is one thing that has been a problem during my development of Love, it is getting feedback. The problem of testing a multi-player game you are developing alone, are obvious. Further, by design Love has a dynamic environment where players are free to do what they want, and takes place over along period of time, making useful data collection almost impossible. Another problem is that some systems in the game either have limits, or are very successful, and I therefore want to be very careful about not breaking them.

An example of that is the shared resources system that gives the game have a very collaborative atmosphere where players are cooperating very well and griefers are almost unheard of. In most team or cooperative games this is a huge problem. Obviously this is great, but it also means that a new player logging in may find them selves in a game of very advanced state that completely overwhelms them with its complexity. If i would separate their experience from the rest of the group, I may make it easier to learn the game, but i would also drive a wedge in to the social fabric of the world.

Early on new players often complained about details like spelling errors in my help texts, and my feeling has always been that if that is your strongest impression of the game, the game cant be very good. I haven't head that in along time, and that tell me the game is now better (The spelling is still terrible). For a long time I believed that "good" meant attention to detail, but I no longer believe that. I believe that "good" is a hump you have to get over, and once on the other side, nothing else much matters. We love Star wars because its so good that we are along for the ride, despite its shortcomings. All the little scruffy bits becomes part of the experience and we hate it when Lucas tries to fix it. Its like when you fall in love, the person you are in love with may have all kinds of annoying habits, but when you are in love they are all endearing. If you are complaining about the little things, its because the important things are not good enough for you to forget the little things. Since testing so often brings out nitpicks about the details, it becomes easy to not see the forest for all the trees.

Yes I'm working on a new project too, a very different kind of game, and while I this time plan to put in an impressive range of tools to measure and track player behavior, It will also have to be released in a very unfinished state. I know that making a game is a long journey of trial and error, but what is scary is that most players don't see it that way. They think that the slice they play today, is exactly how I want it to be, and how it always will be. Yet, without professional testers i have no other option but to make all my failures public.

I'm still adding stuff to Love, testing things, learning things and its getting better and better (and sometimes worse), but the truth is that in a way Love missed the boat. I found out what was wrong with what i was doing far after the game had been released. I still don't know everything i want to know about it. I relesed a new versions with some things i want to test just a few minutes ago. I don't have regrets because I don't think i could have done it any other way. I had to release it in the state it was in order to find out what state it was in. The world has moved on from Love, and now its something I share with a small group of players who hang out in our team speak server, and rarely do we get new players to join, (Although you are welcome to visit, and stay for as long as you want). The servers costs more to run then the game brings in. Yet I'm having so much fun in my own private petridish where I can test ideas and explore new gameplay, and wasn't that what the project was all about from the beginning?


  • Comments(10)

Fill in only if you are not real

The following XHTML tags are allowed: <b>, <br/>, <em>, <i>, <strong>, <u>. CSS styles and Javascript are not permitted.
Posted by Bla Sun, July 01, 2012 20:45:15

"found out what was wrong with what i was doing far after the game had been released"Could you elaborate on this? It's like you're about to share something interesting and then nothing. :(

Posted by Professor Paul1290 Sun, June 10, 2012 02:45:47

@ChrisHYou're post probably got lost due to a technical issue rather than being intentionally deleted.

Posted by ChrisH Fri, June 08, 2012 23:38:34

OK, I briefly logged-in to the latest Love. And died many times while just reading the tutorial/help stuff (which it turns out I already knew) & the HUD icons! And then died again while just standing around getting my bearings. Two points:1. From this I can only conclude that the basic "problem" of enemy AI spamming artillery has still not been "fixed" (so I assume it has been kept for some reason).2. No matter the reason, this is not a beginner-friendly experience. But if you only want to cater to "hard core" players, then no problem I guess.P.S. Given my reply about Twitter was seemingly deleted, I guess it would be better to take this to the forums where more open discussion can (hopefully) be had...?

Posted by Tom Fri, June 08, 2012 12:23:01

I suppose different players approach it from different angles. For me it was the graphics that drew me to the game, and also my interest in procedurally generated content. I was happy to explore the gameplay, but I'm not much of an MMO player and I dislike the idea of a monthly fee. I like to take my time to learn a game, and a multiplayer setting for me is just a constant reminder of how behind I am in learning to play the game, which equals less fun which in turn means I won't be spending money on it.

That's why I like one-off fees and offline games. Something like Minecraft can be just as hard to learn and require a wiki for just about any action, but at least I can take my time with it and play it when I want to. With Love, every day I don't play it I'm essentially wasting money. I also can't play anytime there is a problem with the server or what have you.

I was also referring to the twitter feed being hard to follow, because if you haven't mastered basic gameplay yet you can't appreciate all the obscure fixes and enhancements Eskil mentions in the posts. This makes it feel like you're not part of the community and that in turn lowers a newcomer's interest in the game. For instance all the AI enhancements over time are probably totally lost on most people. Surely by now the AI would be a decent fit (or could be at least adjusted) for a single player experience?

I really want love to succeed and gain traction, which it did in the beginning. But you know, you have to keep that interest up and make it easy for people to join in otherwise they will leave.

Posted by Professor Paul1290 Thu, June 07, 2012 18:29:53

Tom, I do agree that Love is very cryptic and hard to follow if you're not already into it.As a regular player, when I see you and others praising the game for "wandering around in a gorgeous world" part of me goes "wait... what!?".There are lots of things that I would praise about Love, but aesthetics aren't one of them. I might go on about the AI and how they interact with each other and the players. Maybe I would talk about the wide array of tools we get to deal with various situations. Perhaps I would talk about some of the lightweight but interesting strategy aspects of the game. Maybe I would even talk about some of the cool guns you can get.(Yea I know, that last one probably sounds like a bizarre thing to praise a game like Love for, but Love really does have some pretty crazy and cool guns! XD )I could go on, but of course I remember that it's difficult for many new players to get to the point where they can start having fun with all that stuff.Things are getting better with that though. The game is somewhat more new player friendly than it used to be, even if it's still rather confusing. Who knows, maybe we'll actually get around to getting the wiki up to date before Eskil goes in and changes everything again. XDAlso, I wouldn't worry about Love going away just yet, at least not any time soon. As Eskil mentioned he's still working on and adding a lot of cool stuff to the game.Love the product might not be selling very well, but Love the R&D project has been doing very well.

Posted by Tom Wed, June 06, 2012 00:02:23

Actually, I think that if you tried releasing an offline version of love with just one player against the ai or something as a one-off purchase the game could have a second chance, because the world and the graphics are just too charming to pass. You should consider it, also because it would be a shame if you just closed the game to concentrate on the new one you're making.

Posted by Tom Tue, June 05, 2012 23:29:02

What ChrisH said. I actually follow your twitter feed, but if you're not into the game big time, it's quite cryptic and hard to follow. It's a real shame people don't follow the game anymore, but I can't say I didn't see it coming. I miss a lack of regular news, screenshots and all that. I love technical blog posts that tell me programming developements I don't even understand, and if there was one project that had the potential to suck you in like that it was this. Unfortunately it wasn't to be. Regardless, you have earned my interest for the foreseeable future, and I will be eagerly following any game you develop.One thing, for your next game, please don't make it an online only experience. In my opinion this is where love missed the boat. Much as I loved wandering around the gorgeous world, I'm not spending several times what costs me to buy one good game. Learn from minecraft in terms of pricing, and from games like Overgrowth for engaging audiences on a weekly (or at least monthly) basis. You have my respect and admiration for being so talented and determined!

Posted by Eskil Tue, June 05, 2012 19:00:27

Check my Twitter history, it has 1500 updates and changes logged, so a fair bit has changed. The game has a free demo mode that will give you acces to well over 90% of the games features. If you want to try it, and give me feed back I would love to hear it. Dont forget to log on to

Posted by ChrisH Tue, June 05, 2012 16:54:24

I didn't know whether Love had been updated much since I originally tried it (ages ago), or even whether Love was still going. There are a couple of good reasons for that:

1. doesn't (usually) mention anything about Love, or even anything at all (since 2010).

2. The forums look dead. There are (almost) no news there either.

If you want new players, you need to be attracting them. At the most basic that means news items to show you are still active. Ideally news items will tell us about exciting changes in Love, or summarise the huge amount of changes that have occured since the last news update.

It would also be helpful if potential players could try out Love without having to pay. The "demo" used to just show you the menu screen, no actual game play. I never heard any news that this had ever changed. Reading Love's homepage again, it *almost* sounds like this might have changed... but it is hard to tell without trying it out. And that's just wrong. If you have a *playable* demo, then you should be shouting it from the roof tops (metaphorically speaking).


When I first saw Love, I saw that it had a lot of potential. But it had some *serious* playability problems (which quickly killed any fun), and I never heard anything to tell me that you had actually fixed them. I used to see news that you'd tweaked some minor stuff, or added more content (e.g. tokens) that were irrelevant to the playability problems I had experienced. Eventually I concluded that you weren't *going* to fix them (for whatever reason), and I've never heard anything to indicate that has changed. And a complete lack of news hardly helps me think that you might have done so.

Still, perhaps it would be worth trying Love again. But I'm not sure that I want to pay more money to test it, when it's not even clear why I should. To sum up: What the hell has happened to Love in the last few year(s), and why is it any better?

Posted by Paper Tue, June 05, 2012 13:35:04

Howdy Eskil,

I've followed Love (and you) before beta and have enjoyed playing the game.

But the best part of Love has been seeing the on-going development.

For me Love is not *just* a game, it is about you and your game.