Ed Catmull, my hero, and president of the Pixar animation studio (as well as the inventor of texture mapping, subdivision surfaces, and much more) once said that there is two ways you can find out that the movie you are making is bad, either the audience tells you its bad after you have released it, or you figure it out yourself while making the movie. The difference is that if you figure it out yourself before you release it, you can still do something about it. At Pixar the goal is always to try to find problems and fix them before the films are released. The last few years i have visited various studios, and a theme that runs threw all the really good ones is how humble and self critical they are. Its not something emotional, its about being methodical. They are constantly worried about missing something, and they seek everybody's opinion in order to make it better.
A few days a go I had a conversation with a few players who complained about the fog in Love, and said it should be reduced. I found this to be very peculiar because these players have played for years and never complained before, despite the fact that I just reduced to noise a few weeks ago. So I started tweaking the numbers and after twenty minutes I gave them a new executable to try. This one they liked much better, so problem was fixed. The thing was that in the new release, I had in fact increased the fogs brightness, not reduced it. What my player were complaining about was that the fog looked bad, so they wanted me to remove it. Once I make it look good, they no longer wanted me to remove it. This is a lesson that I have learned over and over again making this game, whatever your users say, should always be interpreted as "Look at this!". How you go about fixing the issue is the job of the designer, not the player. A player can never be wrong in their opinion, but they are very likely to be wrong when it comes to how to address it.
At this years GDC I went to a full-day tutorial on user testing, but I walked out after 20 minutes for the simple reason that the people giving the tutorial, believed in what the user testing would give them answers as to how to design a game. User testing is very much like economics. No economic theory can ever take in to account every factor that influences the economy, therefore it can never be accurate. It doesn't make economics useless, it just makes it very dangerous to think that it has the answer. Testing tells you how people react to what you are showing them, not why, and not how they will react if you show them anything else. Unless you are testing the final release, the tests you are doing will have very little bearing on how people may perceive the final product. Testing is essential, but the results of testing is often also weirdly useless.
Swedish state television used to have the saying "We want to show our audience what they didn't know they wanted to see". To me this is the essence of progress, to provides something different but also great. Therefor you you cant ask your audience what they want, you can only ask them what they think of what you have already created. In video games many of the things we imagine we would enjoy turns out to be not so much fun, and some very counter intuitive thing turn out to be great, and as designers, we must learn the difference. We need to learn how to read the players feedback without taking them at face value. The more we understand about how the systems we build work, the fewer iterations we need to find the right solutions.
All this makes testing a very precarious affair in general, and if there is one thing that has been a problem during my development of Love, it is getting feedback. The problem of testing a multi-player game you are developing alone, are obvious. Further, by design Love has a dynamic environment where players are free to do what they want, and takes place over along period of time, making useful data collection almost impossible. Another problem is that some systems in the game either have limits, or are very successful, and I therefore want to be very careful about not breaking them.
An example of that is the shared resources system that gives the game have a very collaborative atmosphere where players are cooperating very well and griefers are almost unheard of. In most team or cooperative games this is a huge problem. Obviously this is great, but it also means that a new player logging in may find them selves in a game of very advanced state that completely overwhelms them with its complexity. If i would separate their experience from the rest of the group, I may make it easier to learn the game, but i would also drive a wedge in to the social fabric of the world.
Early on new players often complained about details like spelling errors in my help texts, and my feeling has always been that if that is your strongest impression of the game, the game cant be very good. I haven't head that in along time, and that tell me the game is now better (The spelling is still terrible). For a long time I believed that "good" meant attention to detail, but I no longer believe that. I believe that "good" is a hump you have to get over, and once on the other side, nothing else much matters. We love Star wars because its so good that we are along for the ride, despite its shortcomings. All the little scruffy bits becomes part of the experience and we hate it when Lucas tries to fix it. Its like when you fall in love, the person you are in love with may have all kinds of annoying habits, but when you are in love they are all endearing. If you are complaining about the little things, its because the important things are not good enough for you to forget the little things. Since testing so often brings out nitpicks about the details, it becomes easy to not see the forest for all the trees.
Yes I'm working on a new project too, a very different kind of game, and while I this time plan to put in an impressive range of tools to measure and track player behavior, It will also have to be released in a very unfinished state. I know that making a game is a long journey of trial and error, but what is scary is that most players don't see it that way. They think that the slice they play today, is exactly how I want it to be, and how it always will be. Yet, without professional testers i have no other option but to make all my failures public.
I'm still adding stuff to Love, testing things, learning things and its getting better and better (and sometimes worse), but the truth is that in a way Love missed the boat. I found out what was wrong with what i was doing far after the game had been released. I still don't know everything i want to know about it. I relesed a new versions with some things i want to test just a few minutes ago. I don't have regrets because I don't think i could have done it any other way. I had to release it in the state it was in order to find out what state it was in. The world has moved on from Love, and now its something I share with a small group of players who hang out in our team speak server, and rarely do we get new players to join, (Although you are welcome to visit, and stay for as long as you want). The servers costs more to run then the game brings in. Yet I'm having so much fun in my own private petridish where I can test ideas and explore new gameplay, and wasn't that what the project was all about from the beginning?