During my C++ class last quarter at UCSC (which was really only about three weeks ago), senior gameplay engineer Michael Duke came to visit and tell us what it’s like being an engineer at EA. Duke has 7 years experience in engineering games (1 year at Schell Games and 6 years at EA) and is heavily involved in university relations (presentations, interviews, etc.). At EA he worked on The Sims 3 series.
If you’re looking for a more general post about what EA looks for in new applicants, check out my other Electronic Arts UCSC Presentation post. If you’re aiming to be an engineer, you’re looking at the right post already, although it wouldn’t hurt to read the other post for other information like where to apply for jobs, other employment opportunities, and more.
Game Development Stages
The first thing Duke talked about was the stages a game goes through from start to finish. He also remarked that most AAA game developers other than EA follow a very similar process.
- Ideation: Coming up with an idea for the game. It usually involves a small team and creative director. The idea can really come from anywhere, in or out of work. It can be a new intellectual property (IP), expansion pack, anything. Talk to a senior producer about it.
- Green Light: Once the ideas are set, build a powerpoint to pitch game explaining how long it will take to build and how much money you think it will need. What usually happens is you define what you need for the next 3-6 months to prove that the game will pull through and then come back to get it screened again with specific information like ship dates and costs.
- Pre-Production: If the idea passes the green light, the game goes into this pre-production. 5-10 people on the pre-production team start building the prototype. Most teams don’t pass this stage. The keys to passing are a solid prototype, good plan, and an actual market for the game.
- Production Green Light: Your game is checked again. You now have to have concrete numbers rather than estimates for information such as budgets, team plans, and ship dates.
- Production: Once you get your production green light you go into official production. Unfortunately, this is not a guarantee your game will actually live to see a shelf or digital download. If team doesn’t meet “intervals” (specific wording he used… not sure if there’s any difference with deadlines?) they promise, the development of the game can be cancelled by the company. The development team still has to worry about staying within the budget, on schedule, with the plan, and really prove themselves that they can make the game succeed. Essentially production focuses on building all of the planned features to get to what they call “alpha functionality”.
A bit of time is taken out of production to evaluate the development and forward changes not originally in the plan to change review. Everyone on the development team is encouraged to send their new ideas to change review (ex. UI change). The production and design team constantly look through the change review and order the list of ideas that should be worked on during the change review development stage. Each requested feature is calculated a cost (development time, difficulty, etc.) which determines its placement in the change review, or if it is even included in the change review at all.
- Change Review: Time is taken out from following the original plan and development deviates a little by following the change review. This addresses problems and requested features that came up after the plan was defined during the actual development process. This stage is more about refinement and polishing.
- Alpha: The game is considered to be in “alpha” when all of the features that were planned are finally in the game. However, everything is very buggy. No longer is the development driven by features, now it is driven by bug-fixing. Quality assurance (QA) starts bug testing the game and sending feedback to the development team. Engineers get hit the hardest during this stage. You might realize that bugs are not coming from your own code, so you have to communicate with those who coded it and find solutions.
- Beta: The game is considered to be in beta the first time the team reaches zero bugs. Duke comically, yet at the same time seriously remarks that zero bugs in the bug list might only last for two minutes. (QA apparently finds bugs really fast!) Some bug fixes are pushed passed the release date as patch fixes since they might be too risky to ship the game with as bug fixes often bring newer bugs. Ideally they want patches to just be things like game balancing or like “little enrichments”, but the reality is a lot of times the games are just too big and they can’t get enough test coverage in. From Duke’s experience, some of the worst bugs are from bug fixes at the very end of beta, so it’s best to avoid fixing bugs as the deadline comes unless they are particularly game breaking.
- Final: This is the first hope that this will the last time they think they will develop the game since the game should be in an acceptable state to ship. The game stays in this stage for at least 2-3 weeks as QA constantly tests the game and and reports any issues they come across. Again, if the problem is not game breaking, then a fix for it will be pushed later on as a patch fix. If the problem is significant enough they will go and work on it.
- Gold: After about 1 week or so with QA tests, if there are no bugs coming up that require immediate attention, then the game finally reaches the point where the main development cycle ends. This version of the game is the version that consumers get on the shipment date.
The development team does postmortems (evaluations of their own game) after the game is released to figure out what they did wrong and how they can learn from their experiences to make better games in the future.
The next thing Duke talked about was Perforce. Perforce is a revision system that EA uses manage their code. (If you’re used to Subversion or Git, it’s in the same realm as that.) According to Duke, most AAA game development companies use Perforce.
EA keeps three different code lines (or more officially, “sandboxes” since each code line is technically a sandbox):
- Development Line: Where the developers check-in their most recent, working code. It is the main build that QA tests.
- Main Line: The code that stays a little more clean and only gets the code from the development line when they are ready to go into the final stage.
- Release Line: The exact copy of the code that is built and shipped to the consumer. It is generally used for emergency fixes that shipped on release.
An important concept Duke emphasized is called personal sandboxing. This is where you separate the code you personally work on from everyone else. The problem is that if you work directly on the development line, you can potentially break the entire game for the entire team (which can span up to hundreds of other people), slowing down development and angering a lot of your co-workers. By personal sandboxing, you keep your code to yourself until its ready to be checked into the development line. Also, it saves your work so just in case if your harddrive fails (which according to Duke happens to someone on his team every couple of months or so), so instead of losing months of work on your computer, at most you’ll lose maybe about a day’s worth. You can also share your sandbox with other people on the team which makes your personal sandbox into a shared sandbox.
Another note about EA’s code management is that they rely on many automated builds which will will run off your code. These builds are run by build machines which take your client specifications, sync to them, and run automated tests on your code and assets to let you know if anything broke. This is to help prevent anyone from breaking the development line.
Duke went on to talk about the workflow on a feature as an engineer.
- Pre-FAMES: The first step is called Pre-FAMES (FAMES stands for Effect Animation Modeling Engineering and Sound, what EA considers the main disciplines are required to make a game). Every discipline gets to see the features in the design document and determines how much work each feature will take. It gives the designer and producer an idea of how much work needs to be done and how long it will take. During the planning process, a process called design iteration is considered where people leave room for feature fixing and refinement from feedback.
- Design Review (DR): The team sits down and designers explain the design and goals. Everyone throws around ideas. Only about 10% of the ideas actually make it into the game, but it’s the time where people start saying, “Wouldn’t it be cool if…?” and stuff.
- Implementation Review (IR) Preparations: Any of the questions in the design with regards to implementation are worked out by the engineers. For advice you can talk to your lead or you can talk to other department to get more specific information on the feature. If the feature is technically impossible, find other possibilities to achieve the feature instead.
- IR: Everyone who is involved in a feature meet up and plan how they will work together. Before this all estimates are rough, but this is where the numbers, dates, and tasks are finalized since details from everyone becomes clear. If there’s any last questions on design this is the time to ask.
- Code: You finally get to code! You still spend a lot of time communicating with a lot of people, making sure you guys are on the same page and that your work will come together the way you want it to. When you make changes, you must communicate those changes to everyone else involved.
- Demo: Demo to it to the designer/producer! A lot of little changes are made from feedback.
- Suggest Chain Review: If there’s any big suggestions that come up in feedback, forward the idea to change review.
- Code review: Code review is the process of checking your code for any bugs or problems. Most people prefer in-person code review. There are also online options where the code is posted online and other people can log-on to a system and give you feedback digitally. Duke says that surprisingly while you talk to another person about your code, you yourself catch so many bugs.
- Check into sandbox: When you check into your personal sandbox, leave detailed comments in logs. It is extremely important to go into detail on what exactly you’ve done, because someone might be looking at your code years down the line! (As an example of how long code can be used, Duke states that The Sims 3 codebase is currently 8 years old and will live on to probably be around 10). Also you yourself will forget about the code you wrote, so comments will greatly help jog your memory if you ever need them. You can never be too careful.
- Run a check build on your code: To ready the work in your personal/shared sandbox for the development line, run a check build on your code. If it fails, then you have to go back and fix your errors until it passes the tests. Then you have to start the check-in process all over again at the code review step.
- Smoke test build: Run a smoke test locally. A smoke test is a manual test of about a minute or less that proves the baseline functionality works. This is to make sure all code in the development line is functional for any engineer on the team to continue to build on. (People who break the development line get publically shamed! Doing it a lot gets it marked on your performance review!)
- Integrate sandbox to development line: If your code passes both the check build and smoke test, integrate it into the development line!
EA’s engineers development methodologies differs depend on what team and features they are working on. Generally there is no single dominant methodology and there is usually a mix that Duke calls “extreme agifall.” On some occasions there are some teams that are well suited for a specific methodology though. The three main methodologies that are used and mixed in EA are:
- Agile development: The core principle is always be building shippable software and focus on specific features for periods of time. It assumes that projects have interchangeable resources and that engineers can do the same work at the same speed, but with engineers with dramatically difference disciplines (ex. one specializes in shaders and rendering while the other specializes in UI) it’s not possible.
One example of this methodology working is its application to tools developers. Since they are a support group to the rest of the team, they always need their tools in a shippable state since the development team is constantly using them.
One example of this methodology not working is in gameplay engineering. It just doesn’t work for it because designs are always changing and there’s always a lot of feedback and iteration involved. Plus, gameplay engineers heavily rely on other resources from other departments such as textures and animations from other artists and animators.
- Waterfall development: Everything is sequential as the work goes through different teams. The problem here is that in games people will be waiting for work to come to them since the work comes through a pipe process. It is just too slow and is too difficult to manage properly.
- Extreme programming: The idea is that programmers work in pairs since ideally they will catch more bugs. Maybe this occurs on a task or two, but a whole project dedicated to this methodology doesn’t work since having multiple people programming on one computer usually slows the process down dramatically.
A common practice used in EA is called stand-ups. This is where every morning people who are working on similar features stand in front of task board and talk about what everyone is working on. From these discussions they plan their next step accordingly.
- EA has company-wide programming standards per language.
- Most code is in C++.
- Some of the libraries they use are as old as 80s and 90s. For example, their exclusive EA STL that they still use today was originally built in the late 80s. This is mainly because they still work fine and there really isn’t any need to change them.
- Naming conventions: Lowercase ‘s’ in front of static variables, lowercase ‘m’ for member variables, classes are CamelCapped.
- Team coding standards: Teams usually set coding standards before they start working. These standards never contradict EA programming standards, they only supplement them. One example of a team standard would be deciding if everyone uses tabs or spaces for indents. This is important because for example, if one programmer formats a file in all tabs and then passes onto a programmer who reformats the file in all spaces, Perforce will say EVERY line in the file has been updated which will hide what was actually updated in the code.
- Unit testing: Engineers may build tests that won’t allow the code or game to run without passing it. While they are great, it’s best not to get too consumed in unit testing since most of the time you’re the only one who will test your feature. Unit tests are best for features where other people will extensively use and test the feature and is more common in the early portion of development or persistent design, and not so much for gameplay features.
- Language restrictions: EA rarely disables or outlaws any practices. They trust in their engineers and their peer reviews.
- Sometimes they strip down scripting. One example is taking out the DOM from Python to create a stackless Python because it was taking too much memory. In Sims 3 they stripped all of the networking from C# since all of the networking was coded in C++. They also took out file I/O since it leaves security vulnerabilities as the consumer will be able to write C# scripts to modify the game.
- Exception handling: They definitely use exception handling. One emphasis was actually how exceptions can actually be good things instead of just obstructive error messages. The topic was “Abort Callstack Without Fear of Yielding”, and Duke’s example of this was Sims 3’s reset exception. In a scenario where a Sim is stuck, by implementing a cheat code and shift clicking the Sim, a player can inject the reset exception and free the Sim from whatever the problem was. The reset exception specifically aborts the Sims call stack and repositions it in a safe location. Without it, Sims can essentially be stuck forever if they are unfortunate to ever come across a looping bug. Sure they don’t want the user to actually need to use the exception, but bugs do happen and this option is there if it ever occurs.
- Threading: Threading is used since PC, Xbox 360, and PS3 support it.
Cross Platform Development
When building across multiple platforms, EA engineers heavily rely on build automation where they develop mainly on one platform, which they call their lead SKU, and then use build automation to build the game on the other platforms. They have found that when they first used PC as the lead SKU, the development process was fine between PC and Xbox 360, but the PS3 version came up with a lot of problems. Now they use PS3 as their lead SKU, meaning that’s where they test that a feature is complete. They then waterfall back to the PC and Xbox 360. It doesn’t guarantee that there won’t be any problems with the other platforms, but it’s a lot easier since there’s usually less problems when PS3 is the lead SKU.
Originally I wrote out the word “skew” because that’s how Duke spelled out the word on his slideshow. I have since corrected the mistake as people have pointed out the correct spelling is actually “SKU”, meaning stock-keeping unit.
There are many different areas of engineering to specialize in:
- Gameplay Engineering: Implement the design of the game (rules of the game, NPCs, etc.) and are at the end of the pipeline and work with everything the other departments have created.
- Tools/Pipeline Engineering: Build and maintain the tools to support the development team.
- Network Engineering: Work with the client and server architecture for network communication.
- Database Engineering: Work with the databases.
- Automation Engineering: Build automated tests that are run on the game builds to make sure code and assets are running properly. Use PERL/Python scripts, can batch program, and know how to use build force triggers.
- Technical Artist: Similar to tools/pipeline engineering (more geared towards pipeline engineering), but the technical artist makes tools more geared for artists instead of engineers. Great for programming and artist hybrids.
- Engine Architecture (Breaks into many roles)
- Graphics Engineering: Work with graphics and shaders.
- Animation Engineering: Work with animation. Be excellent at 3D math.
- Path Planning/Routing: Work with behavior. Lots of use of A* algorithm.
- Resource Management: Work with resources and decides what should be kept in memory when and for how long. Knows what kind of caching systems should be used when.
- Optimization: Specializes in optimizing the game.
- Lighting/Shaders: Focuses mainly on lighting and shaders.
What are you lacking?
A computer science degree is a great basis, but it does not prepare you enough to make games. There are a number of skill sets that are required for game engineering that just aren’t found in most computer science curriculums.
Some things most people can work on are:
- Data structure knowledge for optimization: Most people can name data structures and give a general overview of them, but most don’t know when to use them, what the memory overhead for using them is, what’s actually bad about them, etc.
- Go beyond the classroom: Think about context how your code is going to be used, how often is it going to be used, what kind of inputs are they expecting, how many other people are going to use the code, etc. Everyone generally takes the same classes, so you need to distinguish yourself.
- Find your specialization: A great idea is find what you’re really good at or something you really enjoy and specialize in it, whether it be with artificial intelligence or graphics programming or whatever else you find yourself doing.
- C++ exposure for AAA: Most college curriculums don’t have C++ classes anymore, and even the ones that do usually don’t go low level enough. You need to have a good conscience about optimization because games need to run as smoothly as possible.
- Get a webpage!
- Know the difference between a designer and engineer: Designers control the overall vision and make the decision of what’s right and wrong, while everyone else is usually focusing on creating a specific feature. As an engineer, especially a gameplay engineer, you greatly impact the design. Engineers try to understand what the high-level goals are and achieve them in efficient ways and communicate with designer and brainstorm to get “the best design for the right cost.” Duke specifically says, “As the engineering team it’s not our job to make the choice, it’s our job to make sure the designer and producers are making the most informed choice possible.”
Duke feels that engineers have the most impact on games compared to the other jobs. Unlike designers, Duke brings up that point that if an engineer thinks an idea would be great in the game, he can actually stay after work and code it himself and show it to the designer, whereas non-engineers have to convince someone else that it’s a good idea and implement a prototype. The only thing is that since he is not the designer, he cannot decide if that feature actually gets implemented into the game or not, but having the capability to just pitch prototyped ideas is extremely powerful.
- Gameplay engineers are always the last to work since they deal with everyone else’s work.
- Engineers need to make sure they identify the areas that have the most chance of having problems after they make changes, to isolate areas for QA to test on. You can’t just tell them, “Test the game” and expect for them to just know what and where to test or to just test the whole freaking game. There’s just too much content.
- All projects are archived, so they can be returned to if they are ever needed. They are great for reference.
- Over 80% of the people they review are engineers. They are always looking for engineers. At the moment the positions that are in most demand are frontend engineers (for Origin) and backend server database engineering (since they are beginning to invest in building a backend platform that ties everything they make).
There’s a pretty big discussion about this post on reddit now, so if you’re interested you can read through the comments there as some of them provide confirmation to this post as well as other insightful details about the AAA game industry. You can visit the reddit discussion at: http://www.reddit.com/r/gamedev/comments/rmafw/what_its_like_being_an_ea_engineer/