It has been quite a while since I have made a post here, but I have been keeping myself busy!
For the past 2 and a half years, I have been working with the team from Joy Everafter Stories (formerly Kathy Smart Game Design) as the principal programmer on Frog’s Princess, an interactive storybook based on a reimagining of the Grimm fairy tale of the same name.
I joined the project in mid 2018, when the game was still in a greybox stage of development.
Initially I was charged with making some modifications to the underlying Fungus dialogue system to improve the quality of life for the designers, and to fix some visual issues they were suffering, as well as to split the game up from one HUGE single Unity scene, into 12 chapter scenes.
We did this in order to save memory on device, but also to reduce the memory footprint in Unity’s editor, as it was slowing down to a crawl at times and hindering development. This was a larger task than it may initially appear.
The codebase that I inherited relied quite a bit on objects which were in scene in order to navigate around the book. To solve this I created a system to bake all of the scene data required by the navigation into an XML file, which the program could then load and use as required. The only downside to this system was that any major changes to the book structure would require a rebake of the menu data, so for this I created a dedicated scene and button for the designers to use to be able to do so.
The next task I worked on was the storyline system. Kathy required the book to jump around the chapters at certain points in the story for flashbacks, as well as skip over some pages that may not apply to the currently selected story tree. To solve this problem, we used a shorthand naming system for the Fungus flowchart blocks that the designers could edit, that navigation code would then check the block names for each suffix and treat the block accordingly.
I then moved on to the save game system, for this system I chose to use a static class. I felt this was a more robust design choice at the time. It kept the class out of the reaches of the unity editor, so modifications to scenes or MonoBehaviours would not affect it. It also eliminated the chance of save game data loss on scene change, and prevented multiple save games from being loaded into memory at one time. At the time each unity scene was designed to operate standalone for ease of development, and using a static class avoided the need to instantiate the class and perform a single instance check on scene change.
To date this has proven to be one of the most robust and well encapsulated classes in the game.
I was assigned the menu system next. My predecessor had started a menu system, but by the end of the project it had been all but completely redesigned. We leveraged the unity UI system for a lot of the menu functionality, only using code where absolutely necessary. While this did make the menus much harder to debug, it allowed the designers more freedom, and meant that for the most part we were leaning on tried and tested Unity code.
We used a simple interface class to go between the UI and the navigation controller, as per the code base I inherited, however given more time, I would have liked to refactor this into a full standalone UI class.
Later on in the project I would go on to add a swipe based touch interface to the UI for mobile device users.
Kathy was having some trouble at this time with the limitations of the audio system with regard to what Fungus would allow. She wanted more ambient channels, and a way for the users to be able to adjust individual volume to suit their preferences. I created a mixer in unity, and modified the Fungus codebase to provide 4 additional ambient sound channels to the designers. I later tied this into the user settings menu as per Kathy’s specifications.
At this point the game was essentially feature complete, and very close to a releasable state, however Kathy has an incredible passion for accessible games, and at this point it appeared that we had the budget and time available to pursue some additional features.
Kathy requested full character customisation. Primarily she wanted full colour customisation for diversity purposes, but also with the option for hairstyle changes if possible.
So as to avoid extending the project timeline, I initially tried to make a colour change shader, which would take the pixel colour of the sprite, and change it before rendering on screen. This proved difficult due to both the compression artefacts and situations where colours had been reused on different parts of the images.
After much discussion, and given that the game sprites were derived from rasterisation of vector source images, we decided that it could be worth checking out the Unity Vector Sprite package, which was in preview at the time.
After some testing, we discovered we could achieve better image quality, and successful colour changes at a lower file size with the Vector sprite add-on, and so we set about modifying it for our own purposes.
Being in preview, there were a number of efficiency issues and minor bugs to resolve, but I was able to extend the package into a system which could load an SVG file at runtime, switch on or off certain layers, such as hairstyles and expressions as required, and change colours in the image.
I then set about optimising the code as much as possible. Using LINQ i was able to restrict the number of SVG properties that the filter had to run through in turning the layers off dramatically, I also created a caching system which would only render each character image once, and then simply load from the cache wherever it needed to be reused.
This dropped the initial render time of each SVG on device from a number of seconds down to less than half a second each. With considered page design, this was short enough to be unnoticeable to the player.
As my final task, I was charged with implementing an interface for the visually impaired, and language localisation.
For blind accessibility, I researched and came across the UAP plugin for Unity on the asset store, it appeared to cover all of our functional requirements and we were able to implement it quite easily.
For localisation, Fungus has a robust localisation system built in, which we were able to leverage for dialogue.
Testing and Optimisation
We tested throughout development, both in editor and on device.
During development I worked on multiple optimisations to the game, some for performance, and some to comply with new platform restrictions. As a rule I tried to optimise during development where it was possible and did not hinder progress. Our primary optimisation targets were our audio and images.
For audio, we compressed the sound files using Ogg Vorbis at 70% while keeping audio quality good to the ear. We also set all audio to compressed in memory. While this did increase CPU overhead, for a book-style game, fast player reaction times and FPS are not a concern. This setting had no discernible impact on player experience, but reduced memory usage significantly.
For images, we used the SVG system as stated before for character images. For background images we found that atlasing halved our memory footprint (due to unity building all images as squares, we were able to atlas two backgrounds into one built image) and compressing the images at 50% quality gave us a good balance of size and quality.
Late in development we ran into an issue with Apple’s iOS 12. In this version Apple mandated a 1 GB hard limit on application RAM usage. At the time we were using additive scene loading between chapters, and on PC and Android we saw a maximum memory footprint of about 800 MB of RAM, but builds that previously worked without issue began hard crashing on iOS 12.
Upon further investigation, we learned that the garbage collection algorithm used in Unity is a non-generational, non-compacting type. In use this meant that if we were to load a 300 MB first chapter, then additive load a 400 MB second chapter, upon unloading the first chapter, the memory was not freed, and we were left with a 300 MB hole of empty RAM. Upon loading the third 400 MB chapter, the total RAM usage spiked up to 1.1 GB, even though only 700 MB contained any useful data, and our application would crash.
Once I became aware of this issue, I redesigned the chapter loading routines to clean up the current chapter before loading the next one.
Throughout the project I also created a number of Unity editor plugins for use by the designers, such as a QuickLabeller plugin to allow the designers to quickly apply unity labels to a number of items at once, and a MultiBuild system with which users can create and apply multiple build profiles easily.
I also created a screenshotter function to capture an image of each page, for use in the book navigation menu, and with over 900 pages in the game, this saved a huge amount of time for the designers.
We released the game simultaneously to the Apple App Store, Google Play Store, and Steam.
Google Play was the easiest. Due to my prior experience releasing on the Play store I had only minor issues. Our major learning point was around obb files, as the Play store has a size limit on apk’s and aab’s. We had to configure Unity to split the apk, and using aab’s was not an option.
Apple’s App Store was a little trickier, primarily because it was my first interaction with it. However, since we used TestFlight during development, when it came time to release the game we were fairly well familiar with it.
We also spent much of our time getting the game to display correctly on both the square 4:3 aspect ratio of an iPad, as well as the new ultrawide 13:6 aspect ratio found on the iPhone X.
Steam was by far probably the most difficult. It wasn’t that building for PC was inherently hard, although we did run into issues with UAP, as a required DLL was not being packaged with the build automatically, it was primarily the amount of technical information that the Steam store requires on your release.
As Kathy was the person setting up all of the store pages, I had to assist her with path information and minimum system requirements for the game.
I have learnt a lot from this project, and while I would consider it released, and therefore finished, I know that any changes to the app stores could call me back to support an updated release.
The technical nature of the Steam store’s submission process, and the constantly changing requirements of the App store and Play store mean that in order to keep the game live for more than a couple of years, a team really needs to keep a staff member well versed in the technical aspects of release on hand for the full expected life of the project.
With regard to the code side, I learned that just because Unity has a garbage collector, that does not mean you can take memory for granted. A C++ style approach to game design, where assets are loaded and unloaded hierarchically is essential, and this should be considered in your program design from the start.
It also became apparent just how important it is to set your major features in stone before the project begins. By attempting to add character customisation at the end of the project, the team had to recreate, and re-place in the game, all of the character art. When you consider the game had 900 pages, each with between 2 and 50 characters, that is a lot of work and significant financial cost.
It is also wise when inheriting an existing code base to look at the overall structure of the program and to see if any changes should be made ahead of time. Changing major game functions late in the piece can easily introduce bugs.
Finally, people don’t lie when they say multi-threading is difficult.
While Fungus does not use true multi-threading, it runs almost entirely on Unity coroutines. This makes it incredibly hard to debug, and almost impossible to determine order of operations. Often calling a Fungus block to stop would take 2 Unity game cycles to actually complete, and so it became necessary to create coroutines to watch other coroutines, to make sure that they finish before a new block is executed.
Unity also has an undocumented feature where child coroutines of a MonoBehaviour are destroyed when their parent script is destroyed, or on scene load. This was the cause of a number of scratched head moments during development.
Ultimately I am glad I worked on Frog’s Princess. It was a great learning experience with a wonderful dev team.
I’m eagerly looking forward to applying what I have learned on my next challenge!
Check out the Frog’s Princess website here!
Buy on Steam / Apple App Store / Google Play