About the Ingress Server - History and Plans
Hi, I’m @ofer2 and I’m an engineer on the Ingress team in LA. In an effort to be more transparent and open with the agent community about recent server issues and maintenance windows, I wanted to share a behind-the-scenes look to help clarify where things are today, and where we’re heading.
First, a little bit of history: Ingress started out as a small team within Google. We were originally an autonomous unit trying to answer the question, “What if Google Earth were a videogame?” The community shared fan art for game invites, captured portals seeded from the Historical Marker Database (HMdb.org), and the game began to grow. The game was built on Google’s infrastructure, and it’s what we refer to as our classic servers.
Later, Niantic spun out from Google as an independent startup. While Niantic transitioned to its next phase, our game team used lessons learned from Ingress and created new infrastructure for our second game: Pokemon GO. This new set of platform servers was designed to support even more players, and is both high performance and low cost. Meanwhile, Ingress continued to run on our classic servers.
Shortly after the successful launch of Pokemon GO, our Ingress team was tasked with rewriting the client because its foundational core was no longer supported. Like the classic servers, the original client was also designed in a way that made continued development more difficult and costly. On our server team, we made the decision to also move from the classic servers to our platform servers, and write new Ingress-specific features (like links and fields) on top of it. Unfortunately, both client and server transitions took much longer than expected, with the client being the most pressing need due to the risk of losing iOS support. .
Once Ingress Prime was released, the team began refocusing on fixing bugs and server updates. However, the new server still appeared to be roughly a year away from completion leading to a difficult question: Should we continue moving towards the new server at the risk of further disappointing the community and delaying new significant game features?
We decided to prioritize our community and the implementation of new features, resulting in a complete shift to finding cost reductions on the classic server rather than move onto the new architecture. This decision did still come with a hefty price. For example, Ingress cannot take complete advantage of awesome features that are automatically built-in for our other games utilizing the new server, so we must manually implement all of the missing functionality.
In order to reduce the costs of our classic servers, we chose to move our servers out of a high-cost compute environment into a low-cost one instead. Changing environments is like playing a game of spot the difference with those two pictures: we’ve fixed all the things we found, but with a picture large enough, changes small enough, and some changes mattering while other don't matter, it has been very difficult to anticipate what impacts will occur, which has led to various cases of testing and then rolling back features such as Intel. Intel was deployed into the new environment but the team discovered a central piece was needed first, resulting in the delay. We are optimistic that the central piece is very close but still need to hammer out some remaining issues.
All of the code for the changes have already been written and our team is currently focused on re-deploying the classic servers in this much more cost-friendly environment as well as evaluating the new impacts on the software side. We still need to test, fix bugs, and then publish. We will be starting our next round of testing on Tuesday. Once done, we will go back to testing Intel and eventually, the client servers. At that point, we will be running on a much lower cost for the classic servers which enables us to focus on features that will revitalize the game. New feature specs are already in the works that we hope the community will be excited for once it’s ready to be shared.
I hope this update has provided more clarity into the decision-making of our team and clearer transparency into how we’re actively shaping a new future for Ingress.