About the Ingress Server - History and Plans

ofer2ofer2 ✭✭✭
edited February 14 in Ingress Insights

Hi, I’m @ofer2 and I’m an engineer on the Ingress team in LA. In an effort to be more transparent and open with the agent community about recent server issues and maintenance windows, I wanted to share a behind-the-scenes look to help clarify where things are today, and where we’re heading. 

 

First, a little bit of history: Ingress started out as a small team within Google. We were originally an autonomous unit trying to answer the question, “What if Google Earth were a videogame?” The community shared fan art for game invites, captured portals seeded from the Historical Marker Database (HMdb.org), and the game began to grow. The game was built on Google’s infrastructure, and it’s what we refer to as our classic servers. 

Later, Niantic spun out from Google as an independent startup. While Niantic transitioned to its next phase, our game team used lessons learned from Ingress and created new infrastructure for our second game: Pokemon GO. This new set of platform servers was designed to support even more players, and is both high performance and low cost. Meanwhile, Ingress continued to run on our classic servers.

Shortly after the successful launch of Pokemon GO, our Ingress team was tasked with rewriting the client because its foundational core was no longer supported. Like the classic servers, the original client was also designed in a way that made continued development more difficult and costly. On our server team, we made the decision to also move from the classic servers to our platform servers, and write new Ingress-specific features (like links and fields) on top of it. Unfortunately, both client and server transitions took much longer than expected, with the client being the most pressing need due to the risk of losing iOS support. .

Once Ingress Prime was released, the team began refocusing on fixing bugs and server updates. However, the new server still appeared to be roughly a year away from completion leading to a difficult question: Should we continue moving towards the new server at the risk of further disappointing the community and delaying new significant game features?

We decided to prioritize our community and the implementation of new features, resulting in a complete shift to finding cost reductions on the classic server rather than move onto the new architecture. This decision did still come with a hefty price. For example, Ingress cannot take complete advantage of awesome features that are automatically built-in for our other games utilizing the new server, so we must manually implement all of the missing functionality. 

In order to reduce the costs of our classic servers, we chose to move our servers out of a high-cost compute environment into a low-cost one instead. Changing environments is like playing a game of spot the difference with those two pictures: we’ve fixed all the things we found, but with a picture large enough, changes small enough, and some changes mattering while other don't matter, it has been very difficult to anticipate what impacts will occur, which has led to various cases of testing and then rolling back features such as Intel. Intel was deployed into the new environment but the team discovered a central piece was needed first, resulting in the delay. We are optimistic that the central piece is very close but still need to hammer out some remaining issues.

All of the code for the changes have already been written and our team is currently focused on re-deploying the classic servers in this much more cost-friendly environment as well as evaluating the new impacts on the software side. We still need to test, fix bugs, and then publish. We will be starting our next round of testing on Tuesday. Once done, we will go back to testing Intel and eventually, the client servers. At that point, we will be running on a much lower cost for the classic servers which enables us to focus on features that will revitalize the game. New feature specs are already in the works that we hope the community will be excited for once it’s ready to be shared. 

I hope this update has provided more clarity into the decision-making of our team and clearer transparency into how we’re actively shaping a new future for Ingress. 

 

Post edited by ofer2 on
Tagged:

Comments

  • Thanks again for the transparency. So much better than players (with varying levels of actual knowledge) trying to read the tea leaves of Niantic's intent.


    Thank you.

  • This kind of information is great. Thanks

  • Thank you so much! Little posts like these go a long way, belive me. Working on a product is hard, even more so if you can't really talk about future plans without the community ripping into you constantly asking "what about X"

    Speaking of... ;) You wrote that the development of new features instead of switching to the new server was prioritized. I might read a little too much in between the lines, but the way you wrote it, it seems like the classic servers are here to stay for a bit longer, even though it seems harder to code stuff around it. Out of curiosity, at which point would you personally consider the move to the new server imperative? (And if there is a rough time plan, when? ;))

  • So, that was the news information about the history and plans of Ingress Server!


    Thanks and Happy Valentine's day!♥️♥️

  • This transparency is greatly appreciated. I'm excited about the new features and hope that it's not too rough of a process for y'all haha

  • tehstonetehstone ✭✭✭

    Like the classic servers, the original client was also designed in a way that made continued development more difficult and costly.

    Palms are sweaty

    Knees weak, tech debt heavy

    No comments on my code already

    Hanke's spaghetti




    Seriously though, thanks for this write up! I hope it's not the last of it's kind, and I'm sure many of us would appreciate deeper dives into what goes on behind the scenes!

  • Big thanks for sharing!

  • As far as I know, the 'classic' server methods are basically using the google cloud data servers. So... does this mean we'll be moving to the slower servers now that it's an external entity? Will the game start slowing down because of these cost saving changes?


    Can we ask how the cost numbers are looking? Revenue is basically CMU conversions and that's about it, right? How are anomalies faring as far as revenue vs cost? Is that why we're down to just one 'big party which is a net loss but engages players'?

  • GrogyanGrogyan ✭✭✭✭
    edited February 15

    @ofer2 thanks for the update, looking forward to new features and game mechanics.


    But I can't help thinking, that working with the old, and VERY slow for what is actually needed, is the right approach.


    Could you please say if this current implementation is going just an intermediate step to the higher performing NGP (Prime Server)?

    Post edited by Grogyan on
  • Thank you so much for providing this insight towards an otherwise black box of information regarding the development of this cherished game.


    Sidenote/question: Any timeline for when the game's OSM overlay to be updated? The most recent one is dated from 17 December of 2017 and quite a few improvements/road updates have been implemented since and a regularly scheduled update to the overlay would drastically improve game quality.

  • GrogyanGrogyan ✭✭✭✭
    edited February 15

    My opinion is the maps used for calculating scores NEED updating first.

  • Thanks for all the work, we enjoy ingress.


  • @ofer2 this post was amazing THANK YOU for providing this feedback for us, players. Sometimes (most of it) we really just think Niantic is decommissioning Ingress, because how slow things move.

    But with that letter we can have a backstage view and feel Niantic is yet caring about us, their early players.

    I really think that even this upgrade challenge can become a HUGE case of success by Niantic. Really, and then you guys can present how you did it. Me, as also a software engineer, would be very curious to know how you guys made it.

    Now, talking about present, I would like to address this to all ingress team: I know now its not easy, but I know you guys are doing your best, so THANK YOU!!! I believe that you guys can handle this!

    And finally, lets hope this Server Change doesn’t take so long, because if there is too many features tangled with legacy infra, it will be hard to decouple, A LOT of re-work. So besides only hope, lets also try to build in a way aiming for that future upgrade.

    CHEERS for a prosperous future 🍻!

  • While I appreciate lip service as much as every other paying customer...

    Can you finish fixing what you broke before you go writing lengthy fanmail?

    My account has been unplayable for 3 days. Inventory is locked behind a known server issue.

    I responded to Support's request for the error code in my scanner 48 hours ago.

    NIA started a shiny new fielding event that I can't even participate in.

    Bad move, trying to engineer-speak your way outta the mess that's been made.

  • herobrine0447herobrine0447 ✭✭
    edited February 15

    You guys doing an amazing job, but you still issuing heavy bugs.


    Firstly, some services goes off and makes Ingress unplayable.

    Second, account issues. My actual IGN is EMRE0YUN but forum shows wrong account name due email.

    Thirdly(also connected with 2nd issue), some services not lets in due account issues(Facebook Login i mean)

    Forthly, support team is bit slow. They answers us too lately.


    Otherwise, Niantic is success project with Ingress, Pokémon GO and Harry Potter Wizards Unite. Keep it up, we are always supporting you.

  • GuschtelGuschtel ✭✭✭

    @ofer2 Thank you so much for openly talking to the community. It's much appreciated.

  • This is a very nice share, thank you. A part from the server cost, how is the scalability of the backend now? I mean, for what I understand, PoGo, Harry Potter and Ingress now rely on very similar (yet not identical) environments. Will there be major updates in the foreseable future for the platform?

  • why isn't your name @Nianticofrr2 ? 😆

  • Thanks for the insight.

  • Finally some transparency!


    Great update from you guys. We were expecting transparency from you guys from day one. It is way less frustrating if you at least know what is going ond why this decision has been made that way. I am sure that things are moving forward (sometimes slower than expected) but it is getting into better. As I can imagine you have migrated all server side into container based environment (would be most logical thing to do to reduce hosting overhead and costs for it too). If I can throw a tiny idea: If Niantic events would be scheduled to take turns in different games (Ingress, PoGO, HPWU) and they would not affect other game's events. Reason being that Ingress event is clashing with HPWU and PoGO sometimes which makes it more challenging to achieve best results on either of them. Just to clarify, there is significant user base playing 2 out of 3 or even 3 games at the same time and we got only limited ammount of time available for activities. Scheduling not to overlap events would make it lighter for the load balance of DB server (as assuming it is using same db for all 4 games (Field trip, Ingress, PoGO and HPWU). In big events of Ingress (Anomalies, Mission days and other events) quite often you can feel it becomes "laggy" on the client site for users regardless of which game you are playing. Hope that helps. If there is needed for some insight and on field data I am more than happy to provide that in future. Feel free to get in touch if you would like me to provide some help and other insights.


    One really important thing, we are here to help and I believe it is many of us!! :)


    Keep it going guys! We all love the game!

  • I can’t login right now. Hard to accomplish a live event like that

  • Om3gaMinusOm3gaMinus ✭✭
    edited February 19

    😘

    Post edited by Om3gaMinus on
Sign In or Register to comment.