Update on Game Latency, Missing Wayspots, and the Roadmap

1567911

Comments

  •  (especially since we've asked maintenance to be scheduled at night and not during the day)

    One question: Does this mean that whoever's day time is your night time would then see that lag instead?

  • liyu1liyu1 ✭✭✭

    really annoying when ur full recharging an important portal thats under attack and due to lagg and then suddenly destroyed. (happened to me more then once.)

  • Kevinsky86Kevinsky86 ✭✭✭
    edited October 2022

    Still though. I barely used that button in OG Ingress and even Prime predating that move to cheaper compute space, this issue didn't occur nearly as much as it does now.

    Not that it matters, but I remain convinced this is a capacity issue on some level. The lag problem is at least as old as C.O.R.E. is and that celebrates it's second birthday in three months.

    As much as I appreciate the update, I remain highly skeptical solving this problem is actually on the calendar with any kind of priority.

    Post edited by Kevinsky86 on
  • Thanks for the update; these sorts of communications from the teams are really appreciated, even if it's still "we're working on it." It really sucks to hear nothing but radio silence on all fronts when there are many known issues.

  • SSSputnikSSSputnik ✭✭✭✭✭
    edited October 2022

    @ofer2 Very much like the update. Could we get a monthly update or even every two months, even if nothings been done it reassures us you care.

    You have always been really good at providing insight into what's going on and it's appreciated.

  • HosetteHosette ✭✭✭✭✭

    @Kevinsky86 I suspect it's some combination of a capacity issue and a design/architecture issue.

    I do know that there are things that used to work well for me at some time in the past that are constantly problematic right now. The sequence of events I encounter most frequently is:

    1. Single-reso capture a portal
    2. Try to deploy a mod

    More often than not this fails with an error about the portal being neutral, but then succeeds on the second try. I used to be able to do this pretty reliably.

  • CrownanCrownan ✭✭✭

    I get around 600-700 "RetriesError" on average when I play normally trying too deploy resonators. Ongoing for over a year. Deploying mods sometimes I get 30-60 seconds before it deploys. Recharging has been much smoother lately.

  • Jo0LzJo0Lz ✭✭✭✭✭

    I understand that it's complex, but I fail to see how that explains that it is now taking two years+ to fix. There have been so many people willing to help, run with debug builds, beta test, etc.

    The biggest issue is the fact that communication is non existent. We get a topic to post our 'logs' in, but never get any feedback, the topic dies out. We get a tentative fix, that didn't work, and it's not mentioned anywhere, ever. Even this topic is one year old in November, and nothing noteworthy up until you appeared (thanks again, I can't stress enough how much of a boost this is to morale, to have someone take the time and write something of a reply).

    I'm eagerly awaiting the fix, but you (niantic) have to see yourself that it's all a little too late, as most of the players have lost faith that it will ever be fixed and have moved on to other games.

  • HosetteHosette ✭✭✭✭✭

    @ofer2 Thank you for the clarifications.

    I actually do grok how complicated things like this can be... and I'm also aware that a single problem perceived by users is actually multiple complex issues under the hood, and sometimes compound issues. I'd say that I don't envy you the task of solving this but I sort of do... this is exactly the sort of problem that I love the most because it involves a lot of deep-diving and complex troubleshooting, and it's incredibly rewarding to finally solve the problem.

  • For got paid the subscription NIA don't have latency. There's not problem. Players have problems, scanner don't work, latency, bugs, fakes, multiaccount are not problem because NIA don't have problem with got paid subscriptions.

  • JoDoGo42JoDoGo42 ✭✭
    edited October 2022

    When i read this i really wonder how the 'old' ingress was able to perform at all 😂

    In old days it worked on 256/512kB mobiles, anomalies with many thousands battling in single or very few mobile cells.

    Not that there werent any update errors or even eventual app crashes under these heavy load conditions but comparably rare esp in comparison to nowadays just normal gameplay with just very few agents at once even in big cities.

    Mobile networks, servers and esp mobile phones massively increased their speeds, available ressources and sensors precisions not just in percentages but by remarkable factors.

    Guys, its really a shame what you roll out as enduser software and which stale software concept now hides on servers end to massively slow down ingress gameplay and fup updating in esp battle situation means.

    I would really love to see big anomalies taking place again but i understand its completely impossible with the current setup and system design in general.

    Thats really sad..

  • SSSputnikSSSputnik ✭✭✭✭✭

    Hrm running redacted at a large anomaly you had plenty of lag. Just keep pressing buttons n hope something happened.

  • Turns out that some of our fixes (and when I say our, I mean Ingress coordinating with a central team as we dont generally work on this shared code) didnt fix it. So, we then thought that it might be how the server receiving the request and processing it, only for the client to disconnect before the server could send the response back. After that, we thought it might be that the client is retrying like 10 times and the server is doing a lot of work figuring out that they're duplicate requests.

    Shouldn't the client requests be tagged with some sort of session ID+Request Counter+Attempt Counter, so that when resends are done, it can tell "Oh I already did that Request, this is just a retry" without any real calculation?

  • EvilSuperHerosEvilSuperHeros ✭✭✭✭✭

    That's kinda how it works right now though...years later, with much faster mobile phones, on much faster networks.

  • EvilSuperHerosEvilSuperHeros ✭✭✭✭✭

    Thanks for somewhat of an answer. Hopefully someone can keep us updated at least somewhat going forward. Hearing nothing about a big problem in the game is crappy.

  • EvilSuperHerosEvilSuperHeros ✭✭✭✭✭

    Force sync should fix some issues. It would remind the server to speak to the client me thinks.

  • mortuusmortuus ✭✭✭✭✭

    It doesnt sound they are any closer then 1 year ago isolate the issue and solve it finally? its been so long now i dont know really what they need do to properly fix it. But its nice to get an update atleast even if not the best news we hoped for.

  • ofer2ofer2 ✭✭✭✭✭

    Non technical update: we have a fix that we're trying out, were hopeful that itll work, but we have a couple ideas to try out if it doesnt. Fix should be live either today or tomorrow.


    Technical update: so, we have a new theory for what is happening. So, for some context first: when we hear "latency is high" there isnt a smoking gun that we can look backwards from because we cant reliably reproduce the issue. The issue primarily happens at scale and when there is load. Therefore, we use a different strategy which is to look at the data. We have data which samples requests randomly and then gives us some information about the request. We can filter these sampled requests by how long they took. This process essentially gives us a list of "top 10 sources of latency." That's primarily what we've been looking into to try and make things faster (this is where we've looked into retry stuff, and other various ideas we've had). So, we've done a bunch of work here, reducing the latency of things that we've found. After continuing to look, we've discovered that theres a hole in our data: our infrastructure is designed to scale up / down when the load changes. Google seems to be assigning requests to servers that have just been scaled up, but arent actually ready yet. Although the request isnt "delivered" to the machine, because it isnt ready, it is assigned to the machine and just waits. We're talking to Google about this and we have a workaround that we're trying, but we arent sure that it will fix it. Since this is a weird startup thing which only happens under a lot of load, its hard to test internally. We've done our best, but the final test is what happens with real players. So, we're trying out the current fix, but if that doesnt work we have a few more ideas.


    All of that being said, we know that its a long journey in fixing this, so thanks for bearing with us.

  • ofer2ofer2 ✭✭✭✭✭

    Technical update: so, we have a new theory for what is happening. So, for some context first: when we hear "latency is high" there isnt a smoking gun that we can look backwards from because we cant reliably reproduce the issue. The issue primarily happens at scale and when there is load. Therefore, we use a different strategy which is to look at the data. We have data which samples requests randomly and then gives us some information about the request. We can filter these sampled requests by how long they took. This process essentially gives us a list of "top 10 sources of latency." That's primarily what we've been looking into to try and make things faster (this is where we've looked into retry stuff, and other various ideas we've had). So, we've done a bunch of work here, reducing the latency of things that we've found.

  • fxcfxc ✭✭✭

    @ofer2 When will you start dealing with the problem of multi-accounts and who uses gps spoofing?

  • Excellent update @ofer2. This is the kind of progress update we need 🎉

Sign In or Register to comment.