Are we going to get a heads up when it’s actually live?
I’ve seen a lot of latency the past few days, delayed responses when linking, recharging and hacking. Would be nice to know when it’s live.
in any case deploying resonators was painfully slow this morning and blazing fast after 13:00 cet, I don’t know what causes the change…
I don’t think that falls under their responsibility. That would be more for @NianticBrian or @NianticThia
@ofer2 awesome, Thanks for the technical update, very fascinating.
My question is what you mean by "heavy load"?
Because for suburban and rural players, the data traffic is extremely low, and this fix doesn't sound like it will solve some of the issues in these areas.
Where Ingress stops responding, or portals not loading when other portals in the vicinity have loaded in the scanner, recharging hangs etc
wether it works or not, this kind of update are just priceless. Thank you, i really hope it works tho! So excited! :)
@Grogyan Presumably ofer2 means server load (total traffic to their servers) not the load from individual users. If you're waiting on the server you're waiting regardless of whether you're in the middle of Manhattan, in suburbia, or at the only portal within a 10km radius.
Back during Shonin Anomaly, NZ shared the same server as Japan, which had so much lag, Niantic disabled the Intel server.
I don't believe New Zealand uses the same server anymore, could be wrong.
Never cross the streams!!!
(Thanks for the fix! 💚💙💚)
@Grogyan I'm quite certain there's not a "the server" for a region, but rather a large cluster of them. And I don't know if they have regional server farms or one global one... I can see arguments for both architectures.
The broader point is that if a server is non-responsive then it will be non-responsive to everyone who is making requests to it regardless of whether that player is in a high-density area or a low-density one. It's certainly possible that someone in a high-density area could be more impacted for some requests but that doesn't seem like it would matter for a hack or a deploy.
Thanks for the update again @ofer2 - Keeping eye on lag.
would be sweet have no lag for year 10 double ap event so thanks ofer2.
Based on my limited experience, and from what our own DevOps engineers has explained to me, it sounds as if your new pods would be accepting connections before Terraform (or a similar tool) had run to completion? Shouldn't that be easily fixed by changing the settings to wait for Terraform to complete before accepting external connections?
OK - so - lag may be improved, but still exists, albeit rarer? I've not played enough to get a 100% solid feel but some stuff seems better so far...
Was the fix 'implemented'? We haven't had an update on it yet. Nice to know if that activity was completed.
@FireFight001 That is one of the first things that should be checked when there are startup issues, but it is often much more complicated than that. There can be cases where a host is up and running but isn't ready to serve traffic yet, or isn't ready to serve traffic at scale. I've seen applications that looked like they were ready to accept connections but they were still going through an internal startup sequence. I've seen cases where the application was online but it had to go through an internal warmup sequence before it could serve traffic at scale, and until that was done it worked very slowly and/or dropped traffic on the floor. I've seen really stupid systems that considered a host to be online as soon as the OS booted up and was responsive, without considering that the application had to start too. I've seen systems that assumed that the application would be online in X time and then sometime later (usually after the original author had left the team) something changed and X was no longer long enough, but nobody currently on the project knew about time X. Startup sequences and health checks seem like they should be easy but there are lots of nuances and lots of ways to get them wrong... often in ways that won't become obvious until sometime n the future.
I've been doing production operations for internet companies on and off since 1996, ranging from small startups to one of the best-known companies in the world. I've seen a whole lot of st00pid, but I've also seen a lot of really subtle behavior that was quite hard to diagnose. The one thing I can say for certain is that systems will always find new and interesting ways to fail. These sorts of failures are what keep SRE/DevOps people employed.
 Host, instance, server, pod, etc... for this discussion I'm using host as a catch-all term rather than a term of art.
 Which is absolute lunacy, and whoever was responsible for this particular case should have been smacked upside the head with a rotten trout. (-:
I had bad lagg yesterday afternoon around 5 pm cdt. It got so bad I stopped playing in the park I was in. I ended up meeting a friend at another park, and oddly the lagg wasn't present at that time (about 45 minutes after the previous bad lagg).
Yeah i think its same. But seem to have less recharge lag?
Hey everyone, so for an update: the first fix did have an effect though not as large as we would like to have seen. We saw the worst case latencies reduced by 25%. We deployed another fix yesterday which we'll continue to monitor and tune, but so far it looks like that has done a better job. Also of note: this does not fix the issue that latency will increase during the time in which we deploy new servers (which is what I believe you saw yesterday at 3ish). Please let us know how it feels to you now, but we'll also be watching the data.
thanks for update ofer2 bless you ❤️
What time is 3ish in UTC?
Niantic Time, IE -8 UTC Around 2-4 PM PST (or 4-6 PM CDT ie my time) is when I see the highest amount of time where trying to play the game is utterly useless. Which leads me to believe this is when they are testing cr4p.
@ofer2 thanks for your insights and how the new fix is going.
My personal experience so far is that, some of the lag is noticeably reduced.
There has been a side effect, normally, you could deploy a resonator and hack a portal/or link on a bus as it goes by. There is some lag here when the second action is done before leaving the action circle, just as the bus leaves the action circle that second action is processed a fraction of a second too late, and doesn't go through
This has been exactly my experience.
@ofer2 are you able to check my account? Anomaly shenanigans seems to have duplicated my common kinetic and both definitely functional. Happened like 3:45pmish PST
1D60AEFF is my original
A0F0DBA0 spawned in my inventory after a few errors with a 3 use rare one
@Nysyr We know that this has happened to a few players. We are in the process of creating a flow to go through and fix all players. Sorry about that.
Ahh OK - I have a duped common Kinetic as well.
Ingress has been unplayable lately because of lag. Stuck deploying resonators and mods. Force restarts to black screens. Available links don’t appear in the carousel. Hacks hang.
I love how Niantic say (as per Niantic official on Reddit) “we want new players to stick around” and want to focus on keeping these players..So, they add a bunch of garbage features, yet still wonder why new players don’t persevere! It’s because the long-standing GAME BREAKING issue of latency. Why would any new player stick around?
I really want things to get better because I use ingress as motivation to get out and walk, and ride my bike. But it’s not really good for exercise now due to the amount of time you have to spend at portals because of all the lag, restarting, cancelling out of link carousel to try to bring up all the valid links. The amount of time spent at portals defeats the purpose of exercising because the heart rate is already back to resting by the time you’ve done what you need to do.
ill be cancelling my subscription until things improve, and I will be disputing the past months subscription fees under “goods/services not received” clause, as the game is broken and you simply can’t provide the services we are paying for.
This same lag is in every Niantic game.
Problem is more complicated, and as Ofer2 said he and the rest of the devs would rather work on new content, but have spent a, lot, of time already.
They did implement a work around for one issue that was causing lag recently.
We are still waiting for other fixes to be done
Its awful when its less then 5c out and u have constant lag daily on deploy resonators, when it takes sometimes 1minute to fully deploy one portal its just makes u dont wanna play alot...
If niantic is serious to grow the userbase they really need to fix the current lag and not add useless features that doesnt help new or old players, they cant say for years "we are still investigating and trying" that just doesnt work, if the game is broken and u cant fix it atleast admit it.... dont give false hope and then wonder why userbase is standing still....
Just played 20 minutes and I waited for what it felt for 5 minutes of that. Deploying lag, “command channel” glyph lag, and link lag.
I don’t know what “fixed several soft locks and issues reported at the Epiphany Dawn LA Anomaly” means @ofer2 @nianticOfer but it has in no way improved on the issues we are facing in daily usage.
So, the server side fixes and client patches have had exactly zero effect, the game experience is still sub-par, and it’s about time this is actually fixed. The technical updates are nice, but what does it all add/mean when the game doesn’t work…
Last update break looking random profiles for some reason i made bug report of it. why each new update always breaks something?
and yes server lag is still present☹️