Prime always has issues loading, freezing ect fell sorry for all the first saturday people
I lost a capsule quantum one with 99 items in the thing why remove redacted when we have less issues with it only to replace it with prime which is not working very well
This needs to fixed by the anomaly week end or it will be a great peace of PR of niantic
I had a feeling that Redacted was not a game. It felt like a real agent tool, a scanner for field activities in a world that hid more than it shown.
Prime on the other hand, so shiny and animated looks completely as a kid's game. Big buttons for clumsy fingers and all. Lots of pictures. Animations. Sounds. Fancy graphics... Redacted was not a game.
Prime follows the other two kid's games style. It only missed the grass on the map... But don't let that fool you.
And I can imagine a Prime release with AR camera that lets you take a selfie with a bag on your head xD
Yeah, I knew that without testing anything... Just thought of the mbs used to download every comm picture (with lots of actions around). And the maps in real time. And nearby portals on a longer range than Redacted. And doing that all the time, live. And the portals images to show the ghost picture on the map. I guess some of the data gets refreshed every some minutes so maybe the cache is not important...
Better test that with cache and storage cleaned so they both start from scratch. You may need 1gbit lan connection just for prime xD Sadly Redacted is not available for the rest anymore.
Let's put all of those requests in the middle of an anomaly and get some popcorn!
I have already made a comment about Field Test (over 200 Agents on site), but seen no lesson learnt. FS was even smaller and the same thing happened, plus we do not have Redacted as a backup. If the same thing will happen on Anomaly, which we have paid for, expect refund demands for all expences.
Even GKE afaik, but there is no information available. I would say it depends on the design of the backend services and the resulting bottlenecks for the data backend
Btw, it would be awesome, @NianticBrian, if you had a technical blog like the guys from Eve Online do it - a little insight in your used technologies, your clustering, dealing with outages, site reliability, things like that.
something is not ok with both backend and frontend "optimisations". and with client optimisations. and with communication with players. and i dearsay -- we (players) feel - something is not okay with NIA itself.
I just want to say I attended my first fs in Bradford and was extremely disappointed, I was having severe server lag after 12 o’clock where placing res’s and destroy stuff just caused long loads of sweet fa..hopefully my second fs is more successful
You don't have to complete every mission day mission to get the mission day badge. Instead of doing 18 or 12 do 14 or 8 instead and your rows will be even again.
Alternatively @TriquetraX can do the mission day in a set of six then re-do the 2 missions and complete the 6 for First Saturday to realign everything.
There was a Hacker Newscomment at the time in response to that blog post that said
Yes, an example from the article showed that Niantic was running in a Kubernetes cluster that could only scale to ~1k nodes. So even if the resources were provided they couldn't add those nodes to their cluster.
You could argue that Google was providing GCE and therefore GCE couldn't scale, but GCE is really just hosted Kubernetes and it's scaling limits were known in advance. Luckily GCP was able to push a quick version update and migrate the cluster but that took considerable time, coordination and engineering effort that couldn't be done "seamlessly".
This is a clear situation where the software Niantic chose couldn't handle the resources that were available. It turns out a lot of the choices you made for your worst case capacity aren't necessarily adequate for 50x that amount :)
From a technical architectural standpoint, the major transition that we made was from App Engine, which was a previous generation of Google's Cloud computing products, to Google Container Engine, which was something that only came out a few years after Ingress had started. And so we did take the opportunity to modernize some of our technical infrastructure.
There was mention last year that there was to be a backend rewrite. I think they were waiting to retire Redacted prior to that rewrite being deployed (nobody knows how far down the track they've gotten). And that rewrite would mean a move from App Engine to GKE.
I would thought after redacteds **** that prime would be more stable due to many users stopped playing after that... a few days there is anomaly and my game still freezes atleast 2 times within 1 hour of gameplay.. how is this a good sign ? idont get it.. when is the new backend server coming online? last FS and field test showed it cant handle the load when its at peak....
listening to podcast that @wcpl mentioned - i got slightly worried. i may be no "professional", but i feel, that something does not quite scale out that well. and hearing what pogo player tell (about smoothness)...
Thanks. I show the graphic of the GKE Team for the Pogo release to every customers when it comes to design the infrastructure for the initial launch. A lot of lessons can be learned and I am glad the GKE team made the post.
Nevertheless I am missing that from the Niantic Developer /Platform / SRE Team. The AMA Section has the subtitle "The place for monthly AMAs, Developer Insights, and other similar activities." so I would love to hear some stories from the dev side.
Please find below a translation in French. Hope this helps.
"Suite aux conclusions des Field Tests Nia, nous avons re-dimentionné les serveurs pour encaisser la demande. Aujourd'hui, dès les premières heures du jour (fuseau horaire PDT : Heure de Californie UTC-7), nous avons remarqué un accroissement notable de la latence, et des problèmes de connexions à Ingress. Nous avons à nouveau re-dimentionné les serveurs, ce qui a permis d'obtenir une latence nominale au bout de quelques heures. Je tiens à préciser qu'il s'agit d'un problème côté serveurs.
Concernant les anomalies de samedi 12 octobre 2019, les données récoltées durant la précédente anomalie serviront de base et nous en tiendrons compte par rapport à l'actuel état des inscriptions aux événements, avec une marge supplémentaire."
I was also at home in the UK on a fast Wi-Fi connection, and at about 2pm I had to give up on my daily recharge because every press of the recharge button and every use of a power cube was taking up to a minute to resolve. It was better some time later in the day.
Comments
Prime always has issues loading, freezing ect fell sorry for all the first saturday people
I lost a capsule quantum one with 99 items in the thing why remove redacted when we have less issues with it only to replace it with prime which is not working very well
This needs to fixed by the anomaly week end or it will be a great peace of PR of niantic
I had a feeling that Redacted was not a game. It felt like a real agent tool, a scanner for field activities in a world that hid more than it shown.
Prime on the other hand, so shiny and animated looks completely as a kid's game. Big buttons for clumsy fingers and all. Lots of pictures. Animations. Sounds. Fancy graphics... Redacted was not a game.
Prime follows the other two kid's games style. It only missed the grass on the map... But don't let that fool you.
And I can imagine a Prime release with AR camera that lets you take a selfie with a bag on your head xD
Yeah, I knew that without testing anything... Just thought of the mbs used to download every comm picture (with lots of actions around). And the maps in real time. And nearby portals on a longer range than Redacted. And doing that all the time, live. And the portals images to show the ghost picture on the map. I guess some of the data gets refreshed every some minutes so maybe the cache is not important...
Better test that with cache and storage cleaned so they both start from scratch. You may need 1gbit lan connection just for prime xD Sadly Redacted is not available for the rest anymore.
Let's put all of those requests in the middle of an anomaly and get some popcorn!
What did you actually do a field test for? To then make the same mistakes again with a FS?
For the FS you also had the number of participants before and could have reacted "before".
And for the next anomaly you want to use the data of the last one? Where Redacted was still running and there were still problems with Prime?
Wow there I am really curious....
I hope you will remember that at a paid event people will react differently if the performance is not right again....
I have already made a comment about Field Test (over 200 Agents on site), but seen no lesson learnt. FS was even smaller and the same thing happened, plus we do not have Redacted as a backup. If the same thing will happen on Anomaly, which we have paid for, expect refund demands for all expences.
I'm concerned that even after scaling the servers up for a second time, it still took several hours for the latency to return to normal
Even GKE afaik, but there is no information available. I would say it depends on the design of the backend services and the resulting bottlenecks for the data backend
Btw, it would be awesome, @NianticBrian, if you had a technical blog like the guys from Eve Online do it - a little insight in your used technologies, your clustering, dealing with outages, site reliability, things like that.
something is not ok with both backend and frontend "optimisations". and with client optimisations. and with communication with players. and i dearsay -- we (players) feel - something is not okay with NIA itself.
I wonder if Niantic asked Remy for support during yesterday's mess.
If they did they would still be waiting for a "human" to close the ticket without doing anything. Especially if it were a spoofer report...
Note to self: Don't use Ingress on 12th of October
How about LOL!!! Keep that prime thing up and Keep loosing players.
I just want to say I attended my first fs in Bradford and was extremely disappointed, I was having severe server lag after 12 o’clock where placing res’s and destroy stuff just caused long loads of sweet fa..hopefully my second fs is more successful
That would assume that they either monitor their game or play themselves...
You don't have to complete every mission day mission to get the mission day badge. Instead of doing 18 or 12 do 14 or 8 instead and your rows will be even again.
Alternatively @TriquetraX can do the mission day in a set of six then re-do the 2 missions and complete the 6 for First Saturday to realign everything.
That too
There was a blog post on the Google Cloud blog about the launch of Pogo. https://cloud.google.com/blog/products/gcp/bringing-pokemon-go-to-life-on-google-cloud
There was a Hacker News comment at the time in response to that blog post that said
Yes, an example from the article showed that Niantic was running in a Kubernetes cluster that could only scale to ~1k nodes. So even if the resources were provided they couldn't add those nodes to their cluster.
You could argue that Google was providing GCE and therefore GCE couldn't scale, but GCE is really just hosted Kubernetes and it's scaling limits were known in advance. Luckily GCP was able to push a quick version update and migrate the cluster but that took considerable time, coordination and engineering effort that couldn't be done "seamlessly".
This is a clear situation where the software Niantic chose couldn't handle the resources that were available. It turns out a lot of the choices you made for your worst case capacity aren't necessarily adequate for 50x that amount :)
(disclaimer I work at Google but not on GCP)
There's also this podcast/chat with the Director of Software Engineering at Niantic (as part of Google Cloud Platform podcast) about the Pogo launch (timestamp about talk on the launch), which expands on the above quoted comment.
He actually mentions
From a technical architectural standpoint, the major transition that we made was from App Engine, which was a previous generation of Google's Cloud computing products, to Google Container Engine, which was something that only came out a few years after Ingress had started. And so we did take the opportunity to modernize some of our technical infrastructure.
There was mention last year that there was to be a backend rewrite. I think they were waiting to retire Redacted prior to that rewrite being deployed (nobody knows how far down the track they've gotten). And that rewrite would mean a move from App Engine to GKE.
I would thought after redacteds **** that prime would be more stable due to many users stopped playing after that... a few days there is anomaly and my game still freezes atleast 2 times within 1 hour of gameplay.. how is this a good sign ? idont get it.. when is the new backend server coming online? last FS and field test showed it cant handle the load when its at peak....
listening to podcast that @wcpl mentioned - i got slightly worried. i may be no "professional", but i feel, that something does not quite scale out that well. and hearing what pogo player tell (about smoothness)...
Thanks. I show the graphic of the GKE Team for the Pogo release to every customers when it comes to design the infrastructure for the initial launch. A lot of lessons can be learned and I am glad the GKE team made the post.
Nevertheless I am missing that from the Niantic Developer /Platform / SRE Team. The AMA Section has the subtitle "The place for monthly AMAs, Developer Insights, and other similar activities." so I would love to hear some stories from the dev side.
Please find below a translation in French. Hope this helps.
"Suite aux conclusions des Field Tests Nia, nous avons re-dimentionné les serveurs pour encaisser la demande. Aujourd'hui, dès les premières heures du jour (fuseau horaire PDT : Heure de Californie UTC-7), nous avons remarqué un accroissement notable de la latence, et des problèmes de connexions à Ingress. Nous avons à nouveau re-dimentionné les serveurs, ce qui a permis d'obtenir une latence nominale au bout de quelques heures. Je tiens à préciser qu'il s'agit d'un problème côté serveurs.
Concernant les anomalies de samedi 12 octobre 2019, les données récoltées durant la précédente anomalie serviront de base et nous en tiendrons compte par rapport à l'actuel état des inscriptions aux événements, avec une marge supplémentaire."
Well, Ingress was fine for me, at home in the UK :p
I was also at home in the UK on a fast Wi-Fi connection, and at about 2pm I had to give up on my daily recharge because every press of the recharge button and every use of a power cube was taking up to a minute to resolve. It was better some time later in the day.
This matches my experience exactly.
This aged well.