Following your explanation, I have a question, however. If we take a unique indicator on a single portal screen, what's so difficult in "take hash of currently viewed portal id and user id, and perform "exists" query against unique hacks or unique captures tables"? Such query is already performed on every single hack we make, right? What's so costly if we do the same but only for viewing a portal, and cache the value on client until a hack/capture of this portal is performed? Maybe @ofer2 could give some insight too? Of course this wouldn't show us unique state of every portal in view, and would only be available when viewing single portal (either in scanner or on intel), but it could help as an intermediate solution?
This might be piggybacking on your idea of an intermediate solution: would it be much easier (again, as an intermediate solution) to implement a unique viewer on the scanner map or individual portal screen or both that starts *right now*? IIRC, PoGOs unique markers weren't backfilled, when the feature was implemented everything appeared as not visited (though admittedly there wasn't a medal/tracker for it up to that point either).
I would be thrilled if visual markers were implemented in-scanner/intel tomorrow and everything was marked as 'not visited/captured/etc.' The current unique system could be the only one tied to actual stats and the new visual stuff would be only visual, but the two could work hand in hand. It's far from a perfect system, but could bridge the gap until the old hash system get reworked/adapted and could even allow the Ingress team to significantly lower that priority.
A few months ago I started using the 'Uniques' plugin for IITC and it has been great at encouraging me to knockout my local area, though it's a bit of a cumbersome process to track, it still gives me the feeling of making progress and has significantly increased my recent playtime.
I believe that the agents should give their opinion about it if it would good or bad. And only the Niantic team tell if could de done or not.
I think that none of us knows exactly how Niantic’s inside system works. Let them explain theirselves
they have - see the link @Perringaiden posted from @ofer2 earlier....
what's so difficult in "take hash of currently viewed portal id and user id, and perform "exists" query against unique hacks or unique captures tables"?
My assumptions (@ofer2 would know specifics): It's "doable" but it would be performance cost unfriendly, because you would have to do it for every agent every time their display refreshed. So standing in a field of 200 portals, you'd have to make 200 hashes, check each one against the database then come back with a result.
When recording new uniques, that should all be on a separate thread disconnected from the client response to the hack, happening in 'background time'. The process I described isn't as super response as you might think either, simply because of the indexed search. So you could do it, but it would significantly slow down loading portals in the display. Instead of a 'fire and forget' process checking for a unique and updating if necessary, you'd have to wait for all 200 portals to process that check before anything came back to be displayed.
It might be performance achievable on a "Retrieving the details of a single portal", but again you come back to the likelihood of people implementing "PoGo Map Generator"-like client hammer tools that would search out all portals and tell you your list. In which case they may as well wait for it to be converted and available on Intel because that wouldn't hamper the game.
I definitely think that implementing anything that doesn't include the same data accessible from the Intel map would be a significant failure in understanding how the community would behave.
I would be thrilled if visual markers were implemented in-scanner/intel tomorrow and everything was marked as 'not visited/captured/etc.'
This could be done, but again it would still take actual development work on both the server and client to provide the infrastructure for recording and retrieval.
As an aside to that idea, you could still do the 27 trillion calculations in 'background processing' and have it backfill the new infrastructure over a matter of months. With your data slowly being filled in, it wouldn't be complete initially but would eventually.
All 'doable', not really 'simple'. But requires dedicated development time.
As @KarM3L said, Ofer2 has described the storage mechanism. That gives us plenty of understanding of the complexity of converting the existing data, and why it's a nightmare to use in it's current state.
We all want it. Definitely keep emphasizing this to @NianticBrian, as he has the final say on development priorities. But it will take time.
I'd rather wait a bit more but have a solution which marks all unique captures/hacks on the map, yes. And I would be thrilled as hell too, see I such markers. Haha pioneer and explorer counts go brrrrr :)
Yeah, this is mostly correct. The main thing that I would like to point out as well is that even with development effort, there are additionally costs related to actually running a data backfill process. Using your numbers: 7 million portals + thousands -> millions of actions per player leads to a lot of data to look through. Since the hash is comprised of both the player id and the portal id, none of that work is reusable meaning that all that compute power going through the many millions of entries is dedicated to a single person. This has a very significant cost. Not sure how large, but its possible that that may make it very difficult to do overall. This is the same reason why passwords are secure: because the cost of going though each password and checking to see if its the right one is computationally expensive when you have to try all of them, but fast if you only try one. All this to say that if a feature like this does come out, it will likely involve some form of cost to the player to get your historical data added. Data from now moving forward doesnt have to do that massive search and so can be added relatively cheaply.
I think the misunderstanding here is that the system doesnt actually know when you hack a portal if you've hacked it before. I think you're trying to infer that based upon the unique hacks metric. That metric is computed separately. I.e. if you hack a unique portal and then look at the metric it wont immediately update. Instead, we have a separate process which goes through a bunch of actions at certain time intervals and will then do lookups for the unique actions and then use that to update the metric. I'm not certain of how often that flow runs or if it is queue based or time based (i.e. wait for X number of things before processing, wait Y time before processing, or process what has been added since the last time you started processing). The reason this queue / delay system was created was exactly because its an expensive/slow process and cant be done in real time.
All this to say that if a feature like this does come out, it will likely involve some form of cost to the player to get your historical data added.
This is definitely something I'd be prepared to pay for.
Not sure if I'd pay for it or not. I would gladly trade X number of scans for X number/time worth of unique identifiers though. I mean, Niantic will eventually benefit from my time producing scans, so it would be sweet if Niantic returned that time...
If it's possible to do so with new uniques, you guys should do it. If getting historical uniques is a 1 time transaction a lot of people wouldn't mind...
Excuse me, my good sir, but wasn't Prime "built from scratch"?
Guess this has something to do with the extreme inneficient anti-cheat features. And would also explain a lot about VGs...
Only the client was rebuild for prime, the backend hasn't changed, should of by now, but delayed indefinitely
Don't be condescending; I have more experience in this area than you do.
The OP wasn't asking for all portals in an area, they were asking for it on the portal details screen. So you just have to generate the hash and do one lookup, which is probably one of many such queries that happen to show the portal details screen, e.g., to show if you have a key.
I agree that showing all portals in an area, or on the map, would be considerably more difficult (and more useful).
No, the client was built off the PoGo codebase and the servers are the same ones we had in 2012 (albeit tweaked and in an encapsulation on a different server platform).
I have more experience in this area than you do.
Then use it. I won't attempt to have a pissing contest about 25 years of software development experience.
In terms of a single request, see other points about how providing a Client only data point would destroy the game, the same way that the PoGo Mapping tools almost did there. The "check if it's already unique" is done on a disconnected thread, separate to the client request. It is not done in client time during the API request.
From @ofer2 :
I think the misunderstanding here is that the system doesnt actually know when you hack a portal if you've hacked it before. I think you're trying to infer that based upon the unique hacks metric. That metric is computed separately. I.e. if you hack a unique portal and then look at the metric it wont immediately update. Instead, we have a separate process which goes through a bunch of actions at certain time intervals and will then do lookups for the unique actions and then use that to update the metric.
Right, so I see how splitting the hash database would require # players * # portals operations and that would be expensive. But exposing the hash database to a read-only endpoint (instead of a batch read/write as it is now) should not be as difficult. Though it is some amount of real work -- point taken.
It's interesting that original authors of Ingress chose this design. My guess is it was for privacy, so that there is no way anyone could exfiltrate all of the locations you've ever been, as the data is not exposed beyond the batch process. That's actually pretty clever. But for PoGo they abandoned that aspect. And also, from what I understand, the scraper sites can already tell you everywhere you've ever been (unless you only hacked), so that ship has sailed too.
Add uniques for any hack/capture after the implementation and offer a "backlog" for payment
It's interesting that original authors of Ingress chose this design. My guess is it was for privacy, so that there is no way anyone could exfiltrate all of the locations you've ever been, as the data is not exposed beyond the batch process.
Game log contains information about Event Time, Event Latitude, Event Longitude. What kind privacy can we talk about?
Likely, not privacy, given how much else is freetext in the GDPR download. More likely they didn't want a compound key table with conceivably trillions of records, given that both keys are long strings (GUIDs). That would have been an even worse performance situation.
I'm actually interested in what the PoGo solution is, and likely it'll be the same logic implemented for Ingress in a different language, but the key point is "Still takes time" because "wasn't written to do this".
As long as @NianticOfficial priority is Pikachu wearing a hat, we will never have a unique portal marker
Apparently independent art teams, so they're busy making the NL-1331 van have a hat.
Made a super quick mock up of what it could possibly look like.
Rather than having a marker for what isnt captured, maybe retain the marker for those that are hacked AND captured only. It also shows various versions of showing a portal that it only hacked or only captured as well.
The really large rings around every stop annoy me, but also give me an incentive to spin them away so it doesn't clutter the game map, but also seeing a new unhacked/uncaptured portal would also be exciting. But this is very subtle and probably very hard to see unless youre right next to it or youre looking at the Portal View (maybe except for a larger ring).
Or maybe even just show the dot on the ground in the not hacked or captured portals without showing the energy coming out of it, then hacking/capturing it is an indicator that youve unlocked its energy. Dunno, just putting ideas out there.
There is no such thing as captured only
a capture counts as a visit
so it's only 3 states:
Personally, I would go with a marker above the portal (similar to shards)
marker is one color -> Not visited
marker is other color -> Visited but not captured
no marker -> Captured
Frankly, the "how to display it" is the least of the requirements. And I'm sure the Ingress Art team has plenty of options there. But without the back-end data, they can't display anything.
Prime was built from scratch compared to redacted but uh the portals are also separate and span about 9 years now. I can only imagine the POI database that has survived a company departure and a code rewrite...argh
Prime isn't on NRW platform so it's still kind of its own entity compared to the other games.
You can already see what the client tries to limit on viewing because it will give you burn out messages - submitting passcode through intel, viewing many profiles, and looking at the scoreboard timeline for cycles. I think it's safe to assume these are data intensive calls to the server.
As seen with thethe other games, the RWP is bloated. With excessive lag on important game actions all the time. So thankfully Ingress isn't on the RWP server, yet.
Though, enabling uniques to be shown in Portal View, Map View and Intel can be optimised to not adversely affect gameplay.
I haven't done GDPR, which should show all the unique captures and visits.
When ofer2 mentioned that it isn't discreet data for the tallying, there is a check done.
As soon as subscriptions come in, am sure work on this can go ahead, and not add any additional lag to the game
Keep in mind that there are a lot of players who are colour blind, or have poor vision. So any indicator would have to visually significant.
GDPR does provide the location data of the portal. However, you're also requesting your data on Ingress - this is the information specific to your hash key. Ofer2 stated the encryption is done on the fly in the server.
I think bloated is ambiguous to state and maybe even egregious in some aspects. It's the Unity platform that has pros and cons compared to the RoboVM experience that redacted utilized. A lot of automation is possible as we saw with the hexalthon and if you went to any of the anomalies - the leaderboards are automated due to the usage of an API. As I stated before, Ingress is in this purgatory spot where it's not RWP so it's hard to compare the experience with other games. We're already saw a lot of programming changes from its initial public launch to now.