Intel Map Regularly Broken

13

Comments

  • We just pushed an update to Intel Map to address the issue where some portals, COMM messages, or Regional Scores weren't loading. Please note you may still see an occasional 502 error message, and we're continuing to investigate further. If you're still seeing an issue where some portals aren't loading, please let me know. Thank you.

  • Doing a wide scale load, and while there are still a handful of 502 errors, IITC is successfully trapping and re-requesting them. The Map appears to be going much faster and without anywhere near the errors!

    So far so good @NianticBrian !

  • Ok. So feedback after a full set of tests.

    1. The load is far faster, with only a dozen or so 502s across a few thousand requests.
    2. On the first load, there were still a lot of portals missing, which is more pointing back to the original cache issue.
    3. On the second load, it successfully hit about 90% of portals.

    So, the issues with the gateway are mostly fixed to a point where the map is once again usable. The cache is still not successfully gathering all portals on the first load, but it's definitely back to the situation in early December, which is far more workable.

  • iphone can only get passcode items from intelmap, but it doesn't seem to work now, is there any problem?

  • ofer2ofer2 ✭✭✭✭✭
    edited January 2020

    @Perringaiden I dont know how you're testing, but my guess is that you're running into rate limit issues (this is highly likely the issue if you're seeing different portals load between refreshes). Please wait ~1 hour to give your rate limit time to clear and then run a test (depending on your requests/sec of your test you may still run into rate limit issues, I dont know how you're testing the dashboard). If you've done this and are still seeing issues (despite low request / sec or in an area where you feel like "it should work" but doesnt) please let me know what area it is and I'll take a look and see if I can figure out whats going on.

  • edited January 2020

    What sort of scale are there rate limits at? If I'm hitting some sort of limit, it shouldn't be returning partial tiles or indicating success on incomplete tiles should it? Silent limits that act as if they're succeeding are a pretty bad plan.

    I'll happily say that I usually push the Intel map beyond what the average user would normally do, but if I'm hitting some sort of limits, I should be told that, rather than silently told it "succeeded" when it didn't. It's tough for us to understand what we can or can't do, if the errors don't seem like something we're doing wrong.

  • edited January 2020

    At this reference link, on a 1980x1080 screen:

    Initial load ^^


    After 3 refreshes ^^

    The issue here is that any sort of failure to respond, is transparent to the user. If there's rate limiting, it needs to be spelled out that the map they're viewing is inaccurate, because otherwise people simply don't know that something is there. Usually big links and fields are fine, but when you're scanning 500kms at z=13 looking for any blockers across a path, a short link that doesn't show up is disastrous.

    We can't simply have 'holes' in the data.

    As part of a reference set, I'm also loading pretty much all of LA at z=15, and counting the portals. The last stable count was just over 43k portals on Nov 25th. Since then, the first load was averaging about 38-40k, and the second load finished it off with about 41-42k portals. However, since all the 502 errors were cleaned up, (both before and after the {} responses were fixed) the first load is averaging about 33-35k. That means unless a user knowingly refreshes the page, they could be missing 25% of the portals on the screen.

    In a game where information is king, that's simply unusable.

  • ofer2ofer2 ✭✭✭✭✭
    edited January 2020

    @Perringaiden Thanks for taking the time to help with all this investigation and pictures! I've checked the area you linked in both standard intel + iitc and both report all of the portals in the highlighted area on the first load. The rate limits are pretty high (I dont recall off the top of my head, but its the same as they've always been, but I think that counting all portals in LA might trigger it, thats not a normal activity), but due to a bug (which is what was fixed earlier), one became saturated. It should fix itself as time goes by (as long as you dont keep spamming it and let time pass). Please let me know if you retry this area later and you still experience the problem. (It would also be helpful if you check whether a /r/getEntities request 502d because it could be part of the second issue that we're still investigating). If it doesnt appear to be better (and isnt the 502 issue), I will hard reset the dashboard (which will reset the rate limits) but that will cause all of portals/links/fields to have to reload as well which can make things take longer to load for a bit. If that doesnt work, we will continue to investigate, but please bear with us. The 502s are a separate issue, but as you've said, we're measuring that at ~3 per 400 requests currently. We're still working on fixing that, but that may time some time longer.

    Post edited by ofer2 on
  • I've just loaded the map for the area around The Open University in Milton Keynes UK.

    The bookmarks are portals I need to get as uniques - two are missing their portals.

    There are at least 12 missing unclaimed portals.

    A lot of the blue portals are missing their level number. In a few places this is because there is another portal close by, but a lot are 'placeholder' portals added by IITC at the end if links.

    After a refresh the portal count went up from 166 to 214.

    After refresh:


  • kuprumkuprum ✭✭✭

    It's time to move to send some new candidates for wayfarer..😊


  • In terms of specifics, for that linked area above, I did not receive any 502 errors. I haven't got a way (yet) to track tile by tile so that I can name the specific tile that didn't load, but I may have to discuss building an overlay in IITC if the problem persists.

    In terms of the LA call, is there any reason why you wouldn't be throwing a 429 when we start doing non-standard things that exceed the limits? This would give people like the IITC developers a way to warn users that they're exceeding desirable limits, and help curb our enthusiasm.

    The Intel map is an integral part of Ingress, and as someone else described recently, essentially part of the underlying physics of the game. An inaccurate map which we don't know is inaccurate, is like someone finding proof that the world is a simulation, because there's a hole in space time where their house used to be.

  • Still not working. Can’t plan anything :(

  • ofer2ofer2 ✭✭✭✭✭

    @abbey8k I reset the dashboard. Please retry and you should be able to get in.

  • Intel has certainly become much more responsive since the latest update, however, it is still missing Portals and links.

  • ofer2ofer2 ✭✭✭✭✭

    @Perringaiden I've hard reset the dashboard. All of the rate limits should be reset. As I said before, the hard reset resets all of the loaded portals on the dashboard. This brings me to another reason that the dashboard can be inaccurate: the dashboard was originally written for a platform which limited the amount of work which can run in one request. Therefore, a single request cannot return all of the data to the user. Furthermore, the data might not even be loaded into the dashboard and the dashboard itself has to fetch it. Because of all this fetching, the loading happens in pieces. This is part of why the dashboard can appear inaccurate. It has been this way as long as I can tell. Since I reset the dashboard, it lost all of its internal state and had to load everything from scratch. I went to the location that you linked to above and saw the same portals missing as you did. However, I waited for a bit (1-2 minutes) and then the portals did load. There may be an issue which is causing it to take longer to load (which I will investigate on monday), Again, that being said, there may be an error with the client thinking it has all the data when it doesnt; another thing to check. With all that being said, I have hopefully conveyed that its not as simple as returning a 429 to the client to tell it that its been rate limited. We could put dev effort into making that the case, but it seems better to invest in making things that players should be doing work better than making things that they shouldnt be doing work better (but thats just my opinion). Hopefully that helps.

  • ofer2ofer2 ✭✭✭✭✭

    @Gottestal The dashboard was reset which means that some portals will take longer to load. Please wait for a few minutes on the screen and see if it does eventually load. I will look into seeing if theres a way to make them load faster on monday.

  • There is a new intel dashboard?

  • I've just reloaded the OU page and got 158 portals - should be at least 214.

    If nothing was changed on the server at the end of November, is the probelm caused by something really silly like the total number of portals being too large for a cache so the information keeps being deleted before being accessed?

  • ofer2ofer2 ✭✭✭✭✭

    @beardedtit can you wait looking at intel for a few minutes? the portals should load, they just take a little longer.

  • I think all the portals appeared on the refresh after 11 minutes (quite why the 300sec refresh takes 11 minutes is anybody's guess!). And also after restarting iitc.


  • ofer2ofer2 ✭✭✭✭✭
    edited January 2020

    Interesting, Could the refresh taking longer be a difference between IITC and stock? I was testing on stock.

  • Dunno, it might be the timer only runs while iitc is the active program - I'm using a 10" tablet.

  • I've just loaded a different area, got 51 portals the first time and 103 after the refresh. I don't know that area as well, not sure if anything is missing in the reload.

    Having to reload isn't ideal!

  • ofer2ofer2 ✭✭✭✭✭

    It shouldnt need you to reload, if you just wait it should load (at least it does for me in stock intel).

  • Jo0LzJo0Lz ✭✭✭✭✭

    How can you reset anything for other users? Or are you a developer but are not marked as staff on the forums?

  • Yes, the auto-refresh picks up more portals.

    A manual one might be quicker.


  • Stock doesn't do retries if there are failures, and the refresh timer for both starts from the completion of load, so it could be that because IITC is doing retries, it takes longer to complete the initial load, plus I think the timer is slightly longer.

    Additionally, something @NianticBrian mentioned on the weekend at IFS, was a point of confusion for me. We call call Intel "Intel", but apparently that's what is internally referred to as the "Dashboard". I think a lot of us had assumed your post above was referring to some sort of behind the scenes control mechanism for the map, rather than the map itself.

  • Also, regarding rate limiting, it really doesn't feel like I'm hitting a rate limit, unless the limit is one on the cache loading from the game servers?

    When I to a full load of LA, and only get 35k portals, if I repeat the process immediately, I get a far higher number the second time, usually in the 40-42k mark. If I were being directly rate limited, I'd have expected that my count would go down because if I'm rate limited, I'd be getting less, or have more retries. It doesn't seem to be happening that way.

    It feels to me, from the outside, more like the cache is requesting data from the game servers, but before that data is returning, the cache is responding positively to requests for that tile. So in the initial load, far more of the tiles have not yet been completely cached, but are responding as if they have. On the 2nd load, those tiles have already been cached so they load correctly.

Sign In or Register to comment.