API Rate Limit?

Hi, there are no updates on this.

My rule of thumb is that you should be fine as long as you keep sending no more than 2 requests per second from the same IP.

@ZeroCrystal I've been doing some testing with mal-api.py API wrapper and encountered a 3-5 min cooldown (response = 504) after around 300 sequential requests, sometimes less, usually a little more. mal-api.py says the default req rate is 1/sec. I just assumed this was coming from MAL, not the API. Am I wrong?

p.s. Really appreciate all the time and effort you put into answering questions here.

@SnoopyInOrbit Unfortunately, the real rate limits for the API have never been disclosed since its creation. Most developers set a delay of around one second between requests to avoid getting rate-limited, but it's a value that just came from empirical experimentation.

After the latest incident, I believe MAL decided to put in place an additional layer of rate limiting to temporarily block clients from sending hundreds of requests in a short time.

There's not much we can do about this. My suggestion is to back off for a few minutes when you trigger the rate limiter and then resume your job.

Indeed.

I'm happy to work within the limits, it's just nice to know about them in advance. On the other hand, I do enjoy solving puzzles and puzzling out the behavior of the API has been at least a little bit satisfying.

I've already added cooldown logic that polls in 1 min intervals until the restriction lifts.

For those interested in details (YMMV):

• Cooldown varies between greater than 3 min and less than 5 min. I never saw a cooldown longer than 5 min.
  
• The number of requests between cooldown periods varied a lot so there must be other factors besides request rate at play. The longest uninterrupted run of requests I saw was about 390 at 1/sec. The shortest was about 140. The average seemed to be around 300.
  
• Changing the request rate (slightly) didn't seem to help: 1 req/1 sec, 1 req/2 sec, 1 req/4 sec all seemed to result in similar cooldown throttling. At 1 req/2 sec, the additional delay is about the same as the nominal cooldown so adding longer delays won't improve throughput.

I tested over the course of two days, but it wasn't rigorous - just me observing the behavior. Given the variability, you are going to need a dynamic solution or you are potentially going to lose significant throughput.