MAL integration suddenly started to result in 403 forbidden

Jun 7, 2020 4:11 AM

Offline

Oct 2017

Hi,
Up until yesterday we had a working integration on our board with myanimelist, simply enough users would add their MAL username in their user profiles and their anime/manga lists would show up there.
Then the request would be cached for 4 hours until a refresh would happen on page refresh.

However since today the request has been going to 403 forbidden and the whole integration fell to pieces.
Not sure why as when i test it locally it still works like a charm.

Error:

failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden

Is there some policy i haven't been following? As it almost feels like my server has been blacklisted for some reason.

Please advise (happy to link the board privately as i do not intend to spam it, of course) :)

*edit*

Just noticed that VPN have been blocked recently, WE DO NOT use a VPN for our server, however the site is running through a cloudflare server to improve caching performance, could that be the issue? If so, is there anything that can be done to sort this? Our users love the feature and it would be an absolute shame if we had to take it down because of this, especially if the issue is cloudflare related as it's widely used across the web :/

riofrizJun 7, 2020 5:37 AM

Jun 7, 2020 10:51 AM

GSculerlor

Offline

May 2015

Seems MAL blocked IP ranges from cloud service providers (like DigitalOcean), that's the reason why it return 403

Jun 7, 2020 11:03 AM

riofriz

Offline

Oct 2017

GSculerlor said:
Seems MAL blocked IP ranges from cloud service providers (like DigitalOcean), that's the reason why it return 403

Right, is there any reason why they'd do such a thing?
Did they do some kind of announcement or something I can refer to?
Also, are you aware of any alternative way to simply fetch users anime/manga aside from the classic json view of the user page?

Jun 7, 2020 10:25 PM

Shining_Boy

Offline

Jan 2018

I have a React application hosted on Heroku that fetches JSON data from MAL that is now failing. Last time I tested it was a couple days ago, and it worked fine then. Also found out that the unofficial MAL API Jikan is returning 403 errors as well. I also want to know if this was announced/documented somewhere, and if there is another way I can get user data so that I can fix my app.

Jun 8, 2020 10:18 AM

mike_mal

Offline

Apr 2012

GSculerlor said:
Seems MAL blocked IP ranges from cloud service providers (like DigitalOcean), that's the reason why it return 403

M247 all banned too(lots of VPN based on it servers),and many others. Seems they now just autodetect NONE private IP address and block access

Jun 8, 2020 10:20 AM

riofriz

Offline

Oct 2017

I mean, it's ok if they are doing tests, or if it's a mistake of some sort, I just hope is not permanent, I really don't want to switch to some different anime provider, I've been a loyal user for years now, would be very sad to have to do a switch 🤦‍♂️

Jun 8, 2020 10:40 AM

GSculerlor

Offline

May 2015

riofriz said:
I mean, it's ok if they are doing tests, or if it's a mistake of some sort, I just hope is not permanent, I really don't want to switch to some different anime provider, I've been a loyal user for years now, would be very sad to have to do a switch 🤦‍♂️

It's actually happened 2 weeks ago, but done in 4-5 hours or so. Seems right not is way longer so perhaps it more like permanent move from MAL dev itself, who know

Jun 8, 2020 10:45 AM

riofriz

Offline

Oct 2017

GSculerlor said:

It's actually happened 2 weeks ago, but done in 4-5 hours or so. Seems right not is way longer so perhaps it more like permanent move from MAL dev itself, who know

It'd be nice to have some sort of announcement at least and some ways to get around the problems, legally of course (such as Devs giving guidelines).
And it stopped working two days ago for me, so guess they are still in the process of doing this 🤷‍♂️

Jun 9, 2020 2:38 AM

Luna

★★★★★

Offline

Sep 2008

19246

I don't know any exact details at the moment but from what I've seen in the other thread, it seems that people using cloudflare are also affected.

Generally it seems that the restrictions for web scraping have become much stricter now.

For everyone who created an app/service that relies on scraping the website:
The best way to access data from MAL will be with the API. We still don't have a public release and I'm not involved in the development at all so I can't give you any ETAs. But hopefully we'll know more in the next few weeks.

We'll post more info on the 403 error once we have more information (either here or in the other thread so it's best to watch it too).

Jun 9, 2020 2:47 AM

#10

riofriz

Offline

Oct 2017

Luna said:
I don't know any exact details at the moment but from what I've seen in the other thread, it seems that people using cloudflare are also affected.

Generally it seems that the restrictions for web scraping have become much stricter now.

For everyone who created an app/service that relies on scraping the website:
The best way to access data from MAL will be with the API. We still don't have a public release and I'm not involved in the development at all so I can't give you any ETAs. But hopefully we'll know more in the next few weeks.

We'll post more info on the 403 error once we have more information (either here or in the other thread so it's best to watch it too).

Thanks for the reply, this makes sense, as i suspected it has to do with Cloudflare.
I just want to add that i am not really scraping the site per se, just using this widely available output:

https://myanimelist.net/animelist/riofriz/load.json?offset=0&status=7

to fetch user anime when they enable the feature on our board ahah
but yeah, i'll keep an eye out and see what's what :)

Jun 9, 2020 10:24 AM

#11

Kotofei

Offline

May 2008

Luna said:

For everyone who created an app/service that relies on scraping the website:
The best way to access data from MAL will be with the API. We still don't have a public release and I'm not involved in the development at all so I can't give you any ETAs. But hopefully we'll know more in the next few weeks.

Hi! I hope so and waiting for the news, too.

I want to add what I'm currently watching on my homepage.
To be honest, MAL's lack of API was the reason for me to look for other alternatives, but I wrote a simple scrapper and was too lazy so and decided to stay here :)

But my heroku app doesn't work right now and I'm very sad.

Jun 11, 2020 8:56 AM

#12

Luna

★★★★★

Offline

Sep 2008

19246

Kineta posted an update here:
https://myanimelist.net/forum/?topicid=1841882&show=200#msg59990347

If you are running an app or site which pulls data from MAL and is receiving 403 errors, please contact Kineta.

See All Discussions

» Anime Characters ruzz07 - 6 hours ago	0	by ruzz07 »» 6 hours ago
» Whole site became unresponsive Nomori - Yesterday	4	by traed »» 10 hours ago
» Keep getting Captcha every few hours AlecGH - Mar 23	3	by YamiNoNoroi »» 10 hours ago
Sticky: » Inactive Username Request Thread ( 1 2 3 4 5 ... Last Page ) Kineta - Sep 21, 2015	3430	by rurina-chan »» Yesterday, 9:48 AM
» How do I manage apps access to my account (OAuth tokens)? danann - Sep 17, 2022	6	by Noboru »» Yesterday, 9:24 AM

MAL integration suddenly started to result in 403 forbidden

More topics from this board

» Anime Characters

» Whole site became unresponsive

» Keep getting Captcha every few hours

Sticky: » Inactive Username Request Thread ( 1 2 3 4 5 ... Last Page )

» How do I manage apps access to my account (OAuth tokens)?

MoreTop Anime

MoreTop Airing Anime

MoreMost Popular Characters