| Hello everyone. I wanted to take a quick moment to update the community (and our 3rd party developers) on a change we’re making to our API security. In an ongoing effort to protect against hackers and malicious attacks, we’re implementing stricter policies that govern our API service. Specifically, a new IP blocking system (similar to our current IP block for failed login attempts) will be activated within the next 24 hours. To make this as smooth as possible, I notified our most prominent 3rd party application developers weeks ago, in our hopes that they could accommodate this change in their app. What impact will this have on you, a normal user? There should be minimal to zero impact for the general MAL user. What impact will this have on a 3rd party app user? This depends largely on the app developer, and on their chosen implementation strategy. It may cause authentication issues and delays if not done properly. If you’re an app developer and running into this problem, we recommend you first authenticate to MAL through the client (user side, not server side), then send additional calls through your application and/or middleware once authentication is successful. Moving forward and into the future, I want to let everyone know we plan to completely overhaul our API service - into something standardized and robust. I regret to say this is not on our short to-do list, but it is firmly in our 2017-2018 roadmap. Please let me know if you have any questions or issues once this change is made over the next day. |
| Great idea. I would gladly help improve MAL, if there are 2 things I love, it's anime and being a developer. |
| As long as Pocket MAL continues to work, I'm happy with this change. Better security should mean a better MAL experience! |
 |
| Thank you for your improving our experience even more! |
 |
| any chance you will modernize the look of MAL on the desktop side? |
| I guess I'm not prominent enough to have been notified. Can you give some details about this change? |
LordHighCaptain said: Apologies for not notifying you LordHighCaptain. I've sent you a PM with additional details.I guess I'm not prominent enough to have been notified. Can you give some details about this change? |
| @LordHighCaptain You can blame me for that. I linked a 3rd party developer spreadsheet in a previous API announcement to try and get contact information from everyone. You must not have seen it. I'll send it to you to via PM. If anyone else would like to be added to this sheet, please message me. |
| Currently developing an app for UWP. Can you PM me what changes should I know for the newer API later? |
| thank you for keeping our information and the site as safe as possible |
| I've already asked this question but didn't get any answer for now. What exactly changes were made to the API? How will API users know that IP is blocked? I mean when I call "verify_credentials" how can I distinguish between a case when IP is blocked and a case when creds are actually wrong? |
| Pocket MAL - MAL client for Android |
| Could you tell me the details as to how long these blocks are and after how many failed attempts they trigger? |
 The unofficial MyAnimeList REST API |
| Yay! First info about new API since forever, I hope it'll be robust enough... Thanks! /s G-Lodan said: I've already asked this question but didn't get any answer for now. What exactly changes were made to the API? How will API users know that IP is blocked? I mean when I call "verify_credentials" how can I distinguish between a case when IP is blocked and a case when creds are actually wrong? I'm interested in this too... |
DrutolJul 6, 2017 9:47 PM
| I made Windows 10 and Android MALClient apps! Source! |
| @Kineta I also would like to see the details. Xinil said: Moving forward and into the future, I want to let everyone know we plan to completely overhaul our API service - into something standardized and robust. I regret to say this is not on our short to-do list, but it is firmly in our 2017-2018 roadmap. . Thanks for considering this. Just creating a wrapper is hard enough. Also many idea can be added for this new API. I think you should open a suggestion for it. Well since nobody looks to suggestion forum in MAL... I mean maybe you or other mods check every one of them but the problem is there is no indication of that so it just discourage the user. So back to our topic I hope after API you oo/and others will work on thee code of the site as a whole. There are just lots of thing that can be implemented with advancement we have right now. G-Lodan said: I've already asked this question but didn't get any answer for now. What exactly changes were made to the API? How will API users know that IP is blocked? I mean when I call "verify_credentials" how can I distinguish between a case when IP is blocked and a case when creds are actually wrong? Also this is a good question that needs to be clarified. |
sasalxJun 29, 2017 10:47 AM
| I didn't discuss any details. I have a spreadsheet with app information from users who have given their information to me. Nothing about the API. |
| Could I get some details on this too |
 |
Xinil said: If you’re an app developer and running into this problem, we recommend you first authenticate to MAL through the client (user side, not server side), then send additional calls through your application and/or middleware once authentication is successful. Sadly, this was not enough for iMAL. My server got banned a couple hours ago, probably because of users who did not update to the latest version of the app that includes the additional security checks. I was hoping I would not have to abandon Atarashii (unofficial API) and do the authenticated calls myself directly through the official API, but I don't really have a choice now. |
| Yeah, this broke iMAL. It wouldn't be an issue if there was an official iOS app or something, but as it is iMAL is pretty much the only one that is any good out of like, 3 choices (last I checked). |
 |
| iMAL wouldn't exist if there was an official iOS app. You can use MyAniList for now, it seems to be making direct API calls client-side. It doesn't tell you when you're trying to update your list and you're banned though, so be careful. |
G-Lodan said: I've already asked this question but didn't get any answer for now. What exactly changes were made to the API? How will API users know that IP is blocked? I mean when I call "verify_credentials" how can I distinguish between a case when IP is blocked and a case when creds are actually wrong? Somehow, it makes me so happy to see G-Lodan in this post. |
 |
IATGOF said: iMAL wouldn't exist if there was an official iOS app. You can use MyAniList for now, it seems to be making direct API calls client-side. It doesn't tell you when you're trying to update your list and you're banned though, so be careful. For this we need a better approach rather than "safe and hard". Maybe with the new update we could have 2 blacklist rather than one for this new check. Blacklist for the IP block should have a support for situation like this. Well at least for now while people trying to adapt. |
sasalx said: For this we need a better approach rather than "safe and hard". Maybe with the new update we could have 2 blacklist rather than one for this new check. Blacklist for the IP block should have a support for situation like this. Well at least for now while people trying to adapt. I'm not sure I understand what you're saying but if you're talking about a whitelist for servers (so they can avoid being banned, especially for situations like mine where actual different users are using the same IP to communicate with MAL), we have already suggested that and they have refused. |
IATGOF said: sasalx said: For this we need a better approach rather than "safe and hard". Maybe with the new update we could have 2 blacklist rather than one for this new check. Blacklist for the IP block should have a support for situation like this. Well at least for now while people trying to adapt. I'm not sure I understand what you're saying but if you're talking about a whitelist for servers (so they can avoid being banned, especially for situations like mine where actual different users are using the same IP to communicate with MAL), we have already suggested that and they have refused. Yes. I was talking about something like that. Like a ticket system. Well it is sad to hear that it is banned.I really wonder why it is refused since the official API is a mess rn imo. At least it is considered to be redone. |
shanzuo said: Somehow, it makes me so happy to see G-Lodan in this post. Oh my, maybe it's the first time when somebody is happy to see me :P A new version of Pocket MAL will be available soon, it will show to a user a notification that his/her IP is blocked "... please try again later." |
| Pocket MAL - MAL client for Android |
sasalx said: Well it is sad to hear that it is banned.I really wonder why it is refused since the official API is a mess rn imo. Probably too much work. Also, 3rd party apps generate no ad money. sasalx said: At least it is considered to be redone. I wouldn't hold my breath on that. They've said the same thing in 2015. |
G-Lodan said: shanzuo said: Somehow, it makes me so happy to see G-Lodan in this post. Oh my, maybe it's the first time when somebody is happy to see me :P A new version of Pocket MAL will be available soon, it will show to a user a notification that his/her IP is blocked "... please try again later." The reply made me even happier. I dunno I'm seriously grinning. Thanks a bunch!!! |
|
| By stricter 3rd app policy, does it imply vpn usage? Each time I use a vpn on MAL it warns me not to evade ban, though I never was banned in the first place. So will I be banned next time? |
 |
Lolsebca said: By stricter 3rd app policy, does it imply vpn usage? Each time I use a vpn on MAL it warns me not to evade ban, though I never was banned in the first place. So will I be banned next time? That is probably because the IP that VPN uses at that time is already banned because of the past usage. |
IATGOF said: Xinil said: If you’re an app developer and running into this problem, we recommend you first authenticate to MAL through the client (user side, not server side), then send additional calls through your application and/or middleware once authentication is successful. Sadly, this was not enough for iMAL. My server got banned a couple hours ago, probably because of users who did not update to the latest version of the app that includes the additional security checks. I was hoping I would not have to abandon Atarashii (unofficial API) and do the authenticated calls myself directly through the official API, but I don't really have a choice now. I like iMAL. It is sad that iMAL can not be used. How about changing the IP of Atarashii API Server and using the new IP for the new iMAL version? |
| Changing IPs is often not an easy solution, and it's not practical. With the measures MAL took, it is very possible to inadvertently get banned in under 10 minutes. Speaking of bans, I can confirm the server that the Atarashii! application uses has also been banned despite me taking all measures I could (and implementing suggest fixes like confirming credentials at the client side). Back at the end of May when known developers were notified, I suggested some small changes to prevent this mess, but was soundly ignored after a response that the devs didn't want to consider my suggestions (because they misunderstood the proposal). I've even offered to help MAL use the API I co-develop to get a proper functional feature-rich API immediately and have been ignored on that too. It is my opinion that MAL doesn't want any kind of third party development, but just don't want to come out and say it. |
motokoJun 30, 2017 10:00 AM
| Developer, sysadmin, and anime addict. Have an Android smartphone? Try Atarashii! |
| I agree with you motoko. It is a first aid. Ideally, MAL's devs should rework the API. For example, use oauth2. But in this case, I believe that the probability of new iMAL BAN is dramatically increased because the old iMAL and the new iMAL share the same server. Old iMAL calls /api/account/verify_credentials.xml from the server, so the probability of BAN is high. Since the new iMAL calls /api/account/verify_credentials.xml from the client, the probability of BAN is low. However, I think that the old iMAL and the new iMAL share the same server, so I think that if the old iMAL is BANed, the new iMAL is BANed too. So, by separating the old iMAL and the new iMAL server, I think that the probability of a new iMAL BAN will decrease. I think it would cost money to prepare two servers, so I thought it would be nice to change IP. The old iMAL will die. It's not a smart solution though. |
rintarotJun 30, 2017 11:35 AM
rintarot said: I think it would cost money to prepare two servers Yeah I'm definitely not paying for a new server thank you. Your idea is interesting, but both version of iMAL use the same URL to access the server. Which means even if I were to change the IP address, both would still point to the new IP address and that wouldn't solve the problem. Changing the URL/domain would work, but that would mean I'd need to update the app to point to the new domain. I might as well just use the official API if I'm going to update the app. |
| Thank you for considering IATGOF. I think update is necessary as you say. I thought that it would be relatively easy to prepare a new subdomain because it would not cost you money. On the other hand I thought that it would take quite a long time to fix to use the official API. And the official API is ugly and the Atarashii API is amazing. |
| It's not just iMAL. The Atarashii! application has a similar issue. We issued an update to do some of the fixes, but can still get banned if 5-10 users all have a password issue, like changing it on the website. Changing the server won't help as much. rintarot said: I thought that it would be relatively easy to prepare a new subdomain because it would not cost you money. Depends, since you'd also potentially need a new SSL certificate (unless you can use something like Let's Encrypt). It's not a free action. Thanks, Ratan and I work hard to keep up with MAL's changes to keep it working. |
motokoJun 30, 2017 11:59 AM
| Developer, sysadmin, and anime addict. Have an Android smartphone? Try Atarashii! |
| Thank you motoko. That situation seems to be a problem. Will not the situation improve by urging the user to log in again when 401 returns? Even if 401 returns, if the application retries, it seems to be banned in a flash. It is still a problem of probability, but the probability seems to be improved. Although it is not a fundamental solution, I guess it's better to be able to do it now. motoko said: Depends, since you'd also potentially need a new SSL certificate (unless you can use something like Let's Encrypt). It's not a free action. You're right. Thanks to you, the application is very easy to make. Mod edit: Merged triple post. |
KinetaJul 2, 2017 10:56 AM
rintarot said: I thought that it would be relatively easy to prepare a new subdomain because it would not cost you money. On the other hand I thought that it would take quite a long time to fix to use the official API. It would definitely be easier, but it's not a proper solution. The client checks should help avoid the server getting banned, but they are not foolproof. The suggestions made by motoko would have been a proper solution, but with the implemented security changes it's not really possible to use a central API system for authenticated calls. Changing the API calls to use the official API in the app wasn't really hard, my biggest problem with that is that I no longer have the security of going through a central system. If the unofficial API breaks for any reason, I will not be able to react quickly by updating a server, I'll have to update the app, wait for the app store review and wait for people to update their apps. Which is the main reason I decided to use Atarashii (that and all the added data, and the fact that it's a properly built and documented API) |
rintarot said: Will not the situation improve by urging the user to log in again when 401 returns? Even if 401 returns, if the application retries, it seems to be banned in a flash. It depends. From my understanding, it's a two hour ban after 10 failures. This means that depending on how the failure is done, a few people can trip it. Say you have a user who fails auth for some reason, then are prompted to re-enter their account password. After two tries, they can't try again for two hours enforced by the client. If four clients go through those steps, they can easily get the server banned for several hours. For my server, I auto-ban IPs based on failures in a specific timeframe to try and reduce problems, but this isn't possible to guard against for all situations. |
| Developer, sysadmin, and anime addict. Have an Android smartphone? Try Atarashii! |
understood. If it is not hard to use the official API, it may be better to do so. motoko said: Say you have a user who fails auth for some reason, then are prompted to re-enter their account password. After two tries, they can't try again for two hours enforced by the client. If four clients go through those steps, they can easily get the server banned for several hours. In this case, I think that by using /api/account/verify_credentials.xml from the client it is possible to lower the probability of failure by confirming that the id and password are correct. Because clients have different IP, so IP BAN counts are different from each other. Mod edit: Merged double post. |
KinetaJul 2, 2017 10:56 AM
Matoro_Mahri said: That's great news! You guys should definitely take user contributions into account, like hosting on Github or something similar. I'm sure there are plenty of developers including myself who would love to help make MAL better :) They will not because the website is coded very badly. Contributors would laugh at their dev team. The fact that they use XML is enough. MistyBlue said: Thank you for your improving our experience even more! Meeno_Minhas said: You still can hack many accounts so there is no improvement.thank you for keeping our information and the site as safe as possible KaitharVideo said: Try not to get your hopes up. It was announced in 2015. It is now 2017 and the official one has been like this since 2008 Oh wow almost after 10 years it is still the same.It would be really nice to know if this means a decent API for getting show information is coming. rintarot said: In this case, I think that by using /api/account/verify_credentials.xml from the client it is possible to lower the probability of failure by confirming that the id and password are correct. Because clients have different IP, so IP BAN counts are different from each other. What will happen if the client is using an old version? That is what happened. We released a new version weeks ago and it still got banned. What will happen if someone has an own build because we are open source and will adjust the app code to use another server with a wrong auth? Xinil said: It may cause authentication issues and delays if not done properly. If you’re an app developer and running into this problem, we recommend you first authenticate to MAL through the client (user side, not server side), then send additional calls through your application and/or middleware once authentication is successful. You are telling us that we are bad programmers if the app get slower with advice that will delay the app because additional calls require time. You are blaming us for bad practice while your suggestion is even worse... Xinil said: Xinil I would like to see what has higher priority than the API which was promised before 2015?but it is firmly in our 2017-2018 roadmap This entire “security thing” is bad practice. You had many ways to fix it but you just decided to take the least effort and dismissed other ideas. I am actually not even sure if you are a programmer. Why did the dev team made this decision now. I informed in 2015 about an issue, which could used to hack accounts. Why was the mobile site introduced while the accounts weren’t safe? Actually, there is still a way to harvest many MAL accounts XD Imho this is just to get users the feeling they are safe while they aren’t. |
[center] |
Ratan12 said: They will not because the website is coded very badly. Contributors would laugh at their dev team. The fact that they use XML is enough. To be fair, XML isn't a bad thing. If done correctly, it's a great solution. As for the idea that an API should be low priority, that is very wrong thinking. You should build an API and then the site around that. It'll make security much easier and should speed development. Also, by making the UI independent you can now build a rich mobile application or even a dedicated mobile website without harming the main desktop site. Treating an API as an extra will ensure outdated coding practices continue and will indefinitely delay an API as something will always come up. |
| Developer, sysadmin, and anime addict. Have an Android smartphone? Try Atarashii! |
motoko said: Ratan12 said: They will not because the website is coded very badly. Contributors would laugh at their dev team. The fact that they use XML is enough. To be fair, XML isn't a bad thing. If done correctly, it's a great solution. I have to disagree with this. I mean it 2017 and the amount of thing you can do with new tools and modules is just amazing. But instead with the limited option we are trying be ok. Probably the the reason of decreasing amount of new features is because of the limitations. There is also one thing I can %100 agree with @Ratan12 and that is "what has higher priority than the API" question. Only thing I can think is rewriting the whole site and that is so unlikely. |
sasalx said: I have to disagree with this. I mean it 2017 and the amount of thing you can do with new tools and modules is just amazing. But instead with the limited option we are trying be ok. Probably the the reason of decreasing amount of new features is because of the limitations. XML is simply a format. It can be implemented well, and it can be implemented poorly. Everyone loves JSON, and it certainly is lightweight, but it also has its own limitations. Now, the existing official "API" is pretty limited. That's not really at argument here. My point is simply that you can't look and see it uses XML for data and say that's bad in and of itself. sasalx said: There is also one thing I can %100 agree with @Ratan12 and that is "what has higher priority than the API" question. Only thing I can think is rewriting the whole site and that is so unlikely. The thing is, the amount of time it has been since the promise of a "new API soon" should have been more than enough time to rewrite most of the website. In many cases, it looks like parts of the backend have been changed. My argument is that they could have taken that effort and put it in a solid API and then made the website use that API, allowing them to complete multiple things at once. It seems they are rather sticking with an outdated structure and insufficiently separated front and back end code that requires stupid "security" like they announced that is more theater than anything else. I know some people will blame PHP, which is used by this site. However, PHP has come a very long way in the last decade and there are tons of best practices that cover a lot of area and that are part of a mature language. The real core issue is the scrambled old PHP code where you can't properly ensure input sanitization and compartmentalization. Honestly, though. Even if they were following all those practices, the feeling I get is that MAL is slowly trying to strangle third parties to force everyone to have to come to the site (need to monetize, after all). They'll keep dragging the "API soon" message because it's gone too far to simply say that it's not going to happen. They must feel it better to cause upset among the very smart developers who have been supporting and bringing users to their service than to just tell it straight. That's what makes me the angriest and makes me wonder if I want to continue to direct people to this site. |
| Developer, sysadmin, and anime addict. Have an Android smartphone? Try Atarashii! |
Ratan12 said: Try not to get your hopes up. It was announced in 2015. It was publicly announced on Reddit in 2015, but they were already telling developers they were working on it before that. I have a PM from March 2013 saying "We're working on improving the MAL API in the near future". These are just completely empty promises. motoko said: XML is simply a format. It can be implemented well, and it can be implemented poorly. Everyone loves JSON, and it certainly is lightweight, but it also has its own limitations. Although I agree with you that XML is not a bad thing itself, I see no reason to use it over JSON for a public API like MAL's. It makes no use of the possibility offered by XML over JSON. I think what they meant is that looking at the API and seeing it uses XML hints at either a badly outdated API, or a poor understanding of the users' needs. motoko said: Honestly, though. Even if they were following all those practices, the feeling I get is that MAL is slowly trying to strangle third parties to force everyone to have to come to the site (need to monetize, after all). They'll keep dragging the "API soon" message because it's gone too far to simply say that it's not going to happen. They must feel it better to cause upset among the very smart developers who have been supporting and bringing users to their service than to just tell it straight. That's what makes me the angriest and makes me wonder if I want to continue to direct people to this site. While I can't tell whether it's a concious choice to remove third parties or not, it definitely feels like it. Or at the very least that they do not care at all about third parties. Over the years they have done nothing to help or accomodate third parties, knowing fully how bad their API is and publicly aknowledging it. Every time the subject has come up I've asked for more transparency. If you think sending a notice about the changes scheduled to the API and then ignoring our concerns is enough, it's not. This is all very one-sided. You say you're making a change (or don't, remember the whitelisted user-agent?) and then pay no attention to the actual implications it has on third parties. |
| That's great. But I don't see why that bug where users can view data from people that have their list hidden is not fixed yet? (Afaik it is still possible with one URL I tried. Someone made topics about this in the past already.) If you are talking about security I think such stuff should have highest priority. Some IP block ... I guess it only helps against DDOS and brute forcing (which should not affect people that much if they have secure and long passwords). Does not sound that important too me. (But I guess it was easy to implement that's why it got added 1st.) Also is there some official full documentation about the API? I think the one thread here in MAL I found did not seem to list everything (I saw people in other threads talking about other stuff not listed there and unofficial documentation ... but that wasn't available anymore ... link did not work..) |
Luthandorius said: Also is there some official full documentation about the API? I think the one thread here in MAL I found did not seem to list everything (I saw people in other threads talking about other stuff not listed there and unofficial documentation ... but that wasn't available anymore ... link did not work..) This is the official documentation. As you can see, there is no way to get a user's list (among other things). The commonly accepted way to get lists is by using https://myanimelist.net/malappinfo.php?u=iatgof&status=all&type=anime (which is not documented) |
| Good update, but what 3. party apps use MAL if I may ask? Never heard about any lol. |
SkullProX said: Good update, but what 3. party apps use MAL if I may ask? Never heard about any lol. Many of them are apps for Windows Android or IOS. Also there are some unofficial APIs and wrappers. |
More topics from this board
» [Challenge] You Should Read This Manga 2024 ( 1 2 3 4 5 )Kineta - Feb 23 |
207 |
by Shin_016
»»
Yesterday, 7:32 PM |
|
» Try MAL's New Mobile Site! ( 1 2 3 4 5 ... Last Page )Xinil - Feb 15, 2015 |
423 |
by RED-clover12
»»
Yesterday, 10:19 AM |
|
» Planned 5hr Maintenance, Thursday April 25 @ 1am-6am PTKineta - Apr 22 |
0 |
by Kineta
»»
Apr 22, 8:10 PM |
|
» New Site Update: Peak Anime 🗻 ( 1 2 3 4 5 )Kineta - Mar 31 |
213 |
by Lancelot73
»»
Apr 21, 4:28 AM |
|
» Heavenly Easter Delusion: Devil and Dolce ( 1 2 3 4 5 ... Last Page )Kineta - Mar 27 |
3331 |
by Terra_strong
»»
Apr 17, 8:26 PM |