Importing data from anime-planet?

Offline

Nov 2013

I also get this error. I tried going to the url in the browser and it says the site is too busy. I read that there has been issues with DDoS lately, so it might be the cause.

I thought it might also be something to do with the unofficial API (http://mal-api.com/docs/#read_search_anime), so i tried using the official API (http://myanimelist.net/modules.php?go=api) which requires you to authenticate, but i can't seem to authenticate.

Nov 19, 2013 2:22 AM

#67

ErgoSis

Offline

Jun 2009

True, unofficial API is down as it seems.

To authenticate you can simply provide HTTP header:

titlereq = urllib2.Request("http://myanimelist.net/api/anime/search.xml")
titlereq.add_header("Authorization", "Basic %s" % base64.encodestring("%s:%s" % (malusername, malpassword)).replace("n", ""))
titlereq.add_header("User-Agent", "Mozilla/5.0 (Windows NT 6.2; Win64; x64;) Gecko/20100101 Firefox/20.0")
queryTitle = urllib2.urlopen(titlereq, urllib.urlencode({ "q" : animeName })).read()

But if a method to detect MAL anime ID had been found then one shouldn't write XML simply because it's possible to add bunch of series from the same script via native API then.

params = {'id' : animeID, 'data' : xmlData}
url = 'http://myanimelist.net/api/animelist/add/'+animeID+'.xml'
urllib2.urlopen(urllib2.Request(url, urllib.urlencode(params)))

A story has no beginning or end; arbitrarily one chooses that moment of experience from which to look back or from which to look ahead.

Nov 19, 2013 4:55 AM

#68

Offline

Nov 2013

Ah, it seems that i needed a user agent header.

Though now i'm getting a json error:

Traceback (most recent call last):
File "ap_scrapper.py", line 42, in <module>
search=json.loads(queryTitle,"utf8")
File "C:\Code\Python27\lib\json\__init__.py", line 351, in loads
return cls(encoding=encoding, **kw).decode(s)
File "C:\Code\Python27\lib\json\decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Code\Python27\lib\json\decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Here's the script i used for reference http://pastebin.com/rUm0DmG2

I'm not experienced with python, so i'm not really sure whats wrong.

Nov 19, 2013 5:40 PM

#69

ErgoSis

Offline

Jun 2009

That's because native MAL API doesn't support JSON output. It uses XML with some unknown encoding (because python breaks on it often if it's decoded as UTF-8).

Updated scripts a bit:

Native API search version of the script: http://pastebin.com/ceRkkJiA
Native API search/adding version of the script (from the post #74): http://pastebin.com/igVyHeXL

You will need python 3.3.3 to run it.

ErgoSisDec 4, 2013 6:33 AM

A story has no beginning or end; arbitrarily one chooses that moment of experience from which to look back or from which to look ahead.

Nov 19, 2013 10:33 PM

#70

Offline

Nov 2013

Excellent, thanks heaps for that!

I was able to generate an xml of about 80% of my list.

Just importing my list to anidb now before i can import it to MAL.

Thanks again!

EDIT: I imported the list into AniDB fine, but i can't seem to import it into MAL. I exported the AniDB list as an xml as per the instructions on MAL importer page.

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

elementalestNov 20, 2013 3:15 AM

Nov 23, 2013 12:52 AM

#71

Offline

Nov 2013

I still can't import the list to MAL, not sure what going on. So i tried a direct method as suggested above.

Here's the script i used: http://pastebin.com/FKwMCUN2

Traceback (most recent call last):
File "ap_scrapper_direct.py", line 151, in <module>
urllib.request.urlopen(url)
File "C:CodePython33liburllibrequest.py", line 156, in urlopen
return opener.open(url, data, timeout)
File "C:CodePython33liburllibrequest.py", line 475, in open
response = meth(req, response)
File "C:CodePython33liburllibrequest.py", line 587, in http_response
'http', request, response, code, msg, hdrs)
File "C:CodePython33liburllibrequest.py", line 513, in error
return self._call_chain(*args)
File "C:CodePython33liburllibrequest.py", line 447, in _call_chain
result = func(*args)
File "C:CodePython33liburllibrequest.py", line 595, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 501: Not Implemented

I suspect that something is wrong with the xml data, but i have checked a dozen times and can't figure out what. Any help would be great.

Here's an example of the xml data i generated

<?xml version="1.0" encoding="UTF-8"?>
<entry>
<episode></episode>
<status>6</status>
<score>0</score>
<downloaded_episodes></downloaded_episodes>
<storage_type></storage_type>
<storage_value></storage_value>
<times_rewatched></times_rewatched>
<rewatch_value></rewatch_value>
<date_start></date_start>
<date_finish></date_finish>
<priority></priority>
<enable_discussion></enable_discussion>
<enable_rewatching></enable_rewatching>
<comments></comments>
<fansub_group></fansub_group>
<tags></tags>
</entry>

Nov 23, 2013 5:00 AM

#72

ErgoSis

Offline

Jun 2009

You should try to import result from the original script to MAL directly by adding <myinfo> block to the resulting XML. The problem resolved with AniDB in the middle is missing MAL IDs which is not the case with latest script. Don't know waht's wrong with "Not implemented" errors. Should work that way. Maybe try to remove anime id from POST data (params var).

A story has no beginning or end; arbitrarily one chooses that moment of experience from which to look back or from which to look ahead.

Nov 23, 2013 6:40 AM

#73

Prism Nana Fanshare: let's share the awesomeness!

Offline

Nov 2013

I tried importing the original list first thing. It asked if i was missing "<update_on_import>1</update_on_import>", which i wasn't. Not exactly sure what your referring to with the <myinfo> block?

However, i figured out what the issue is with the 'not implemented' error. If you have an anime already added in your list, you can't add it again (obviously), so it throws an error. The very first anime in the list was already added in MAL, so tried removing it and it worked until it got to the next one that was already in MAL.

So i updated the script to update the anime instead if the adding fails. Now everything works perfectly. So even if you already have it in you MAL list it will essentially be overwritten by what it read from the AP list. Its not very elegant, as it would be better to get a MAL list and then compare and only update whats needed, but it works for a once off thing. I also don't know how to get your own MAL list (except perhaps exporting it manually and reading the file).

Anyway, here it is for anyone else: http://pastebin.com/ngpvnLTT

Thanks for all the help ErgoSis!

Dec 3, 2013 12:14 AM

#74

kureshii

Offline

Jun 2013

After spending a couple of days getting animedb_id's from MAL search, I've finally gotten my AP list into MAL as well. Just a note: you need the <myinfo> section for the import to work, at least that was the case in my experience. Particularly this section:

<myinfo>
<user_export_type>1</user_export_type>
</myinfo>

I have a working python2.7 script that will generate <series_animedb_id>, as well as a lookup file for matching A-P titles to MAL titles and animedb_id's. Poke me if you're interested.

Dec 25, 2013 10:36 AM

#75

Haidb

Offline

Dec 2013

Although off-topic for the thread; I recently took the code in this thread and adapted it to scrape ANN anime lists to import them into MAL.

The script currently only scrapes the "category 3" ANN list, which is usually anime which is under "Seen All". It works fairly well, however, I am sure someone who is more familiar with Python 3.3 or Beautiful Soup could improve the code to make it far more generally useful.

The Script: http://pastebin.com/gzmgSBYZ

Thank you to all who have contributed to this thread. I've found the discussion here immensely useful!

Dec 28, 2013 4:58 PM

#76

sampleslayer

Offline

Nov 2012

jojovonjo said:

sampleslayer said:
Why there is no "the god of high school " manga to add to my list?

If I remember right. It does not meet database guidelines.

what do u mean? it is not manga?

Dec 28, 2013 5:05 PM

#77

IntroverTurtle

Offline

Nov 2010

26413

sampleslayer said:

jojovonjo said:

sampleslayer said:
Why there is no "the god of high school " manga to add to my list?

If I remember right. It does not meet database guidelines.

what do u mean? it is not manga?

It is an unpublished webtoon so it's not on here.

Short of the day: Monotonous Purgatory(MAL) ✰Public Domain Club | One Piece Club✰

Jun 25, 2014 6:59 AM

#78

AkahanaAmaya

Offline

Feb 2014

it keeps giving this error:

Exporting rough variant of myanimelist format...
================
http://myanimelist.net/api/anime/search.xml?q=11eyes
Decoding of anime data failed. Error: not well-formed (invalid token): line 5, column 147
================
Request to http://myanimelist.net/api/anime/search.xml?q=30-sai+no+Hoken+Taiiku failed. HTTP Error 401: Unauthorized

This is the script I used: http://pastebin.com/hyhV9uWJ
Any suggestions?

Jun 30, 2014 8:54 AM

#79

AkahanaAmaya

Offline

Feb 2014

By any chance, did MAL disable the search api? I think this error is related to the search.xml file.

Prism Nana Fanshare: let's share the awesomeness!

Aug 19, 2014 8:24 AM

#80

cloudtail

Offline

Dec 2013

Using the version of the script from ergosi's post (http://pastebin.com/igVyHeXL) the series that were added ended up being invisible due to the status (Completed, Watching etc) not working properly. I fixed it using this :

found = animeItem.find("span")
found = str(found)
i = found.index("status")+6
animeStatus = found[i:i+1]

if animeStatus=="1":
status="Completed"
#"Watched"
elif animeStatus=="2":
status="Watching"
#"Watching"
elif animeStatus=="3":
status="Dropped"
#"Dropped"
#status=animeStatus
elif animeStatus=="4":
status="Plan to Watch"
#"Want to Watch"
elif animeStatus=="5":
status="On-Hold"
#"Stalled"
elif animeStatus=="6":
status="Dropped"
#"Won't Watch"

instead of this :

animeStatus = animeItem.find("td","tableStatus").text.replace("status box","").replace("t", "").replace("n", "").replace("r", "").replace(" ", "");
if animeStatus=="Watched":
status="2"
#"Completed"
elif animeStatus=="Stalled":
status="3"
#"On-Hold"
elif animeStatus=="WanttoWatch":
status="6"
#"Plan to Watch"
elif animeStatus=="Won'tWatch":
status="4"
#"Dropped"
else:
status="1"
#"Watching"
#status=animeStatus

Aug 19, 2014 9:12 AM

#81

Offline

Oct 2009

7669

AkahanaAmaya said:
By any chance, did MAL disable the search api? I think this error is related to the search.xml file.

No, you (or script) just need to authorize first

Dec 6, 2014 5:27 PM

#82

MazrimTaim

Offline

Dec 2014

I'm having some trouble getting this script running.

On Python 3.4 the BeautifulSoup install gives me this:

Python 3.4.2 (v3.4.2:ab2c023a9432, Oct 6 2014, 22:15:05) [MSC v.1600 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from bs4 import BeautifulSoup
Traceback (most recent call last):
File "", line 1, in
File "C:Python34libsite-packagesbs4__init__.py", line 355
print soup.prettify()
^
SyntaxError: invalid syntax
>>>

On Python 2.7 BeautifulSoup seems to install. I can run the script and it asks for my username. When I press enter the script closes immediately. Text is output to the console but it closes too fast for me to read. No XML is created.

I have modified the script as instructed. My list is 9 pages long http://www.anime-planet.com/users/MazrimTaim/anime

Anyone have any ideas?

Edit: Managed to Print Screen the Window as it closed, looks like the 403 error which I think was mentioned previously in this thread. Trying another script in 2.7 closes the dialogue immediately with this error: http://gyazo.com/a6e298bbaac312eea41c50b89736ed6f

The script in question for this is: http://pastebin.com/x0gHs2Bb

Edit 2: So it looks like the fix for that involves using Python 3, for which BeautifulSoup does not install for me. On 3.3.3 I get the same error as listed above.

MazrimTaimDec 8, 2014 10:49 AM

Jul 6, 2015 10:20 AM

#83

calophi

Offline

Jun 2008

Anyone still hanging around in here? I am trying to use this script: http://pastebin.com/igVyHeXL

I get the following error:

File "igVyHeXL.py", line 127, in
xmlData += "t" + str(int(float(animeItem.img["name"])*2)).replace("t", "").replace("n", "").replace("r", "").replace(" ", "") + "n"
TypeError: 'NoneType' object is not subscriptable

Jul 6, 2015 11:43 AM

#84

calophi

Offline

Jun 2008

I fixed the issue - AP I guess changed where their rating is. But now everything added under "watching" and I can't figure out how to make it update the status. Ugh.

Jul 7, 2015 4:48 AM

#85

calophi

Offline

Jun 2008

Okay, I got it fully working if anyone still cares. Here is the updated script. I don't write python so I'm sorry if it isn't super purdy, but it works.

http://pastebin.com/p2ygYZHa

Aug 22, 2015 10:16 PM

#86

MoFried

Offline

Aug 2014

calophi said:
Okay, I got it fully working if anyone still cares. Here is the updated script. I don't write python so I'm sorry if it isn't super purdy, but it works.

http://pastebin.com/p2ygYZHa

I want to have your babies so badly right now.

i.e. Thank you.

Oct 16, 2015 3:52 PM

#87

bekr99

Offline

Oct 2015

is there a way to import the anime list from kiss anime to here???

Oct 16, 2015 4:45 PM

#88

Offline

Oct 2009

7669

bekr99 said:
is there a way to import the anime list from kiss anime to here???

Ask on that site

Oct 29, 2015 1:06 AM

#89

Gygun

Offline

Jun 2010

Thanks man! Worked fine!

Dec 1, 2015 11:59 PM

#90

Offline

Apr 2015

Could somebody give me a truly play by play walkthrough of doing this, starting from me reading this post to my anime-planet list being imported to MAL? Pretty, pretty please

Dec 2, 2015 12:43 AM

#91

Offline

Oct 2009

7669

moelump said:

Could somebody give me a truly play by play walkthrough of doing this, starting from me reading this post to my anime-planet list being imported to MAL? Pretty, pretty please

Have you read what is written there?

#Additional info and packages:
# Python 3.3.3 - http://python.org/download/
# BeautifulSoup - http://www.crummy.com/software/BeautifulSoup/#Download
#Tips:
# * You can leave your MAL username empty if it's the same as on AnimePlanet.
# * To install BeautifulSoup unpack it anywhere and type "setup.py install" in the console from that folder.
# * In order to successfully import the exported Anime-Planet animelist to MAL, first export MAL animelist,
# and copy the block just after tag.

Dec 2, 2015 2:11 AM

#92

Offline

Apr 2015

Serhiyko said:

moelump said:

Could somebody give me a truly play by play walkthrough of doing this, starting from me reading this post to my anime-planet list being imported to MAL? Pretty, pretty please

Have you read what is written there?

I'm getting stuck at the installing BeautifulSoup part

ANd when I try to run his scripted it says there is no module named request

moelumpDec 2, 2015 2:21 AM

Dec 2, 2015 7:46 AM

#93

Offline

Oct 2009

7669

moelump said:
I'm getting stuck at the installing BeautifulSoup part

Do you know what console aka Command Prompt is? Hit WIN+R, type CMD (and hit enter), and type in the window that appears

C:\\beautifulsoup4-4.4.1\setup.by install

Change "C:\\beautifulsoup4-4.4.1\" with wherever you extracted the files (take note that there is beautifulsoup4-4.4.1.tar archive packed inside beautifulsoup4-4.4.1.tar.gz archive)
Simply double-clicking on setup.by from the explorer won't do
By the way, the last Python version that supports Windows XP is 3.4.3

SerhiykoDec 2, 2015 8:01 AM

Dec 2, 2015 11:47 AM

#94

Offline

Apr 2015

Serhiyko said:

moelump said:
I'm getting stuck at the installing BeautifulSoup part

Do you know what console aka Command Prompt is? Hit WIN+R, type CMD (and hit enter), and type in the window that appears

C:beautifulsoup4-4.4.1setup.by install

Change "C:beautifulsoup4-4.4.1" with wherever you extracted the files (take note that there is beautifulsoup4-4.4.1.tar archive packed inside beautifulsoup4-4.4.1.tar.gz archive)
Simply double-clicking on setup.by from the explorer won't do
By the way, the last Python version that supports Windows XP is 3.4.3

Thank you, thank you

And now because I'm a big ol' idiot how would I go about using the anime-planet export script in a way that it works?

Dec 2, 2015 12:30 PM

#95

Offline

Oct 2009

7669

moelump said:
Thank you, thank you

And now because I'm a big ol' idiot how would I go about using the anime-planet export script in a way that it works?

Sarcasm won't help here
Are you still stuck at installing BeautifulSoup, or have you managed to install it? If yes, does the script run? Does it ask you for your MAL username, password, etc.?

Dec 2, 2015 1:00 PM

#96

Offline

Apr 2015

Serhiyko said:

moelump said:
Thank you, thank you

And now because I'm a big ol' idiot how would I go about using the anime-planet export script in a way that it works?

Sarcasm won't help here
Are you still stuck at installing BeautifulSoup, or have you managed to install it? If yes, does the script run? Does it ask you for your MAL username, password, etc.?

there was no sarcasm though

and yea I have it installed now, just asking what exactly to do next

Dec 2, 2015 2:27 PM

#97

Offline

Oct 2009

7669

moelump said:
there was no sarcasm though

and yea I have it installed now, just asking what exactly to do next

Oh. Well, copy the above script and save it in a file ending in .py (for example, ap.py). You can do that with a simple notepad. You can save it in a .txt file first, just make sure to rename it to .py after that (if you have problems with this, that probably means that system hides file extensions, here's how you can unhide them: http://www.google.com/search?q=windows+show+file+extension)
You should be able to run it now. If you did everything right, it will ask you for your username

SerhiykoDec 2, 2015 2:38 PM

Dec 3, 2015 2:58 AM

#98

Offline

Apr 2015

It's saying no module named bs4 when I try

Dec 3, 2015 7:54 AM

#99