Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Grimoire Archive
#1
Information 
[Image: AB210_C0_BA84_FAD144_FB745_B5_DAC3_C32_E616_CA298.jpg]

General Thread to handle Animegrimoire archive.
Archival method which we used here are quite unusual than regular backup. The main keys are; we'll never share direct link from backup cloud into public, we're storing files that completely unique to avoid cloud provider from taking down files based on matched file hash (just like what MEGA, Dropbox, Google Drive did), but these method are come with it's price. So far, we're limited with computing power and bandwidth limit to push file into public per IP. Like, MEGA, for example, it's safe to say that we're limited into continuous 1GB per hour.

Here's a simplified diagram to explain most of backup process:
Show Contentbackup_diagram:
Most of backend processing are done with Python and Bash, while for CLI uploads into public sharing are heavily rely on megatools. another viable options for upload are gdrive-cli but so far we're hesitant to use it.

By 01st October 2018, we're reaching 222.6Gb total.
Show ContentArchive Total:

As for file list, you could check it here. Each sheet name are self-explanatory, but i want to point out that Relics are sourced from old Cyber12 encodes, and most of them are 360p. Freezed itself, is taken from dlc/Completed Season and it's outside Legendary Aurora's account.



This thread means to encourage Anime Uploader submitting their encodes here to be preserved. As long as we have certain backup of your thread, we will help restore yours in case it got deleted due expired urls, or another case such as DMCA takedown notice. Starting Fall 2018 season, i'll be archiving your On-going season after it's finished for a week without your request as long as you met these certain requirement:
  1. Correctly following main guideline.
  2. animegrimoire watermark are remain unmodified with additional text such as "encoded by ---", "uploaded by--" .etc
  3. used resolution are restricted to 720p without involving upscaling. Except: it was coming from DVD rip (576p) and downscale from 1080p BD to 720p. modifying these are considered as outside of guidelines (such as, you're clearly using 720p sources but you're downscaling them to 576p)
  4. File name pattern, see:
       [Image: Filename.png]
    Quick guide to massively rename files are here, or if you want to just use saved preset are way more quickly which you could see here. You could download it from main guide thread. Overall encoding process are like this
  5. folder download link must be available in the end of episode releases.

OK for a quick rundown. First point is obvious, especially for On-going season.
Second, it should be done automatically if you are following the main guide. other way, i could skip it as long as i could verify it's availability in your thread while your ID are 1 month or older, and you own more than 1 (One) Thread, as for additional text/note, they're outside of our support, we simply won't bother to archive what isn't dedicated for this forum.
Third, fifth are self-explanatory.
Fourth, well this is troublesome. I asked this for the sake of archive tidiness, there are too few people that interested for Episode Title which could be accessible in their wiki, or public anime list site (if they're really interested). And another thing which worth to mention, replacing plain dash with _-_ are plain messy if it's going to archive list, and never expect i'll do it for you.

if in later date i skipped your ongoing season into backup (because you somehow missed the minimum requirement), you could always sent me a PM for archive request. Just make sure you met minimal requirement above and you'll be fine. Once archiving is done, i'll notify on your requested thread. Stay safe!

[Image: 7_D4_F69_EC105910203670_A00_BE9_F155_E6496_C3218.jpg]
[Image: animegrimoire.gif]
Reply
#2
I like this thread. Thank you for sharing this info and i will try my best to follow your guide.
¯\_(ツ)_/¯
Reply
#3
Now if you look on it, this one are seems just another copycat from CryFS. well not to mention the creator are heavily influenced by it  Big Smile

Sure it just splitting files into encrypted container, and all files metadata are obfuscated. but the best part of this ingenuity is to bridge stdout part into cloud upload API, which if you understand the simplified diagram, it just "a better mv" which rsync are excel on it too. other part of it are just databases for documenting purpose, and possibility to pull as efficient as possible for each container (because there are fair chance for files overlapping).

The only thing left are just the backup storage i guess (well that's the very problem in first place). For now it's safe in magenta, mega, and recently gdrive also tested, hopefully in future we could afford 0x0, or another big guy like IPFS. so far those are way better (well, self-hosted) as torrent are outdated on these days because blockchain memes (also, rip demonoid)
[Image: omae-wa.png]
Reply
#4
(07 October 2018, 07:32 AM)Lynx Wrote: I like this thread. Thank you for sharing this info and i will try my best to follow your guide.

You're welcome  Big Grin

(07 October 2018, 08:06 AM)Internet Explorer Wrote: Now if you look on it, this one are seems just another copycat from CryFS. well not to mention the creator are heavily influenced by it  Big Smile

I won't deny that this project looks similiar to CryFS, but this one started from a simple puzzle game. lol

(07 October 2018, 08:06 AM)Internet Explorer Wrote: Sure it just splitting files into encrypted container, and all files metadata are obfuscated. but the best part of this ingenuity is to bridge stdout part into cloud upload API, which if you understand the simplified diagram, it just "a better mv" which rsync are excel on it too. other part of it are just databases for documenting purpose, and possibility to pull as efficient as possible for each container (because there are fair chance for files overlapping).

you're missing the best part. this thing works on single py file, and way much lesser libraries compared those container FS. it's made for Fire & Forget, as i'm pretty sure i'm the most lazy person in this forum to maintain things

(07 October 2018, 08:06 AM)Internet Explorer Wrote: The only thing left are just the backup storage i guess (well that's the very problem in first place). For now it's safe in magenta, mega, and recently gdrive also tested, hopefully in future we could afford 0x0, or another big guy like IPFS. so far those are way better (well, self-hosted) as torrent are outdated on these days because blockchain memes (also, rip demonoid)

uh for this part, null pointer looks ok (heck it just uses curl to upload) but for now we're not gonna deploy private server with big storage (it's pricey in long run), and also for IPFS, i'd rather not using animegrimoire as test rabbit. their project is inferior to mine (precisely, my needs).

For google drive, for now i already confirmed that your butt is safe if your folders has no shared link (in term of unencrypted files) but your account will get flaged (it has been years they doing this for files that ever smoked DMCA reports), so whenever you start creating sharing link from your flagged folders, you'll get suspended.

I'd say this one is viable option, but in implementation i'll proxy disks using sshfs in aws cluster before uploading them using gdrive (even though it's in encrypted form). once you look at botnet, you'll never look back, so sanitize your hands, kids  Big Smile
[Image: animegrimoire.gif]
Reply




Users browsing this thread: 1 Guest(s)