Slashdot Mirror
Ask Slashdot: Huge Digital Media Libraries
An anonymous reader writes "Like many slashdotters, I have several TB of digital media: music, books, movies, tv shows, games, comics, you name it. I've put it all in a few HDs, but handling it all has proven to be less than optimal. I'm covered when it comes to music, since [pretty much any music player/library manager] allows me to quickly find songs by interpreter, album, genre... For everything else, all I have is a series of hierarchical folder structures, but hierarchies have limitations. I can find Blade Runner easily, but what if I wanted all of Scott Ridley's films? Where is 'Good Omens', in the Terry Pratchett folder or in Neil Gaiman's? Furthermore, in a collection with hundreds of similar items, it would help to have some extra clues such as covers (for comic books) or synopsis for TV shows' episodes. Do you have any software to help you handle digital media libraries? Specialized software (say, something that only work for comics, something else for movies), or generic media libraries? Opensource alternatives are preferred, but commercial software is fine as well."
I tend to just use directories of symlinks on the odd occasion where I want a logical collection of something. Kind of the hacked file system equivalent of a playlist. I can even put additional detail in the symlink name that I would leave out of my “main tree”. Generally though, a simple hierarchical structure has worked fine for me and my 6+ TB of media. If I anticipate wanting to search for something down the road, I also sometimes put it in the file name (indexed by slocate every night).
You are probably looking for a tagging/metadata tool but I think the problem with those is you have to obsessively tag/provide that metadata and they aren’t going to integrate with all your favourite viewers and such. It just seems more trouble than it’s worth to me, but with different levels of motivation and borderline OCD, it could work very well (and probably does for many).
You can admit it. Slashdot understands that you have a large *personal* media collection *AHEM*.
"I like it when the red water comes out.."
There's a little box at top-right of the file explorer window. You can type words there...
No sig today...
Use data crow and make a container for every HDD. It works for music, movies(imdb details import) and software. http://www.datacrow.net/
Just use what the libraries use:
Fedora
What you're looking for in general is either a repository (if you want it to manage the files) or a catalog (if you want it to just track info about the files). A catalog might also be called a 'registry' when dealing with sciences archives, where the term 'catalog' is used for something else.
For more options, see any of the following lists on wikipedia:
http://en.wikipedia.org/wiki/Category:Free_institutional_repository_software
http://en.wikipedia.org/wiki/Category:Digital_library_software
http://en.wikipedia.org/wiki/List_of_next-generation_library_catalogs
Build it, and they will come^Hplain.
Used iTunes? It's like that for books but less bloated. Syncs to many devices, and can scrape RSS feeds from magazines, build them into EPUBs and sync them to an ereader, like a text-based podcast. This works surprisingly well, superior in some ways to reading the same material on the Web.
And it's FLOSS.
http://calibre-ebook.com/about
How much of it do you really re-watch? You'll spend the rest of your life transferring it from medium to medium. Is it worth it?
I never used it because I don't have TB of content but XBMC seems to be be something that could get your attention.
Granted database management is part of my day-job, but it really doesn't take all that long to put together. The tedious part is data entry. Movies and books weren't so bad with a imdb and Amazon scraper script. But data cleanup still takes forever.
I'm likely going to be flamed and modded into purgatory for this, but I use iTunes for most of this-at least, for music and videos. Some PDFs are starting to go in there if I want access to them on the go on my iPhone or iPad.
I understand that Apple's universe isn't perfect, but for me it all works together pretty nicely. I replaced my high-maintenance, increasingly noisy, power-hungry media PC with a second-generation Apple TV. This works great except that it won't play many video formats. Therefore, I've had to go through the obnoxious step of using VideoDrive to transcode videos into an Apple-approved format. However, it's not the end of the world.
Otherwise, I guess everyone's different, but personally I want to spend my time doing fun stuff like riding my bicycle or spending time with my family, not categorizing my "vast media collection". I guess I'm just getting old, but iTunes does a good enough job, with less work than any DIY system I've successfully maintained in the past.
www.clarke.ca
Ridley Scott
Well, I guess the key starting point is what operating system are you running on? But for my rather extensive movie and tv collection, on windows I've found Mediaportal: http://www.team-mediaportal.com to be fantastic with its range of plugins to cope very well with TV and movies. Specifically the MP-TVSeries plugin for TV, which interfaces with the TVDB and gets all the information you suggest you want about your shows, including actor information, fanart, banners, posters, thumbnails, and the list continues. You can also sync all these details with an online tracking website such as the relatively new trakt.tv so you and your buddies can see what you recently watched (all done automatically once you have an account at the website and configure the plugin). There is an equivalent for movies called movingpictures which does pretty much the same thing, and you can set up your own categories for sorting too. Mediaportal is a spin-off from XBMC, but is also opensource, and free, but only runs on windows (and makes a very good HTPC software on the whole). So if you're not running windows, maybe try XBMC instead. That is just my personal experience, your mileage may vary.
With so many files, I don't treat my system as a filesystem, and more like a Google-type search. I imported all my media into Drupal as page nodes and hotlinked to the backend files. Where possible, I had has much metadata as possible included about each file. Time/date, subject, type of media, keywords, where, descriptions if they were entered, and searchable text. Navigating is then done by media type, searching, and browsing through various keywords. Drupal then presents the media in the browser. If you wanted to get fancy you can reuse the metadata to present lists using the views module. For direct access via the application (like the audio player), that is when I go to the backend, but generally at that point I know what I want after going through the website.
Here in Canada, Libraries are being asked to repurchase eBooks they have bought and distributed after about 30 people have "checked it out". Its all a bit of a farce, all things considered.
I went to battle M.C. Escher, but drew a blank.
You meant Ridley Scott and not Scott Ridley, right?
Building Better Software
Honestly, if you don't mind relinquishing some control over the media (and risk that Netflix chooses not to carry some obscure cult classics) using those cloud based "delivery on demand" options makes a lot of sense. They're certainly attractive to non-techie types that have no interest in managing media servers or migrating storage mediums every decade.
> One has to wonder where he GOT those terabytes of digital content... ...looks at shelf full of leather bound CD/DVD binders. "Sci-Fi" spans 4 binders. "Comedy" spans 3 binders.
A Pirate and a Puritan look the same on a balance sheet.
For books - Calibre, let it convert things to epub format and let it deal with the directory structure. For comics, "ComicRack" is the absolute best. It allows cover view, can convert cbr/cbz/pdfs (though it prefers cbz for metadata), and allows the importation/scraping of metadata and saving it directly to the file.
I use MediaTomb for my digital media library. It manages all my music, videos, and photos and is quite extensible through scripting if you are familiar with JavaScript. Then I use XBMC or my PS3 as the front-end to MediaTomb. I'm currently managing over 1 TB of data without issue. I cannot speak for other media, such as books, as all my books are still in dead tree format.
this.
and since the information database MyMovies pulls from is community maintained and vetted, it is far superior to the metadata that you will get from even good metadata sources like IMDB.
You can use MYMovies for frontend interface on Windows Media Center or you can utilize one of a number of other front ends and only use the MyMovies Collection Management DB backend.
The nice thing about this product is that it does store actor/director/crew information, so you can easily pull up any director and see all of the movies in your collection and with a simple click then pull up any of those movies for immediate playback.
I have 20TB+ of movies/tv shows in my collection (yes i own them, i buy dvd's instead of paying for cable/satellite service) and utilize the MM front end and backend and everything is fast to utilize.
Another nice thing about MM is that it is completely free to use for home use and allows for all of the "nifty" special features people like, including screen customization, movie art, etc.
their newest version (still in beta till the end of the month) also now includes specific features for handling tv shows ripped to individual episodes if you are one the "acquires" these.
I've got Photoshop Elements. The editing is certainly overkill for what you're talking about, but the library features are quite good. The key is that there's a separate database of tags and metadata which can be sorted and searched like...a database.
I'm sure there are similar things out there, but I think the key is to try a handful to see how they do it, and narrow down your choices accordingly.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
Take a look at tagged filesystems. You can do the same thing by hand using symlinks but with much greater pain.
http://www.tagsistant.net/
http://nascent.freeshell.org/programming/TagFS/
http://semanticweb.org/wiki/SemFS
The following are not really filesystems. You need to use specific programs to search the tag space.
http://pages.stern.nyu.edu/~marriaga/software/oyepa/
http://blueslugs.com/2005/07/12/tag1-delicious-style-file-tagging/
"Liechtenstein is the world's largest producer of sausage casings, potassium storage units, and false teeth."
I certainly won't "flame" you for this suggestion. As maligned as the iTunes software is, I think its ability to index media and very quickly retrieve it by a number of different fields is pretty darn good -- especially for a program you can download free of charge for both Windows and the Mac.
As a Mac user myself, I started using another free program to manage my movies and saved TV shows though. I really like Plex (www.plexapp.com) for the purpose. It doesn't have the restrictions on playable video formats that iTunes has, and has a great UI to serve as your media center via a remote control.
I believe the latest update to Plex added some interesting, if slightly obscure, functionality -- like the ability to search the subtitles of your movies for specific strings, too.
With Calibre (http://calibre-ebook.com/) you can deal with the book problem the same way you would use iTunes to catalog music and video. It is available for Windows, Linux and OSX. I have personally used it for both OSX and Windows for a few years and it has never let me down.
The video problem is much harder because the tagging is nowhere as mature as what we have available for music. What really drives me nuts about this is that there is no consistent way to apply parental ratings to content in a way that it is recognized by OSX and Windows. This keeps me from sharing my videos across the home network since there is no way I can easily block certain videos from my son's Xbox and his iMac. I would have to manually set play lists, which is a lot more work, it would be nice if I could tag content as PG-13 or above and let the Xbox use its built-in content ratings mechanism.
Pedro
----
The Insomniac Coder
You will apparently be shocked to discover this, but some people actually re-read books! And re-watch movies!
There are a handful of movies I've seen over a dozen times (Not including the family ones my son demands we watch on occasion). Some of them I still haven't got all of the juice out of (The Cook, The Thief, His Wife, and Her Lover), and some of them I just enjoy watching over and over (The Princess Bride, Amelie). I reread Fool On The Hill and Lord Of The Rings about every second year. Our library is tucked away in the basement out of the way, and isn't going to impress anyone. We have as many books as we do because we enjoy them, read them, and share them with friends.
Purging is great, but don't apply your opinions to the rest of us. Some of us have different points of view.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
ID3 is a de facto standard widely used for music. It is targeted at MP3 file format but many alternative music file formats also support embedding it.
see http://en.wikipedia.org/wiki/ID3 for more details.
Now, for everything outside music, the need/usage/online-store haven't managed to create a standard (even de facto) for meta-tagging files.
Since all formats might not support metadata, the simplest would be to use the file system meta-data/extended attributes.
see http://en.wikipedia.org/wiki/Comparison_of_file_systems for the list of FS supporting extended attributes.
As with music, you'll need dedicated applications to edit and browse those tags. Since you already have a folder structure, you could jump start those tags with the information that can be retrieved from the folder names. For movies and books, you might be able to complement those tags from IMDb or Amazon.
Sig (appended to the end of comments you post, 120 chars)
I own the David Attenborough Life Collection. It's 24 DVDs. Even assuming they are the single-layer type, that would be:
112.8 Gb. Roughly.
Now say I own, say, several boxsets of comedies, series, documentaries, a few dozen movies, maybe even a couple of dozen free promotional DVD's with full-feature films on them that are given out when the film stops selling.
Now, *NOT* including anything I've recorded from TV / Movie channels for my own consumption, not including any home videos, not including ANY Blu-Rays, etc. I can *EASILY* fill terabytes of data without even blinking an eyelid.
Hey, I could probably fill a terabyte or two with DVD images of cartoons (proper children's cartoons) and stuff I watched when I was younger (I have the complete set of Dangermouse, Batfink, etc.) and that's hardly something I go out and buy every day and keep buying. A terabyte, or even half a dozen terabytes, is NOTHING. It's just when you have to copy it all into a single place, like this guy is doing, that it appears to be a lot.
It's just that he's obsessively backing it up and/or converting it to free formats so that he can just browse from a media library, like the ones he desires, so it's all on one hard disk (or more likely RAID). It's not "abnormal". It's nowhere near "evidence of piracy". It's just a media collection stored on disk instead of the original DVD's.
No. "technically" DVD and BR rips are fair use.
Your attempt to hand Apple a stranglehold on the future on a silver platter is not appreciated.
A Pirate and a Puritan look the same on a balance sheet.
I looked at exactly this problem and came up with my own custom solution.
I wrote a Perl script that queried IMDB, there are simple CPAN libraries out there. The highest rank search based off the filename was always the correct movie.
Then I pulled out the director, lead actors, proper title etc. Any details that you actually care about.
Finally I created the directory structure for each detail and put a hardlink to the file. The original files were all kept in a single flat directory for storage, symlinks would work just as well if you prefer.
The end product is exactly what you are looking for: ...
Media
-> Directors
--> Ridley Scott
---> Actual movie file 1
---> Actual movie file 2
--> Tim Burton
---> Actual movie file 1
---> Actual movie file 2
-> Actors
-->
No issues with duplicates or anything like that. No requirement for your media player to understand some sort of database. No problems sharing it across a network filesystem.
All less than a page of Perl. Unfortunately the code is currently inaccessible to me.
Depends on the Country the Poster is in!
In the UK there's NO "Fair Use" clause for copying digital media.
You can't legally rip that CD/DVD/Blu-ray for personal use...
Laters Sol "Have you found the secrets of the universe? Asked Zebade "I'm sure I left them here somewhere"
Probably the best interface and organizer of movies, TV shows, etc., that I have seen. Add in the Moving Pictures an MP-TVSeries plugins and you are all set with an awesome interface.
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
MediaPortal is the windows way to do it. GPL'ed and uses MySQL or MS Access (MySQL is preferred).
We were all warned a long time ago that MS products sucked, remember the Magic 8 Ball said, "Outlook not so good"
XBMC
Works great. It indexes my 2TB of movies and Tv shows... although I am trimming the TV shows down, no reason to watch season 1 episode 3 of glee more than once.
Plus how often do you REALLY need to sit down and watch " the last 3 Sigoruny Weaver" movies.. you browse by Genre 99.99783% of the time. People all claim they want to search video content by director, actor,etc but in reality they never use that. And yes I know what I am talking about I have installed and helped with several hundred installs ofhttp://www.kaleidescape.com/ media systems... after a year you discover that by title and by genre are the ONLY search methods used. These are owned by people far FAR richer than all of you here on slashdot combined and have very little time to waste.. they dont search by director or actor ever.
Honestly step one is to be realistic about your media organization. Do not go nuts and cross reference on everything...
"Show me all the films that had a Audi R6 in it", "show me all films that were shot on a Tuesday in France when it was raining....."
Do not look at laser with remaining good eye.
Some stuff gets "expired" from Netflix. The same goes for Hulu.
When you decide you want to watch something again, it might not be readily available.
This is usually the reason for having your own media collection. This seems pretty obvious for Music but it seems like a real revelation for anything else.
A Pirate and a Puritan look the same on a balance sheet.
You'll spend the rest of your life transferring it from medium to medium.
Nope. Actually, every iteration gets exponentially faster. So, recording anything analog means 1:1 recording speeds. But for CDs, it's about 16:1. And once the data is on a hard drive, it might as well be instant.
Is it worth it?
"You ever see The Wire?"
"Nope, never got around to it."
"Here's a copy."
NOTE: the preceding hypothetical conversation assumes you have friends.
Having everything I want locally saves me time searching the internet for it
Agreed.
:D
Perhaps I'm the odd man out, but I do like going over my media collection (which is automatically sorted) and just trimming the fluff everyone once in a while. Making sure files are named right, getting all the movie trailers, filling in a gap where I'm missing a season, and so on.
It's not really that it's a compulsion, but more of a hobby. I like having a movie and TV show collection, and the fact that I can have a digital one sitting on a RAID array increases the usability and coolness factors.
Organize TV Shows with Sickbeard. Organize movies with either Media Center Master or MyMovies. Better stuff for movies undoubtedly exists, but I'm not too sure what it would be.
Also, if you want your computer to surprise you with new content and you're not afraid of complex config files, give FlexGet a try.
Finding time to watch all of it.... that's the real kicker
Boot Windows, Linux, and ESX over the network for free.
PVD sounds like it would meet your needs. It's free and can scrape multiple sources for movie/TV show information and posters. It uses SQlite for the database.
http://www.videodb.info/forum_en/
Eagles may soar, but weasels don't get sucked into jet engines.
J River Media Center is BY FAR, the best commercial tagging software available. It does have its limits around HTPC use (its theater view is lacking). However, your tagging environment is fully customizable. You can create views based on custom metadata if needed. Most programs, tagging is contained within a box, with J River, there is NO box. It pairs nicely with XBMC for HTPC application. Tag in J River. Play in XBMC. (Get the android apps for both, you'll love it).
Your son isn't going to die if he sees some gore or some tits.
I have two questions. How old are you and how many kids do you have? My guess is the answers are "young" and "zero". So I guess I should let my 6 year old son just watch ass-to-mouth porn then. I don't think so. Contrary to your simplistic world view, there are some things that children are not ready for. Even this weekend, I was watching Phantom Menace with my oldest son (he's 6) for the first time. It led to all kinds of very difficult questions about death that he was probably only ready for 80% of ensuing discussion.
I'm currently 38 and I've had ready access to porn since I had my first 300 baud modem in 1986. I'm no prude and certainly don't think that children should be shielded from everything. But your speaking in absolutes that kids will either be ready to comprehend it or be disinterested in it just sounds pretty dumb.
I'm a big tall mofo.