PDA

View Full Version : Long term data archival - storege guys please help



Pelias
May 13 2011, 03:04:04 PM
Hello [strike:kweu2j1t]S[/strike:kweu2j1t]FHC!

I decided to finally scan all slides and negatives my parents made. Some of them are over 20 years old, so it really is about time.

However - I ran into problem - how to store the scanned data? The whole idea with scanning is to preserve the images, so in next 20 years, they will still be there. We're looking at about 120GB so it's not trival.

From my personal experience, typical dvds (even Verbatim or TDK) can become fucking unreadable, worthless pieces of shit after mere 5 years (it depends on production batch, some TDKs have died on me in 4 years, some no names are still readable after 8y, but the whole point is that I don't want to risk). Besides, it would have to be an awfully lot of DVDs.

HDDs - they tend to die randomly and afaik, producers don't really guarantee they will work after 2 year warranty runs out. Doesn't seem like optimal solution for 20 year storage.

Online storage - with 120GB, it would cost fucking bonkers and judging from typical internet startup lifetime, 99% of current cloud services will be long dead in next 10 years. Also, they might accidentally delete my files because they couldn't tell png from some fucking Call of Duty IX or Quake 7 rips, or some other idiocy that tends to happen on the net.

So, how do pros do it? How can I do it? I'm awfully close to deciding fuck it all and just keeping the analogs:/

Aea
May 13 2011, 03:18:08 PM
This is a puzzling problem for many people and an extremely expensive thing in the long term.

My personal approach may be S3, but that's $.149*120*12*20 = $4,291.20 for 20 years. *

The other approach would be to always keep multiple digital copies personally and move them to new media after 5 years (and ensure you have multiple copies). For instance get a safety deposit box (some banks give these to you for free depending on balance), or a safe, and 2 (or so) portable 120+GB HDs (pretty cheap). Make two copies, different locations, as well as keeping it on your computer. Whenever you get a new computer, copy the data to it. Replace the drives every 5 years, etc.


The pros use solutions like redundant data storage (think S3), or microfilm, or using expensive physical printing (e.g. a good quality DVD should last 10+ years IIRC).

* Edit, this may seem like much now, but it really isn't. $214.56 in todays money (per year) seems like a lot, but history tells us with inflation that it won't be that much and the price of storage is _always_ decreasing.

Pelias
May 13 2011, 03:29:23 PM
dude, that's like 1/3 of avg yearly salary after taxes, so sadly I ave to say no to 3S.

Can hdds survive that long if they are not turned on? Can I safeguard it against bank doing anything funny involving magnetic fields in their deposit box safe? In fact, what are optimal conditions for hdd and dvd storage?

Aea
May 13 2011, 03:36:58 PM
dude, that's like 1/3 of avg yearly salary after taxes, so sadly I ave to say no to 3S.

Can hdds survive that long if they are not turned on? Can I safeguard it against bank doing anything funny involving magnetic fields in their deposit box safe? In fact, what are optimal conditions for hdd and dvd storage?

Could always put them in a small http://en.wikipedia.org/wiki/Faraday_cage ? I wouldn't worry about that at all though, since HDDs are rather impervious to any magnetic field possible in a safety deposit box TBH.

Reading

http://blogs.oreilly.com/lightroom/2008 ... photo.html (http://blogs.oreilly.com/lightroom/2008/01/safe-deposit-backups-of-photo.html)
http://boards.straightdope.com/sdmb/arc ... 15946.html (http://boards.straightdope.com/sdmb/archive/index.php/t-515946.html)

You may want to consider an HDD now, and then an SSD once prices decrease (the chance of anything going foul would be absolutely minimal one would think).

bundus
May 13 2011, 03:41:58 PM
well, I thought you were going for hp ultriam tape drives or something like that. Its the best backup out there.

20 years just for say half a tb being safe. HDD or thumb drives backed up every five years would probably be best(cheapest). and vacum seal that shit.

fffuuu
May 13 2011, 04:00:12 PM
if you don't want to do it in a pro manner, buy some laptop hdds of various brands, some flash drives, and have multiple copies of the data, in multiple locations.

EntroX
May 13 2011, 05:39:13 PM
just use a good brand hard drive, or maybe even a SSD since they handle a bit of more punishment as things nowdays, just pack it well vOv

just google any particular model of HDD/SSD and you should be able to see their operation and storage temp ranges, but it'd have to be quite extreme to fuck them up tbh.

tapes work well too i guess, but i'd say its a bit too much ;p

theBlind
May 13 2011, 06:09:37 PM
I can only echo what's been said before.
Optical media is crap for data storage. It's expensive in terms of GB/€ and craps out after random(4,8) years.
Tapes may work, but are expensive as hell.

The best solution IMO is HDDs, buy mid end ones from good brands, ie. neither the cheapest nor the largest, you want the ones that build up on reliable, tested technology. And you want 2-3 of them, each with a mirror of the data. (Several mirrors per drive actually, so if one sector becomes corrupt you even have a backup on the same drive). Store them somewhere cool, dry and without dust reaching them (a shut cardboard box taped shut somewhere cool and dry would do. Place each HDD in a different location, possibly in different houses, in a save deposit box, etc.
Access the data from your own PC, not the backups of course. Don't do anything fancy like RAID, should go without saying.
Interface technology to those harddrives will become obsolete long before the HDDs themselves. So you have to keep up with the technological advances. If you really want to go sure, you'd need to deposit several üieces of tech to actually access the HDDs, meaning motherboards, CPU, RAM - a whole PC actually. If you're slightly less worried, you just keep up with technological trends and transfer the data to the next generation of storage devices with the then current interfaces. You'd have to do this anyway even if you go with the entire PC on hold setup, but then you'd have to buy a new generation of hardware as well every time you migrate.

tl;dr: HDDs will give best GB/€, will vastly outlast the availability of the devices needed to access them, so you'll have to migrate the data periodically. There won't be a from now until ever solution that you can actually rely on.

Aea
May 13 2011, 06:28:24 PM
I'm just going to add that SSD is probably the best choice, if you can afford it. Hard Drives are mechanical devices and several people have expressed concerns that not spinning them up for years may cause possibly problems w/ the bearings? I'm somewhat skeptical, but SSDs are practically indestructible.

fffuuu
May 13 2011, 08:23:28 PM
SSD/Flash are much more reliable, but HDD are much cheaper per GB. this wouldn't be a problem if OP didn't need to store 120gb.

160gb laptop drives are about 40usd a pop,

equivalent solid state, either usbflash or SSD, is 200+ USD

I would do both tbh, and redundant copies.

storage costs will go down in the future, so in 5 years you can dig all the copies out, and copy the data over to more modern media, which will be cheaper, and hopefully more resilient.

theBlind
May 13 2011, 11:37:10 PM
storage costs will go down in the future, so in 5 years you can dig all the copies out, and copy the data over to more modern media, which will be cheaper, and hopefully more resilient.
This is important - it's not "can" but "must". In 10 years time, the technology to access todays harddisks will no longer be trivial to come by. In 20 years, it'll be virtually impossible to acquire.

Aea
May 13 2011, 11:52:55 PM
storage costs will go down in the future, so in 5 years you can dig all the copies out, and copy the data over to more modern media, which will be cheaper, and hopefully more resilient.
This is important - it's not "can" but "must". In 10 years time, the technology to access todays harddisks will no longer be trivial to come by. In 20 years, it'll be virtually impossible to acquire.

IDE was designed/released in 1986, you can get an adapter on NewEgg for $10.

SATA was designed/released in 2003, I'm going to bet it will remain the defacto standard for at least ten more years, and will be easily available ten years into the future.

:obama:

ThonEney
May 13 2011, 11:56:31 PM
Even if it isn't accessing it would probably be easy,you got a lot of old equipment left and information on what works how so even if you couldn't for some reason find it you could have someone make you say an adapter.

balistic void
May 14 2011, 12:19:54 AM
Punchcards.

Mimiru
May 14 2011, 03:21:29 AM
Multiple copies on different mediums, some off site.

Ryas Nia
May 14 2011, 03:48:25 AM
http://www.mediasupply.com/tayu4dv16xsi.html

taiyo yuden archival DVD are expensive but the best you can get, they should hold their data for 20 years, its how i have my wedding photos backed up. Just keep them out of direct sunlight in a temperature controlled environment like a bank vault or dry part of your basement.

Jason Marshall
May 14 2011, 07:06:53 AM
Reprint them all on archival paper.

e. Yeah, this is a serious answer. Along with your digital soloution reprint everything so you have a new original hardcopy.

Pelias
May 14 2011, 03:15:48 PM
Tempting, but batshit expensive.

Jason Marshall
May 15 2011, 05:53:47 AM
Has far as printing them out on decent paper goes, my printer charges $0.40-50 USD per square foot. Thats on paper that is guaranteed to last 100 years. Prices get cheaper the more you print all the way down to .12 for the biggest jobs ive done.

Helen
May 15 2011, 12:09:31 PM
\ Goes off to google archival paper.

Bombcrater
May 15 2011, 01:25:12 PM
IDE was designed/released in 1986, you can get an adapter on NewEgg for $10.
It's nowhere near that clean cut. Using 25-year old hard drives on a modern system is a complete crap shoot, especially if the drives have been in storage for years. I've had to do this for real many times and the success rate is poor. Drives deteriorate with age even when stored under good conditions and often fail because of problems not obvious at the time - drives from the 80s/early 90s have huge mortality rates due to bearing lubricant breaking down and track markers fading out. How will today's drives with their fluid bearings and ultra-high areal density survive 25 years in storage? We just don't know.

Interface incompatibility is a big issue. Early IDE drives often just don't work on modern UDMA controllers, sometimes due to quirks in the drive's IDE implementation, but more often simply due to bugs in the controller firmware/drivers. No manufacturer is interested in testing with decades old drives. SCSI has similar issues and I'm sure SATA will in the future.

My own system for long-term data storage is pretty ghetto. All data is kept live on my main PC, with an external drive as first line backup. Every time I retire a drive because it's too small or too noisy or something I put a copy of everything that needs preserved on it and add it to a pool of drives that sit on a shelf. When any drive in the pool reaches 5 years old it's wiped and either discarded or used for something unimportant. Variations of this system have worked well for me for 20 years.

Jason Marshall
May 16 2011, 03:04:35 AM
\ Goes off to google archival paper.

You will probably come up with "Acid Free" paper. Same stuff.

theBlind
May 16 2011, 08:19:36 AM
IDE was designed/released in 1986, you can get an adapter on NewEgg for $10.


Like bombcrater said, it's the difference between the manufactureres promises and reality that tends to bite you in the ass.
If only all such promises would hold up in the real application, life would be so very much easier.

Hurricane
May 16 2011, 08:20:26 AM
going with Bombcrater on this one.

I too have a RAID (MIRROR) setup which I keep all my data on, "LIVE" inside of my PC (for a slightly better solution get a NAS since spool up / down is the biggest issue with HDD's and the temp changes from inside a gaming rig) and a ext. HDD that is my first line of "oh shit".

Apart from that I have Archived ZIP Files with restore files both on RAID and on ext. HDD and if you want you can even keep the restore files off site.

Same as bombcrater I advice to use your "live" hdd's and have both a ext. HDD as 1st line of oh shit and a 2nd oh shit device (2nd one off site) moving the data around between the LIVE system and the spare copies during the years.

Also, keep your kopies safe in a dry and stable environment in terms of humidity and temperature.

mjukis
May 16 2011, 08:59:47 AM
120GB is not a gigantic amount of data, keep it on your computer and just do regular backups to regular HDDs.

Pelias
May 20 2011, 12:36:57 AM
Has far as printing them out on decent paper goes, my printer charges $0.40-50 USD per square foot. Thats on paper that is guaranteed to last 100 years. Prices get cheaper the more you print all the way down to .12 for the biggest jobs ive done.

You need both archival grade paper and inks.

High-end Epson ink is pretty good, but printers start at 1000$ and ink itself...isn't cheap. You can order printouts from a print shop, but you have no guarantee or way of checking if they're using originals, or cheap CISS that will start to to lose color intensity after 1-2 years.

mjukis:
I will stick with that +several hdd backups.

Jason Marshall
May 21 2011, 12:08:40 PM
If anyone is seriously interested my cousin does all the printing. Uses Epson and canon wide format printers and offers cheaper and archival inks. Can probably get jobs done at the family rate.

elmicker
May 22 2011, 01:29:14 AM
Doesn't the data in flash storage decay eventually?

Joe Appleby
May 23 2011, 03:55:44 PM
Photos? Get them reprinted as Jason Marshall said. Or if you want to be real pro, make slides out of them. As said before, HDD access tech changes and who knows what will be used in 20 years. Proof: Anyone remember SCSI? With slides all you need is a light source, a lens is optional. Store them safely and dry and they will last you for decades.
German government is making microfilms from art etc and storing them in a abandoned mine. To paraphrase their director: after a war, all you need is a candle to restore the German culture if it goes really bad.

Pelias
May 24 2011, 06:31:29 PM
Except slides:
-can be eaten by mold
-not always as time resistant as advertised (the oldest Agfa ones I have, already lost most of their red color - german technology for the FAIL)
-good luck cleaning them (stuff like 20 rear old fingerprints is pretty much impossible to get rid off)
-they are also easily scratched, etc...

Jason Marshall:
Where do you live (what continent?)