I debated doing this recently but ultimately spent a lot of money on an LTO-6 tape drive instead, just due to the cost efficiency over time for larger amounts of data. I justified it to myself as I needed to either expand my NAS or clear stuff off it, and the drive was the same price as buying two new HDDs to expand the NAS thanks to the obscene price inflation at the moment. As I’d likely need to further expand my NAS next year (I’m shooting a lot of photo and video) I think this works out a much more cost effective way of storing this data long term when I likely don’t need to access it much.
After reading reading the "Environmental and shipping specifications for LTO tape cartridges" [1] I came to the conclusion that tapes are not that great.
Environmental Allowable Recommended
Factor Storage Storage
Temperature 16 to 32°C 16 to 25°C
(61 to 90°F) (61 to 77°F)
In the summer it's not unusual to reach temperatures over 30°C in the house.
There's also this story [2]:
> I had a room full of servers, many with 16-36 disks per server and had an AC failure. Things turned off as planned, but it hit 115°F [46°C] for a bit. None of the disks died, majority of the tapes did.
Holy cow, thank you for this. On that same IBM[1] link, it reads:
Maximum wet bulb temperature (Only LTO-8 and earlier cartridges) / Maximum Dew Point temperature (LTO-9 and later cartridges), all across the board, allowable / recommended / shipping: 26°C (79°F).
In Florida, we like to refer to 79°F dew points as "Tuesday."
More to the point, the time I want those LTO cartridges to earn their keep is during a disaster scenario. The most likely disaster scenario--other than fire--is hurricane. High temperature and humidity, and no power to run the HVAC to mitigate those. (Unless you have generators with fuel to run for days, and your HVAC, generators, and fuel tanks survived.)
I doubt that the recommended storage conditions for qic tapes were much different from the above; and I've seen a system boot from a qic-120 tape that had been kept in a non-climate controlled ministorage locker in Santa Fe NM for 17 years.
Oh very interesting! This isn't (currently) too much of a concern for me as I live in Wales, and though it's ridiculously humid here there's not much risk of it hitting those temperatures inside my house the vast majority of the year. I shoot a lot of photographic film as well so I'm used to storing these sorts of media, though I probably won't be shoving the tapes in the fridge beside the film. Currently they sit in my windowless office.
They’re engineered like this because 99.9% of tape storage media is sold to corporations who store their backups in climate controlled buildings with backup generators.
Here in the north of Scotland where it never really gets below 5°C or above 25°C making it t-shirt weather all year round, I think the tapes will be okay in a dry reasonably-insulated storage room.
The problem with optical disks is that nowadays they have a too small capacity in comparison with magnetic tapes or magnetic disks, so they are useful only for archiving small amounts of data, of a few TB at most.
If you want to archive up to 100 TB or even 200 TB, the cheapest way to do that is with HDDs, but you must keep in mind that you must use at least duplicate HDDs for everything, preferably stored in different locations, to avoid losing data due to files that become corrupted after a long term storage or due to some accident. Moreover, you must migrate the data to new HDDs periodically, e.g. every 5 years.
If you want to archive several hundred TB of data or more, magnetic tapes become cheaper and they are more reliable than HDDs. You still need to migrate the data, e.g. every 10 years, but much less because of the risk of corruption or mechanical failures, but because the tape format may become obsolete and appropriate tape drives that can read it might no longer be available.
I have used tape (of various flavours, QIC, DAT, LTO) and DVD+R and DVD-R for decades. Bluray has been the most reliable of all of them for me overall.
Tape is amazing, and resilient, but the drives are fragile and expensive.
The sad part is that Panasonic/Sony/Pioneer have pulled out of making drives (I believe Panasonic still make some for their own vertically integrated products), which just leaves Hitachi-LG.
I see several used LTO-6 tape sales on Ebay, for about $13 per tape. The cheapest used SAS drive I see is about $30. I'd trust a used tape over a used hard drive every time. Tapes have a 30 year life span, I can't imagine a hard drive would come close.
Ignoring the cost of a tape drive, tapes are cheaper per TB, yes. I wouldn’t write off used hard drives though - buying only the cheapest per TB used enterprise drives, I’ve only had a ~5% failure rate across roughly 200 drives with 2-5 years of runtime (while I’ve owned them - they usually have 4-6 years when I get them). I usually get rid of them once they hit 8-10 years of uptime (or if any get any uncorrectable sectors).
If you only need ~50TB it’s probably both cheaper and easier to just buy hard drives, even today, factoring in the cost of the tape drive. Maybe even 100TB. I’m personally backing up ~225TB off-site these days and I still find AWS S3 glacier deep archive to be cheap enough - at current prices I’ve decided the hassle of doing tape again isn’t worth my time unless I’m over 300TB.
I'm not sure I'd trust a used tape. They're generally only rated for a couple hundred or so writes. They'll totally sit on a shelf for 30 years, but writes tear them up.
If I write and verify the write, then that's good enough for me. I don't only buy used tapes, I have about 100 that are new/unused that my important data go to, and really important stuff goes on at least 2 tapes.
> The biggest advantage of LTO is that it's designed to be offline.
It's been a while since I did enterprise backup, but when I did, most backup software had a 'bootstrap procedure' that you could use even if you were completely destroyed.
First, most enterprise software does not need a license for restorations, only for backups. In a disaster recovery scenario you're unlikely to have the keys.
At end of every backup run/day, an e-mail is sent out with the tapes and offsets that have the indexes and configuration backups. You can send this message to an e-mail address that is separate from corporate e-mail (in case that gets wiped as well); back in the day you could send it to /dev/lp0 (and stick a copy in the box that the offsite tapes are transported in).
So even if your entire infrastructure is wipe (e.g., see NotPetya and Maersk), there's still a starting point for restoring your restore-backup server, and rebuilding up your infrastructure.
Remember: your backups have to be inaccessible to attackers.
I can see this making a lot more sense for photo/video. The Holocron approach is neat when your truly irreplaceable data fits on a handful of discs, but I would not want to burn and manage dozens of them every year
Yeah I'm tending to shoot around 1TB of data per wedding I cover now that video is in fashion for it. Thankfully it's exactly the sorta thing tape is suitable for - once I'm done I can just archive a couple of weddings off per tape. It's generally very unlikely I get requests for that data later (once they've ordered prints) so doesn't make much sense keeping it on spinning disks in my NAS.
I've been thinking about different techniques for making my NAS backup-able to cold storage and... the thing that has given me pause with tape is how closed-down the ecosystem for the LTO drives themselves seems to be. What's your strategy for knowing you'll be able to access your tapes to transfer them forward to the next generation in 10-15 years when the tapes start reaching the end of their dependable life?
Looks like I'm a bit out of date, but older generations of LTO drives were able to read tapes from up to two generations ago, and write tapes from one generation back. Looks like this isn't the case since LTO-8. It looks for 8 and 9 theycan read and write to tapes back one generation. 10 breaks the pattern and can only read LTO-10 tapes.
If you have money for LTO-10 drives and tapes, then backwards compatibility shouldn't really be a problem. The drive alone is $20,000 to $30,000. If you can afford an LTO-10 setup, then an array of lower LTO drives seems affordable.
> is how closed-down the ecosystem for the LTO drives themselves seems to be.
Linear Tape Open is remarkably Open for an enterprise product with Open in the name.
Personally, I feel like archival storage of personal digital data is not really solvable. I feel pretty confident about keeping my online storage going while I'm able. After that, there's nothing effective to preserve it when I'm gone. That's not a huge deal for me, I don't expect my heirs to want or need these things... Photobooks with annotations are more likely to last and have some chance of being apprechiated.
EBay has a bunch of LTO-3 tape drives for sale for <$300. LTO-3 is 22 years old and can read LTO-1, originally released 26 years ago. I expect them to still be for sale 20 years from now, albeit fewer in number. I see no reason why LTO-7/8/9 drives won’t be readily available for the foreseeable future.
I am also considering an LTO solution of some kind, but I also found out that LTO drives seemingly need some TLC and maintenance if you don’t want your tapes to jam up. I feel like at least with optical media, it’s nearly impossible to screw it up.
I really wish a multi-terabyte optical media solution took off, because that would be the path I’d ultimately take, instead of going with expensive LTO devices and media.
I bought a used LTO5 tape drive from ebay, and I've been getting used LTO5 tapes. I've never had a problem. The tape drive lives out in my garage inside my backup server. I have a 10Gbit link so I can write 140MB/s over ethernet from the server inside the house. It works perfectly, has for years, nothing "jams up" (so far). I've had to use a tape drive cleaner cartridge once since I got this thing years ago for routine maintenance. I verify writes for anything important.
Optical just didn't make sense to me. Overall it was way more expensive, and a lot slower to burn BluRay discs.
The max BluRay disc size is 128GB (which is smaller than a lot of SD cards these days), and the fastest you can write to a BluRay disc is 18MB/s. That's a lot of of swapping discs to burn my many Linux distros.
And those 128GB BluRay discs cost between $30 and $40 each. Used tapes (with a lot of life left in them) on ebay are far less expensive per TB - the lowest I've paid is about $1.40/TB for LTO5 tapes without a case. I typically pay about $4/TB for unused LTO5 tapes with a case on ebay. It's getting a bit more difficult to find those deals, but they come up every now and then.
In bulk the 128gb blu ray discs aren't nearly as expensive -- I picked up a 25 pack for about $230 18 months ago
The bigger problem is finding a drive that can read and write them -- the 128gb format support is a bit harder to find. I thought I would be good with my nice Pioneer BDRW drive but it didnt have the support so I had to buy an LG unit. So now my desktop has two BDRWs -- how very 2006!
> ultimately spent a lot of money on an LTO-6 tape drive instead
I'm semiseriously considering getting the old LTO3 library from work and putting it to use. Sure, it's "only" 400GB per tape but I have a literal vanload - the racking takes up the same volume as a mid-size Ford Transit - of new and used LTO3 tapes to dispose of too.
Are Verbatim's M-Disc Blu-Rays still really M-Disc? There was something about how they changed the recipe a few years ago, and the counter-argument was that Blu-Ray discs are non-organic anyway, which was the main feature of M-Disc DVD.
There's very few companies now making new Blu-Ray drives too. Pioneer has supposedly stopped producing them, and the ASUS drive I bought earlier this year was one of the only SKUs on the market. Sony stopped producing high-capacity recordable BDXL discs too which means they aren't much of an option now for large backups.
> While the immediate burn quality seems relatively satisfactory, I was still not satisfied as to whether these discs were truly worth their premium over commodity blank media. Looking at the discs, it seemed that the recording layers were likely made of similar materials based on the visual aspect, unlike the obvious difference with DVDs. Also, it seemed that regular recorders were intended to be able to record to the BD-R’s (according to the Verbatim text).
...
> Digging further around the M-disc site, it is noted that their section on longevity testing actually features no data about the BD-R discs at all. Indeed, if there were tests, the reports were not actually featured on the site at all. Further poking around in M-Disc Technology pages shows only references to their DVD products. In fact, it seems their patented “etched in stone” material may only apply to their DVD products.
...
> From my research into the data, it seems likely that the M-Disc BD-R is not substantially different from other quality BD-R discs on the market as they all use inorganic recording compounds (with the exception of BD-R LTH of course). From a visual inspection, the disc appears the same bright chocolate-brown, unlike the obvious distinction with regular dye based DVDs and the M-Disc DVD which was a silver-grey. But my main concern lies in the demonstrated lifetimes of quality BD-Rs which are similar to the M-Disc DVD (to which the M-Disc BD-R is claimed to have also achieved). In essence, it appears that the MKM/Verbatim BD-R DL has already reached the same level of endurance as the M-Disc DVD had claimed without being “special” at all. It is thus, hard to justify paying more for an M-Disc BD-R, and I cannot really recommend it.
The "not really M-Disc" stuff comes from a post somebody made on the "Data Hoarder" Reddit board a few years ago. The discs were changed to enable higher burning speeds which resulted in a slight difference in appearance, but he/she didn't seem to have realized that Blu-Ray M-Discs were never substantially different from BD-Rs and instead assumed that the tweaked discs were more of a scam than the originals.
My understanding is that the original M-Disc pitch mattered much more for DVD because it replaced the organic dye layer, while BD-R already uses an inorganic recording layer
Basically your media only needs to outlive your drives, retrieval mechanisms, software interfaces, and drivers. Or more properly, your backups need to outlive your live system and the need to restore any data from them. Certainly, data may be more precious than a live system, but you also need to concern yourself with format transfers, in terms of software file formats, filesystem formats, and media design/format.
This is not a difficult achievement. The people marketing long-term archival storage media are capitalizing on ignorance of people who have worked with books and photographs, which are "readable" by human senses. Books and photographs are media which can be preserved and fundamentally produced on archival media from the get-go.
Backup media, since it is necessary to have machines read and interpret them, are an entirely different beast, subject to an entire chain of things to preserve. As I've commented before regarding BD-RE XL, my media and the Blu-Ray drive itself outlasted the system it was backing up, and therefore did their job, without any investment or wasting money on M-DISC "Engraved in Stone Tablets" for the gentiles.
Did this experiment and it works but with caveats:
1. The disks bigger than 50GB require a BDXL drive to read, which means spending $300 on a disk player. Which also means rare to find in the future. So if you want larger compatibility use 50GB or smaller M-disk.
2. Japan sells supposedly original MDisk but they look rather black than gold, so they are not the original. Still strength and longevity is quite good on Blu-ray.
3. Storing the disk on materials that degrade like cloth/plastic sleeves may be counterproductive. I got a hard case but it takes effort and is increasingly hard to find.
For something explicitly meant to be recovered in a worst-case scenario 10–20 years from now, optimizing for boring, widely available readers seems more valuable than squeezing another 50GB onto each disc
If you want to pass stuff on stuff so it will last for ages, then: Get it done already. Print it, email it, post it somewhere and share links, have it framed and then delivered by a singing courier -- however it is done, then just do it and get it over with.
Now it's their problem to take care of, ignore, make fun of, or whatever, which was always ultimately going to be the case anyway.
This allows you to back with living your life while you still have one.
There's little point to bothering with predicting what the future will look like after you're dead, and even less point to trying to influence how others will behave on your behalf.
If you really want to have a lasting legacy of documentation, then build it and send it. Send it early, and send it often. Be the most interesting person on Earth today for your group.
Chances are fantastic that nobody will want to volunteer to build it for you after you're gone. But if it's important for you today, then make it easy on them: Send it right now.
...or don't. Really.
Telling some great stories at gatherings while perhaps sharing a couple of visual-reinforcement photos on a phone is enough unless you're a Kennedy or something. A wonderfully-interactive story is worth a thousand photographs. If you've got interesting stuff to share, then it will be interesting to hear, and to interact with, and it will remembered.
For awhile.
The best stories won't be chosen by you, but they'll be retold and will ebb and flow, and will subsequently die in a generation or three at most, but that's OK: You'll be dead, too. Get over it. By generation 4, those kids won't know who Uncle Fred was anyway except for a kind of sleepy myth about the carefully-preserved BD M-Disc archives (whatever those are) that were binned by Aunt Sally two decades prior because even Aunt Sally didn't care anymore, and couldn't be stuffed to keep them.
So keep living, and keep making and sharing new stories. Give people something to remember on their own accord. Don't force it, just do it and share it. Remember: If you were never very interesting to begin with, then they'll just bin whatever you left for them to discover when you're dead.
So be interesting. Live a decent and good life while you have it. Share it; no secrets. If it seems worth sharing after the funeral, then it's worth sharing right now.
(Except! In the odd chance that you are a Kennedy or something, then: Hire biographers. Hire them early, and often. Get that money, and line up the publishing of that stuff while you can still direct the proceeds. This is the path to preserving a legacy in a way that others can see down the road. (It might make it all the way to generation 5! (But it probably won't.)))
As others have suggested, photos of previous generations look a lot like strangers, absent a story connecting them across a lifetime, and ideally all the way to the present.
The story furthermore must have enough intrinsic drama to engage readers, and fall into the hands of a writer with the skills to turn the raw facts into a biography.
The biographic text provides a structure to hang the photos on, yielding a "photobiography". Search for the term and you'll find examples.
I love curating my photo library but after I kick the bucket I have to assume the number of photos that my future relatives care about will number in the dozens.
Same here. 240,947 photos in there right now and organized enough that I can find stuff (of course a single nice social gathering or outdoor adventure can be 100+ photos - cheap to store so why get too picky).
So why is it that photos >20 years old just aren't that interesting any more? Because photos need to be augmented by memories, or by stories from others. Once they're just pictures... and once I'm gone... no more interest.
We'd never throw out old physical photo albums. We have a cupboard full of them that my mom left behind. Some of the photos in them are of people we didn't even know, and now won't even know the names of. We keep them because we still have a little bit of connection to them - we knew the people to whom they were significant. But soon they'll just be landfill.
That's how human memory must work. We can't have every generation leave behind hundreds of thousands of pictures, piling up forever. Memories of past generations get purged until 2-3 generations back they're forgotten.
Of course there are exceptions to every rule. An uncle was persuaded to write his memoirs about his childhood on the farm in the middle of the last century. That will still be a good read after he, and those who knew him as a person, are gone. Assuming the books survive - it was a small, private printing only.
I've talked to many deceased estate cleaners over the years, I have seen them in action many times. The amount of stuff people want to keep from relatives is a rounding error of 0%.
Whole lives of collecting sentimental goods all end up going in a giant bin, a whole house cleaned out in less than 2 hours. Photo albums are rarely kept, they just go in with it all and off to the tip usually in a single morning.
What you think is important to you, is basically just trash to your relatives/kids. They will probably have want they want before you are gone.
Blu-ray drives are becoming increasingly rare sadly, so it's not great to hand over to others in the years ahead. An external hard disk drive kept safe and secure might be your only option. I had the same issue 6 years ago winding down my company and archiving data. Only sensible options were M-DISC and external hard disks.
But a month or two after I initially wrote that, my brother and I found a couple boxes of photos from my parents and grandparents. No notes. No dates. No albums. Just envelopes and envelopes of undated photos.
Taking the time to make an annotated printed photo book, including dates, people and place names, and any other context is going to make those images a lot less irrelevant to future generations, and printed media, stored well, should easily last 50-100 years.
I would not make a bet that you'll be able to procure a bdxl reader in that time period.
This has been my system for a while now. The funny side is that, I have yet to have a drive failure so the amount of local backups is getting silly.
The other thing I did was I ended up collecting HDD's out of old DVR's people were throwing out. Those drives are used for cold storage but not for anything important because they already have tens of thousands of hours on them and in not the best temperature conditions. Again, non have failed yet.
Funnily enough, the only HDD I have failing is a 2.1GB Quantum Fireball from 1997 in my Pentium 1! Really should swap that for an SD card via IDE.
I've had drive failures now and then. I once dropped a 10T drive on the floor. I thoroughly examined it, and it was not only merely dead, it was really most sincerely dead!
A couple of years ago, I repaired one of a pair of drives (I think they were 750GB) that had the connectors mysteriously snapped off[1] while I was poking around in a client's system.
It was a special system, Linux-based, running proprietary software for expensive proprietary hardware.
I apologized profusely. I got them some new hard drives coming right away. I found a forgettable donor drive in the boneyard that had a close-enough footprint for the connector, harvested it with hot air, and put its connector onto one of the broken client drives.
A little bit of dd and some mdadm magic later, and they were back in business with that RAID-1 array like it never happened (other than just a few days of downtime on a system they only used occasionally).
---
A different time, 25 or more years ago now, I somehow broke a 4-pin Molex[2] power connector off of one of my own IDE drives. I don't remember the details, but I do remember that I fixed it by soldering a Y cable power splitter into the holes on the board. This gave me a drive that worked, and that had power pass-through. It was kind of nice to have (and it worked for years and years after that).
Anyway, I guess shit happens sometimes.
---
[1] I wasn't rough with anything; I was just moving wiring out of the way, as one does, to attack the dust bison that had accumulated over the previous decade or more with some canned air.
I somehow broke the connectors of both drives, anyway. I can't explain it and I don't have to, so I guess I won't bother trying. They were fine when I started and they were broken when I finished.
[2]: Those aren't Molex connectors. Everyone calls 'em that, like everyone calls adhesive bandages Band-Aids, but the old drives used a design characterized by the Amphenol Mate-N-Lok connector design which had nothing to do with Molex.
I've been programming for 50 years now. I've had all sorts of failures, including two incidents of my desktop computer burst into flames.
I now set the desktop on a piece of sheet steel. I also only use steel enclosures for them.
I have a Carver amp that runs 24/7 in my office, for the last 40 years. It's now emitting a 60 cycle hum, which I learned is due to decaying capacitors. I guess I need to get a cap kit and fix it.
Yes. AVCHD for video (or BDMV), and JPEGs in folders. Labelled. They play in every BD player I've tried them in from 2006 onwards.
And the crazy thing is - we actually watch home movies in a linear format as a consequence. Rather than picking a clip here and a clip there amongst a sea of photos, video can be laid out in a linear timeline and watched without editing.
(I still use a professional Panasonic video camera that shoots AVCHD, which makes this super easy. Because editing video is not something I'm ever likely to have time to do)
The problem with archiving data is you always need to be prepared to use the "next big thing" once your preferred method becomes unobtainable.
I currently have :
- Photos in iCloud (Live working copy)
- Backup at home on NAS (updated nightly, snapshots)
- Backup in pCloud (lifetime, mine or theirs, no illusions, hopefully lasts a couple of months if I die unexpectedly)
- Yearly archive on external HDDs, identical copies stored in separate locations, clearly labeled "Photo backup". No encryption, no archiving, just plain APFS formatted drives that can be plugged into a computer and read.
- Yearly archive M-disc blu-ray discs, again, identical copies, stored in separate locations. One disc per year labeled as such, ie. "Photo Backup 2020". The idea being you restore them in chronological order and have a complete set.
My dad created/curated a family archive dating back to the 19th century. Since I moved it to immich people sometimes look at it! In fairness, though, they mostly look at the new stuff.
Nope, I think it's really hard to assemble an album with sufficient context to warrant your time and interest.
If an album is well annotated with dates and locations and names and descriptions, it might be fun, but I suspect the number of old photo albums with this kind of effort put in them is rare.
No. People in my grandparents' time and place couldn't afford cameras. There are maybe one or two photos of a person. Some siblings are only notes on an old church ledger.
LTO tapes are the worst possible format for passing on family photos to family members. I dread to think of anyone in my family trying to setup an LTO drive when they barely understand how to use their smartphone.
I mean if you don't print them and want digital and we're talking having them for the next generation(s) then pretty much whatever current format you choose, it won't be the one that's widely used in the future so it will require some tech knowledge. And LTO tech stayed pretty settled and compatible for quite a few years now.
Digitally printed photos are also quickly deteriorating, I learned when I look at the fridge where we posted some of our best shots. These are about ten years old. Opposed to analog printed photos that are more than 45 years old and are still in the same condition as ever.
Why? Both sorts of prints are exposed to the same amount of UV as they are pinned to the same fridge.
I wonder if there is a better printing technology comparable to analog "prints" (of film). That would actually make more sense (to me) than saving everything on M-Disc.
Ah, sorry I thought the old ones were stashed somewhere. Yeah, I'd like that too, although not sure who would I trust telling me that these will actually be like this in 30 years.
In the meantime I found out that there are in fact printing technologies based on pigments with a longer lifespan. For instance Epson Surecolor or Canon imagePROGRAF. The alternative is to use online services who produce actual "c-prints" of digital images.
LTO drives/tape are not friendly, consumer grade technology. One look on a search engine, and it's all obscure refurbished server parts costing hundreds. The prices of them going forwards might increase sharply too and make it unviable...a bit like external hard drives right now. So yeah, printing everything out and hoping for the best is the only sensible option...
There are companies that can read the tapes for you, especially for a one-off transfer.
This is an _additional_ layer of backups, on top of the usual iCloud/whatever things. LTO is certainly not user-friendly, but neither is it especially user-hostile.
And I think I trust tapes far better than any other alternative for long-term storage as they have by far the smallest areal density of information compared to disks (hard or optical).
I have DVD disks that are barely 10 years old and that I can't read anymore. Hard drives also die in storage and/or during transport.
These companies charge a megabuck. Maintaining the drives doesn't come cheap. I am (like other ageing former mainframe operatives) the holder of a variety of tape forms, and nobody local wants the hassle of giving me a bitstream, without significant vig. Like thousands.
Recently the computer museum helped recover early 1970s unix tapes. It was a big deal.
I got introduced to LTO by needing to read several tapes with archival data. It was pretty cheap, around $200 back around 2014. I can certainly believe that the old mainframe tapes are way more trouble.
And let's not exaggerate the complexity of setting up external tape drives, even on Windows/macOS.
> And let's not exaggerate the complexity of setting up external tape drives, even on Windows/macOS.
Having found a matching Sun Ultra 5 generation drive, I tried to get it to work with a SCSI card, and had .. many many problems. It may be the rubber bands. It may be stiction. it may be ants. It may be me.
I would personally love to de-exaggerate the complexitity: Old hardware is gnarly. I sourced parts, it wasn't working.
Have you tried LTO tapes and drives? They are seriously better than the bad old tape drives. They even have a standardized filesystem (LTFS) that can be mounted as a normal (albeit very slow) disk. No need to deal with arcane TAR stuff.
And yeah, I started with a ZX-Spectrum with cassette tapes. I was pleasantly surprised by the overall LTO experience.
Your grandkids will just say "make me some memories" and their AI will look at your posting history and gen up some fresh new ancestors very similar to the originals.
Sorry if it was not clear: The NAS is at my house but houses may burn down. THE discs are at my parents house for off-site backup. Like the sibling poster says, I would be the one going through the things anyway. But if unsure you can also ask a friend to keep them, one blu-ray can hold tons of photos and doesnt take much space.
Fair. Nothing is risk free. Move overseas, be more than 10 days away, other sibs have the burdens "we threw out those old unreadable backups Dad had stashed from his time at the computer center".. But your strategy isn't the worst by far. My 3-2-1 has 2 of the 3 in the same physical location. Not good.
They probably will be the one going through their parents stuff and hopefully by the time its an issue can move it to their kids' place. Options for trusted free hosting are limited sometimes. ;)
Each year I select about 100 representative photos and get a photo book printed. It’s not a perfect solution but at least they might get something to see 100 years from now.
Sealed pelican case has decent precedent. A small regional bank I was aware of used basically this exact strategy. There were couriers going between branches daily, so it was easy to cross-ship a sealed DR/backup box to a remote branch every day. It stayed in the remote vault until its return journey
> In case of a catastrophic fire for example, both the computer and backup drive could be destroyed.
I just keep my offsite backup drives in my office drawer. There is a note on them that says they are personal backup and not to be plugged into company equipment. If you have an office this is probably the easiest solution.
If a disaster wipes out my home and my office then there are bigger things to worry about than my data.
I've gone all in on M-Disc (and even regular HTL inorganic bluray). I've been using it as an archival medium for about five years.
With hard drive prices being what they are, I feel like my decision is increasingly justified. Storing large amounts of personal data (documents/photos etc) either in the cloud or "hot" on local drives, where the access patterns are infrequent, has simply stopped scaling.
Having discipline around burning disks has also made more disciplined about archiving.
The burn speeds of 6x Bluray media aren't substantially slower than SMR hard drives. During a recent archiving session, my SMR 4TB hard drive was sustaining 60MB/s. My Bluray burner sustains 27MB/s.
A pack of 20 BD-R DL Blurays (50GB each - a total of 1TB) from Japan was less expensive than a 1TB 2.5" external hard drive.
Cloud storage isn't tenable either. Despite having a 1000/400 fibre connection, with international links from Australia being down, backing up to Hetzner only sustains 5MB/s - backing up 5TB to AWS or equivalent would be faster, but likely to cost around $700 a year. This is on top of the $140 a month for business grade internet with reasonable upload speeds.
I'd love an LTO drive, but that requires a server or appropriate desktop, and seems overkill for 5-10TB. Even for secondhand drives, I'd be looking at multiple thousands for a a complete setup.
Continually upgrading the NAS with more bays and more drives is also not sustainable.
I index and catalogue each disk with Neofinder (great piece of software) that allows offline browsing and searching of each disk. The searching and browsing is more performant than the software provided on my Synology.
And as an upside, videos and photos of the kids can be viewed by anyone capable of sticking a disc in the drive, without needing access to servers or cloud providers (I burn some in BDMV format with no DRM, so it should play in any Bluray player or computer).
Neither TFA or the comments seem to start with the two most obvious rules: Digital data can be perfectly replicated and it can be checksummed. Those should be the pillars of any backup. Regular transfer to new physical media and check.
> They needn’t be particularly beautiful, either. Better not to stir strong aesthetic or religious feelings with inscriptions and the like.
I remember a discussion on architecture where the point was that, while newer buildings are built with more sustainable techniques, they are also ugly and therefore no one cares if you want to demolish one. Buildings perceived to be pretty, on the other hand, are more likely to stay around longer and therefore have a lower environmental impact in the long run.
Making your M-Disk aesthetically attractive may be a way to ensure that they are around for a longer time, if only because someone may confuse them with electronic waste and take them somewhere away from nature to ensure they are properly destroyed.
Sure, both is true. I didn't want to make the object too attractive, so that it makes visitors want to take it with them or disclose its location. And not too ugly to provoke an urge to break it or throw into the water.
I would be so afraid of breaking the disc if I ever tried to break the glass to retrieve it. Hopefully there is a mechanical engineer around when someone wants to retrieve it.
Having long-term backups like this I wonder if bundling a known-working Blu-ray reader or even a combination writer is a good idea. Hypothetically a reader would be hard to come by. With a reader I would either need a adapter to make it work.
I'm surprised this is not more common. I have been doing this since 2016 (started with DVDs and quickly moved to Bluray since writable DVDs are more prone to failure).
Do you periodically re-read the old discs to check them? I burn a par2 recovery set (around 10 percent) onto each disc alongside the data, and once a year run a full verify with dvdisaster or just par2 verify. A disc that starts throwing correctable errors gets re-burned while it can still be read cleanly, which matters more to me than the rated lifetime of any single disc.
Just use a magnetic hard disk and your data will be fine for many decades if not centuries. You don't need M-Blu Rays (which have very limited capacity anyway). A hard disk is cheap and stores several Terabytes.
You obviously need to store the data in a physically remote place like your shed, car or safe deposit box in case your house burns down.
Never, ever use a SSD if you want your data to be safe for any amount of time. What's worse, SSDs can fail unexpectedly with almost zero chance of data recovery.
> Just use a magnetic hard disk and your data will be fine for many decades if not centuries.
A few months ago I needed to pull from an offline backup. The disk, which used CMR and was purchased new from an actual computer supply company (i.e. not Amazon), was burnt in for about 72 hours before it was loaded up with data and shoved in a padded box. Ten years later, the drive was pulled, and there were so many media surface errors that most of the data was unrecoverable. (I haven’t bothered opening the drive to see if there is some obvious sign of what exactly failed.)
In my case the backups were created in multiples using two different brands for exactly this reason, and the second copy was fine. But I wouldn’t say it’s the case a single magnetic disk can take the place of BD.
Mechanical disks are also getting expensive. The disk purchased to replace the failed backup was being sold at three times its original release MSRP. Some of this is due to the AI hysteria but some of it is also that the market is shrinking and becoming K-shaped.
Ten years in general is way too long to leave any piece of electronics unpowered, let alone a fine mechanics device like a HDD with moving parts. They need regular exercise.
Oh for sure don't trust any single copy. But I wonder, how many surface errors? How much of the data would have been lost if all your drives degraded that much?
For example, if most files are multiple gigabytes, the drive has 3 million tracks, and 3 thousand of those tracks are ruined: one degraded copy leads to most files being broken, two degraded copies leads to only a single digit number of files being broken, and three degraded copies means >99% odds of recovering everything.
And with the magic of par2 files, a single drive with a couple percent of corrupted tracks can be fully recoverable. Any cold backup I do gets some of those.
Tell that to my circa-2000s WD MyBook that clicks and fails to initialize after ~17 years in climate-controlled storage in its original packaging. I'm about to attempt surgery on the heads to see if I can get it going again.
I also have a binder full of DVD-Rs, of about the same age and containing mostly the same data, that all read perfectly with no issues.
You're asking for trouble if you count on just one backup solution to keep your data safe.
> Just use a magnetic hard disk and your data will be fine for many decades if not centuries
It is absolutely not safe for many decades. If you haven't encountered any drive failures yet — you're lucky.
> A hard disk is cheap and stores several Terabytes
You should check prices again, I wouldn't say cheap considering price bump during the last year (or around so). M-Disks are way less convenient, but pretty reasonable for their cost.
I've had one drive for almost 20 years (don't know how long exactly). It is powered up once in a while and stored inside the house. No read errors of any kind as far as I can tell.
Even if perfectly held still in a sealed box with perfect conditions, hard disk drives need controllers; they need the chipset to drive them, which has ROM and other stored microcode that is susceptible to bit flips due to cosmic ray impingement. I've traced faults in a few 20+ year old HDD's due to failed controllers likely due to such loss (either in the main controller fail, a ROM/BIOS chip fail, or less likely but still possible the cache IC, or motor controller). The data is there on the spinning rust, you just won't have a way to access it.
The failure of SSD's you warn of and the way the HDD's fail due to a dead controller aren't that different.
This won't happen with a physically written media like a CD, DVD, or Blu Ray disc. With physical media, you're separating out the media from the reader (the drive). If your disc reader dies or you cant find it, etc, you can go find any another one on ebay even decades from now, very likely.
Doing surgery on an HDD controller PCB to replace a dead controller, while possible, sucks, and may be incredibly difficult if not impossible to source parts for in the future.
If the data storage hardware cannot read a block that it has stored within itself, then has the hardware not failed?
---
Suppose I hire a scribe named Gunther, and this scribe named Gunther confidently states that he has a lifespan of a mean ~quarter-millennium. He can't really explain how he does it in simple terms, but Gunther states that he is an expert at transcribing and storing and retrieving records with great longevity and accuracy.
Gunther suggests that he can keep doing this for as long as he's around, give or take.
Two hundred years ago, I told Gunther to file a transcription of a a 512-byte thing and to store it where he can find it upon my command for as long as he may live. Gunther did so with glee, and stored it however he did.
Today, 200 years hence: I ask Gunther to retrieve that 512-byte thing and email it to me. But Gunther cannot do so; for his perspective, the 512-byte record is gone. Perhaps he can't see it well-enough anymore, or perhaps it has been destroyed, or perhaps it was stolen or maybe even something else. Maybe he can see the record, but the contents just don't make sense to Gunther anymore after all this time.
Gunther is very sorry about this, and I still don't have my 512-byte record retrieved.
Whatever it was: Only Gunther really knew where that 512-byte block of information was stored, and Gunther was the only soul that had access to the area where it was stored, and Gunther is still very much alive to retrieve it. But Gunther still fails to retrieve it. He went to exactly what he knew must be the right spot, and he just couldn't get it done.
Gunther is a failure in this matter. It may have been a stupendous task, but Gunther failed it anyway.
I place a mark on his failure record, and Gunther knows that he earned that mark of failure.
This marked failure will affect Gunther's Mean Time Between Failures, or MTBF.
---
If Gunther is a hard drive, then: The hard drive now has a recorded failure. It may have taken 200+ years, but failure is failure.
(And in a pure and just world, we can chalk that failure up on the MTBF charts for everyone to see exactly what that MTBF has now become for any given Gunther/hard drive.
Except: We also need to know the success rates in order to tabulate that MTBF. Failure alone doesn't get us there.
And in hard drives, nobody publishes good data about success rates. We don't talk about the drives that live forever; they usually get disused and eventually recycled before they properly fail.
IE, we broadly only discuss failure rates (including outliers), and not success rates (we generally send those most-surviving drives off to make new cars out of in some foundry somewhere; we don't wait for them to fail).
Without both measures, a true in-the-field MTBF cannot be established. A percentage-of-death rate without a percentage-of-ultimate-life rate doesn't mean anything in terms of MTBF.
(Which brings us back to those randomly-selected EXOS drives from Seagate, with a rating of 2.5 million hours MTBF.))
You could say so from a consumer perspective, but it is not what the MTBF you quoted measures. Read errors are not considered a hardware failure. Not an opinion.
That's a lot of work to avoid accepting your previous comment was incorrect.
I prepended that chapter with a question: If the data storage hardware cannot read a block that it has stored within itself, then has the hardware not failed?
I then presented practical examples of how I understand failures to behave in data storage devices, such as Gunther.
Can you answer the initial question?
Or are you just here to brow-beat your version of what you believe to be correct -- without the benefit of citation, or logic, or reason?
(Remember, kind HN user: I did not dismiss you at all. I engaged on all cylinders. And I'm here to learn. If you have something to teach, then: Please teach it. If not, then: Please disengage.
I'm not interested in logical fallacies, though, like appeals to authority: Those non-constructs waste our time.)
And MTBF is with active use. Electronics which are not powered on can last almost indefinitely and under a huge range of environmental conditions. From freezing cold to scorching heat.
You just need to make sure you let the electronics acclimatize before powering them on. I always let my external HD acclimatize for 24 hours before powering it on after I grab it from my shed where it's been lying for months in freezing conditions. No problem whatsoever.
The post title is wrong and was poorly transcribed from the article. The product is named M-DISC.
It is a convention that optical discs are discs, and floppy disks or hard disks are disks. It is not a hard-and-fast spelling rule, but since there is indeed a set of official standards for Compact Disc and Digital Versatile* Disc, you can be sure that any conforming optical disc won't be spelled with a "k".
* Were you surprised that the "V" doesn't stand for "Video"?
100% everything here. That level of tamper evidence should be good enough for the reasonable individual, IMHO.
On the enterprise scale, the question I think everyone should be asking is: Do they have LTO tapes on a shelf? Fundamentally the same issue (but larger), in terms of needing to ensure it's physically separate, and secure, and tamper evident. =)
Anything in the cloud or connected to your network can be hacked. The backup of last resort must be on a shelf.
There's also this story [2]:
> I had a room full of servers, many with 16-36 disks per server and had an AC failure. Things turned off as planned, but it hit 115°F [46°C] for a bit. None of the disks died, majority of the tapes did.
[1]: https://www.ibm.com/docs/en/storage-deep-archive/1.3.1?topic...
[2]: https://news.ycombinator.com/item?id=21066981
Maximum wet bulb temperature (Only LTO-8 and earlier cartridges) / Maximum Dew Point temperature (LTO-9 and later cartridges), all across the board, allowable / recommended / shipping: 26°C (79°F).
In Florida, we like to refer to 79°F dew points as "Tuesday."
More to the point, the time I want those LTO cartridges to earn their keep is during a disaster scenario. The most likely disaster scenario--other than fire--is hurricane. High temperature and humidity, and no power to run the HVAC to mitigate those. (Unless you have generators with fuel to run for days, and your HVAC, generators, and fuel tanks survived.)
I doubt that the recommended storage conditions for qic tapes were much different from the above; and I've seen a system boot from a qic-120 tape that had been kept in a non-climate controlled ministorage locker in Santa Fe NM for 17 years.
If you want to archive up to 100 TB or even 200 TB, the cheapest way to do that is with HDDs, but you must keep in mind that you must use at least duplicate HDDs for everything, preferably stored in different locations, to avoid losing data due to files that become corrupted after a long term storage or due to some accident. Moreover, you must migrate the data to new HDDs periodically, e.g. every 5 years.
If you want to archive several hundred TB of data or more, magnetic tapes become cheaper and they are more reliable than HDDs. You still need to migrate the data, e.g. every 10 years, but much less because of the risk of corruption or mechanical failures, but because the tape format may become obsolete and appropriate tape drives that can read it might no longer be available.
Even with the inflated prices nowadays 3tb 3.5" sata drives are cheaper than 2.5tb LTO-6 tapes.
Personally I don't trust optical. I had some DVD-RW drives and many of them became unreadable after a very short time.
(I don't know what the absolute vs practicate reliability factors are for any of these)
EDIT: Hmmm... https://www.howtogeek.com/the-thousand-year-disc-that-failed...
Tape is amazing, and resilient, but the drives are fragile and expensive.
The sad part is that Panasonic/Sony/Pioneer have pulled out of making drives (I believe Panasonic still make some for their own vertically integrated products), which just leaves Hitachi-LG.
[0]: https://en.wikipedia.org/wiki/Blu-ray#Hard-coating_technolog...
If you only need ~50TB it’s probably both cheaper and easier to just buy hard drives, even today, factoring in the cost of the tape drive. Maybe even 100TB. I’m personally backing up ~225TB off-site these days and I still find AWS S3 glacier deep archive to be cheap enough - at current prices I’ve decided the hassle of doing tape again isn’t worth my time unless I’m over 300TB.
It's been a while since I did enterprise backup, but when I did, most backup software had a 'bootstrap procedure' that you could use even if you were completely destroyed.
First, most enterprise software does not need a license for restorations, only for backups. In a disaster recovery scenario you're unlikely to have the keys.
At end of every backup run/day, an e-mail is sent out with the tapes and offsets that have the indexes and configuration backups. You can send this message to an e-mail address that is separate from corporate e-mail (in case that gets wiped as well); back in the day you could send it to /dev/lp0 (and stick a copy in the box that the offsite tapes are transported in).
So even if your entire infrastructure is wipe (e.g., see NotPetya and Maersk), there's still a starting point for restoring your restore-backup server, and rebuilding up your infrastructure.
Remember: your backups have to be inaccessible to attackers.
https://www.lto.org/lto-generation-compatibility/
Linear Tape Open is remarkably Open for an enterprise product with Open in the name.
Personally, I feel like archival storage of personal digital data is not really solvable. I feel pretty confident about keeping my online storage going while I'm able. After that, there's nothing effective to preserve it when I'm gone. That's not a huge deal for me, I don't expect my heirs to want or need these things... Photobooks with annotations are more likely to last and have some chance of being apprechiated.
I really wish a multi-terabyte optical media solution took off, because that would be the path I’d ultimately take, instead of going with expensive LTO devices and media.
Optical just didn't make sense to me. Overall it was way more expensive, and a lot slower to burn BluRay discs.
The max BluRay disc size is 128GB (which is smaller than a lot of SD cards these days), and the fastest you can write to a BluRay disc is 18MB/s. That's a lot of of swapping discs to burn my many Linux distros.
And those 128GB BluRay discs cost between $30 and $40 each. Used tapes (with a lot of life left in them) on ebay are far less expensive per TB - the lowest I've paid is about $1.40/TB for LTO5 tapes without a case. I typically pay about $4/TB for unused LTO5 tapes with a case on ebay. It's getting a bit more difficult to find those deals, but they come up every now and then.
The bigger problem is finding a drive that can read and write them -- the 128gb format support is a bit harder to find. I thought I would be good with my nice Pioneer BDRW drive but it didnt have the support so I had to buy an LG unit. So now my desktop has two BDRWs -- how very 2006!
The LTO5 drive cost me about $150 on ebay, and an HBA card was $30. Yay for datacenter hand-me-downs.
I'm semiseriously considering getting the old LTO3 library from work and putting it to use. Sure, it's "only" 400GB per tape but I have a literal vanload - the racking takes up the same volume as a mid-size Ford Transit - of new and used LTO3 tapes to dispose of too.
A petabyte would only be 800kg or so!
> While the immediate burn quality seems relatively satisfactory, I was still not satisfied as to whether these discs were truly worth their premium over commodity blank media. Looking at the discs, it seemed that the recording layers were likely made of similar materials based on the visual aspect, unlike the obvious difference with DVDs. Also, it seemed that regular recorders were intended to be able to record to the BD-R’s (according to the Verbatim text).
...
> Digging further around the M-disc site, it is noted that their section on longevity testing actually features no data about the BD-R discs at all. Indeed, if there were tests, the reports were not actually featured on the site at all. Further poking around in M-Disc Technology pages shows only references to their DVD products. In fact, it seems their patented “etched in stone” material may only apply to their DVD products.
...
> From my research into the data, it seems likely that the M-Disc BD-R is not substantially different from other quality BD-R discs on the market as they all use inorganic recording compounds (with the exception of BD-R LTH of course). From a visual inspection, the disc appears the same bright chocolate-brown, unlike the obvious distinction with regular dye based DVDs and the M-Disc DVD which was a silver-grey. But my main concern lies in the demonstrated lifetimes of quality BD-Rs which are similar to the M-Disc DVD (to which the M-Disc BD-R is claimed to have also achieved). In essence, it appears that the MKM/Verbatim BD-R DL has already reached the same level of endurance as the M-Disc DVD had claimed without being “special” at all. It is thus, hard to justify paying more for an M-Disc BD-R, and I cannot really recommend it.
The "not really M-Disc" stuff comes from a post somebody made on the "Data Hoarder" Reddit board a few years ago. The discs were changed to enable higher burning speeds which resulted in a slight difference in appearance, but he/she didn't seem to have realized that Blu-Ray M-Discs were never substantially different from BD-Rs and instead assumed that the tweaked discs were more of a scam than the originals.
https://en.wikipedia.org/wiki/Golden_plates
https://en.wikipedia.org/wiki/M-DISC#History
Basically your media only needs to outlive your drives, retrieval mechanisms, software interfaces, and drivers. Or more properly, your backups need to outlive your live system and the need to restore any data from them. Certainly, data may be more precious than a live system, but you also need to concern yourself with format transfers, in terms of software file formats, filesystem formats, and media design/format.
This is not a difficult achievement. The people marketing long-term archival storage media are capitalizing on ignorance of people who have worked with books and photographs, which are "readable" by human senses. Books and photographs are media which can be preserved and fundamentally produced on archival media from the get-go.
Backup media, since it is necessary to have machines read and interpret them, are an entirely different beast, subject to an entire chain of things to preserve. As I've commented before regarding BD-RE XL, my media and the Blu-Ray drive itself outlasted the system it was backing up, and therefore did their job, without any investment or wasting money on M-DISC "Engraved in Stone Tablets" for the gentiles.
1. The disks bigger than 50GB require a BDXL drive to read, which means spending $300 on a disk player. Which also means rare to find in the future. So if you want larger compatibility use 50GB or smaller M-disk.
2. Japan sells supposedly original MDisk but they look rather black than gold, so they are not the original. Still strength and longevity is quite good on Blu-ray.
3. Storing the disk on materials that degrade like cloth/plastic sleeves may be counterproductive. I got a hard case but it takes effort and is increasingly hard to find.
Still the 50GB play nicely even on old players.
Now it's their problem to take care of, ignore, make fun of, or whatever, which was always ultimately going to be the case anyway.
This allows you to back with living your life while you still have one.
There's little point to bothering with predicting what the future will look like after you're dead, and even less point to trying to influence how others will behave on your behalf.
If you really want to have a lasting legacy of documentation, then build it and send it. Send it early, and send it often. Be the most interesting person on Earth today for your group.
Chances are fantastic that nobody will want to volunteer to build it for you after you're gone. But if it's important for you today, then make it easy on them: Send it right now.
...or don't. Really.
Telling some great stories at gatherings while perhaps sharing a couple of visual-reinforcement photos on a phone is enough unless you're a Kennedy or something. A wonderfully-interactive story is worth a thousand photographs. If you've got interesting stuff to share, then it will be interesting to hear, and to interact with, and it will remembered.
For awhile.
The best stories won't be chosen by you, but they'll be retold and will ebb and flow, and will subsequently die in a generation or three at most, but that's OK: You'll be dead, too. Get over it. By generation 4, those kids won't know who Uncle Fred was anyway except for a kind of sleepy myth about the carefully-preserved BD M-Disc archives (whatever those are) that were binned by Aunt Sally two decades prior because even Aunt Sally didn't care anymore, and couldn't be stuffed to keep them.
So keep living, and keep making and sharing new stories. Give people something to remember on their own accord. Don't force it, just do it and share it. Remember: If you were never very interesting to begin with, then they'll just bin whatever you left for them to discover when you're dead.
So be interesting. Live a decent and good life while you have it. Share it; no secrets. If it seems worth sharing after the funeral, then it's worth sharing right now.
(Except! In the odd chance that you are a Kennedy or something, then: Hire biographers. Hire them early, and often. Get that money, and line up the publishing of that stuff while you can still direct the proceeds. This is the path to preserving a legacy in a way that others can see down the road. (It might make it all the way to generation 5! (But it probably won't.)))
The story furthermore must have enough intrinsic drama to engage readers, and fall into the hands of a writer with the skills to turn the raw facts into a biography.
The biographic text provides a structure to hang the photos on, yielding a "photobiography". Search for the term and you'll find examples.
So why is it that photos >20 years old just aren't that interesting any more? Because photos need to be augmented by memories, or by stories from others. Once they're just pictures... and once I'm gone... no more interest.
We'd never throw out old physical photo albums. We have a cupboard full of them that my mom left behind. Some of the photos in them are of people we didn't even know, and now won't even know the names of. We keep them because we still have a little bit of connection to them - we knew the people to whom they were significant. But soon they'll just be landfill.
That's how human memory must work. We can't have every generation leave behind hundreds of thousands of pictures, piling up forever. Memories of past generations get purged until 2-3 generations back they're forgotten.
Of course there are exceptions to every rule. An uncle was persuaded to write his memoirs about his childhood on the farm in the middle of the last century. That will still be a good read after he, and those who knew him as a person, are gone. Assuming the books survive - it was a small, private printing only.
Whole lives of collecting sentimental goods all end up going in a giant bin, a whole house cleaned out in less than 2 hours. Photo albums are rarely kept, they just go in with it all and off to the tip usually in a single morning.
What you think is important to you, is basically just trash to your relatives/kids. They will probably have want they want before you are gone.
Being even remotely relevant to future generations is quite another.
I had so many of my users ask your question that I wrote this up: https://photostructure.com/guide/how-do-i-safely-store-files...
But a month or two after I initially wrote that, my brother and I found a couple boxes of photos from my parents and grandparents. No notes. No dates. No albums. Just envelopes and envelopes of undated photos.
Taking the time to make an annotated printed photo book, including dates, people and place names, and any other context is going to make those images a lot less irrelevant to future generations, and printed media, stored well, should easily last 50-100 years.
I would not make a bet that you'll be able to procure a bdxl reader in that time period.
Or just keep it attached to your system, so spyware, adware, and the NSA will make free backups for you!
The other thing I did was I ended up collecting HDD's out of old DVR's people were throwing out. Those drives are used for cold storage but not for anything important because they already have tens of thousands of hours on them and in not the best temperature conditions. Again, non have failed yet.
Funnily enough, the only HDD I have failing is a 2.1GB Quantum Fireball from 1997 in my Pentium 1! Really should swap that for an SD card via IDE.
It was a special system, Linux-based, running proprietary software for expensive proprietary hardware.
I apologized profusely. I got them some new hard drives coming right away. I found a forgettable donor drive in the boneyard that had a close-enough footprint for the connector, harvested it with hot air, and put its connector onto one of the broken client drives.
A little bit of dd and some mdadm magic later, and they were back in business with that RAID-1 array like it never happened (other than just a few days of downtime on a system they only used occasionally).
---
A different time, 25 or more years ago now, I somehow broke a 4-pin Molex[2] power connector off of one of my own IDE drives. I don't remember the details, but I do remember that I fixed it by soldering a Y cable power splitter into the holes on the board. This gave me a drive that worked, and that had power pass-through. It was kind of nice to have (and it worked for years and years after that).
Anyway, I guess shit happens sometimes.
---
[1] I wasn't rough with anything; I was just moving wiring out of the way, as one does, to attack the dust bison that had accumulated over the previous decade or more with some canned air.
I somehow broke the connectors of both drives, anyway. I can't explain it and I don't have to, so I guess I won't bother trying. They were fine when I started and they were broken when I finished.
[2]: Those aren't Molex connectors. Everyone calls 'em that, like everyone calls adhesive bandages Band-Aids, but the old drives used a design characterized by the Amphenol Mate-N-Lok connector design which had nothing to do with Molex.
I've been programming for 50 years now. I've had all sorts of failures, including two incidents of my desktop computer burst into flames.
I now set the desktop on a piece of sheet steel. I also only use steel enclosures for them.
I have a Carver amp that runs 24/7 in my office, for the last 40 years. It's now emitting a 60 cycle hum, which I learned is due to decaying capacitors. I guess I need to get a cap kit and fix it.
And the crazy thing is - we actually watch home movies in a linear format as a consequence. Rather than picking a clip here and a clip there amongst a sea of photos, video can be laid out in a linear timeline and watched without editing.
(I still use a professional Panasonic video camera that shoots AVCHD, which makes this super easy. Because editing video is not something I'm ever likely to have time to do)
The problem with archiving data is you always need to be prepared to use the "next big thing" once your preferred method becomes unobtainable.
I currently have :
- Photos in iCloud (Live working copy)
- Backup at home on NAS (updated nightly, snapshots)
- Backup in pCloud (lifetime, mine or theirs, no illusions, hopefully lasts a couple of months if I die unexpectedly)
- Yearly archive on external HDDs, identical copies stored in separate locations, clearly labeled "Photo backup". No encryption, no archiving, just plain APFS formatted drives that can be plugged into a computer and read.
- Yearly archive M-disc blu-ray discs, again, identical copies, stored in separate locations. One disc per year labeled as such, ie. "Photo Backup 2020". The idea being you restore them in chronological order and have a complete set.
I occasionally look at my own, but that's about it.
If an album is well annotated with dates and locations and names and descriptions, it might be fun, but I suspect the number of old photo albums with this kind of effort put in them is rare.
I wonder if there is a better printing technology comparable to analog "prints" (of film). That would actually make more sense (to me) than saving everything on M-Disc.
This is an _additional_ layer of backups, on top of the usual iCloud/whatever things. LTO is certainly not user-friendly, but neither is it especially user-hostile.
And I think I trust tapes far better than any other alternative for long-term storage as they have by far the smallest areal density of information compared to disks (hard or optical).
I have DVD disks that are barely 10 years old and that I can't read anymore. Hard drives also die in storage and/or during transport.
Recently the computer museum helped recover early 1970s unix tapes. It was a big deal.
And let's not exaggerate the complexity of setting up external tape drives, even on Windows/macOS.
Having found a matching Sun Ultra 5 generation drive, I tried to get it to work with a SCSI card, and had .. many many problems. It may be the rubber bands. It may be stiction. it may be ants. It may be me.
I would personally love to de-exaggerate the complexitity: Old hardware is gnarly. I sourced parts, it wasn't working.
And yeah, I started with a ZX-Spectrum with cassette tapes. I was pleasantly surprised by the overall LTO experience.
- A photo album extended with a curated set of photos each year. The real deal!
- NAS at [my] home and blu-ray off site (parents home) for all photos
Updated: added [my] for clarification
I just keep my offsite backup drives in my office drawer. There is a note on them that says they are personal backup and not to be plugged into company equipment. If you have an office this is probably the easiest solution.
If a disaster wipes out my home and my office then there are bigger things to worry about than my data.
With hard drive prices being what they are, I feel like my decision is increasingly justified. Storing large amounts of personal data (documents/photos etc) either in the cloud or "hot" on local drives, where the access patterns are infrequent, has simply stopped scaling.
Having discipline around burning disks has also made more disciplined about archiving.
The burn speeds of 6x Bluray media aren't substantially slower than SMR hard drives. During a recent archiving session, my SMR 4TB hard drive was sustaining 60MB/s. My Bluray burner sustains 27MB/s.
A pack of 20 BD-R DL Blurays (50GB each - a total of 1TB) from Japan was less expensive than a 1TB 2.5" external hard drive.
Cloud storage isn't tenable either. Despite having a 1000/400 fibre connection, with international links from Australia being down, backing up to Hetzner only sustains 5MB/s - backing up 5TB to AWS or equivalent would be faster, but likely to cost around $700 a year. This is on top of the $140 a month for business grade internet with reasonable upload speeds.
I'd love an LTO drive, but that requires a server or appropriate desktop, and seems overkill for 5-10TB. Even for secondhand drives, I'd be looking at multiple thousands for a a complete setup.
Continually upgrading the NAS with more bays and more drives is also not sustainable.
I index and catalogue each disk with Neofinder (great piece of software) that allows offline browsing and searching of each disk. The searching and browsing is more performant than the software provided on my Synology.
And as an upside, videos and photos of the kids can be viewed by anyone capable of sticking a disc in the drive, without needing access to servers or cloud providers (I burn some in BDMV format with no DRM, so it should play in any Bluray player or computer).
https://www.tautology2.net/2022/12/14/where-will-the-arrow-o...
> They needn’t be particularly beautiful, either. Better not to stir strong aesthetic or religious feelings with inscriptions and the like.
I remember a discussion on architecture where the point was that, while newer buildings are built with more sustainable techniques, they are also ugly and therefore no one cares if you want to demolish one. Buildings perceived to be pretty, on the other hand, are more likely to stay around longer and therefore have a lower environmental impact in the long run.
Making your M-Disk aesthetically attractive may be a way to ensure that they are around for a longer time, if only because someone may confuse them with electronic waste and take them somewhere away from nature to ensure they are properly destroyed.
Just use a magnetic hard disk and your data will be fine for many decades if not centuries. You don't need M-Blu Rays (which have very limited capacity anyway). A hard disk is cheap and stores several Terabytes.
You obviously need to store the data in a physically remote place like your shed, car or safe deposit box in case your house burns down.
Never, ever use a SSD if you want your data to be safe for any amount of time. What's worse, SSDs can fail unexpectedly with almost zero chance of data recovery.
A few months ago I needed to pull from an offline backup. The disk, which used CMR and was purchased new from an actual computer supply company (i.e. not Amazon), was burnt in for about 72 hours before it was loaded up with data and shoved in a padded box. Ten years later, the drive was pulled, and there were so many media surface errors that most of the data was unrecoverable. (I haven’t bothered opening the drive to see if there is some obvious sign of what exactly failed.)
In my case the backups were created in multiples using two different brands for exactly this reason, and the second copy was fine. But I wouldn’t say it’s the case a single magnetic disk can take the place of BD.
Mechanical disks are also getting expensive. The disk purchased to replace the failed backup was being sold at three times its original release MSRP. Some of this is due to the AI hysteria but some of it is also that the market is shrinking and becoming K-shaped.
For example, if most files are multiple gigabytes, the drive has 3 million tracks, and 3 thousand of those tracks are ruined: one degraded copy leads to most files being broken, two degraded copies leads to only a single digit number of files being broken, and three degraded copies means >99% odds of recovering everything.
And with the magic of par2 files, a single drive with a couple percent of corrupted tracks can be fully recoverable. Any cold backup I do gets some of those.
Comically untrue, and trivial to verify that it is.
You obviously need to store the data in a physically remote place like your shed, car
Let's make it worse by keeping them someplace with terrible environmentals.
If you're gonna be wrong, don't be just a little wrong...
I also have a binder full of DVD-Rs, of about the same age and containing mostly the same data, that all read perfectly with no issues.
You're asking for trouble if you count on just one backup solution to keep your data safe.
It is absolutely not safe for many decades. If you haven't encountered any drive failures yet — you're lucky.
> A hard disk is cheap and stores several Terabytes
You should check prices again, I wouldn't say cheap considering price bump during the last year (or around so). M-Disks are way less convenient, but pretty reasonable for their cost.
The failure of SSD's you warn of and the way the HDD's fail due to a dead controller aren't that different.
This won't happen with a physically written media like a CD, DVD, or Blu Ray disc. With physical media, you're separating out the media from the reader (the drive). If your disc reader dies or you cant find it, etc, you can go find any another one on ebay even decades from now, very likely.
Doing surgery on an HDD controller PCB to replace a dead controller, while possible, sucks, and may be incredibly difficult if not impossible to source parts for in the future.
I don't expect anything with moving parts to last 1/10th that long, but MTBF is MTBF. :)
[1]: https://www.seagate.com/files/www-content/datasheets/pdfs/ex...
This recent study I found [1] shows almost 4% HDD loss after only 2.5 years for consumer hardware, enterprise products being a bit better.
[1] https://research.cs.wisc.edu/adsl/Publications/latent-sigmet...
---
Suppose I hire a scribe named Gunther, and this scribe named Gunther confidently states that he has a lifespan of a mean ~quarter-millennium. He can't really explain how he does it in simple terms, but Gunther states that he is an expert at transcribing and storing and retrieving records with great longevity and accuracy.
Gunther suggests that he can keep doing this for as long as he's around, give or take.
Two hundred years ago, I told Gunther to file a transcription of a a 512-byte thing and to store it where he can find it upon my command for as long as he may live. Gunther did so with glee, and stored it however he did.
Today, 200 years hence: I ask Gunther to retrieve that 512-byte thing and email it to me. But Gunther cannot do so; for his perspective, the 512-byte record is gone. Perhaps he can't see it well-enough anymore, or perhaps it has been destroyed, or perhaps it was stolen or maybe even something else. Maybe he can see the record, but the contents just don't make sense to Gunther anymore after all this time.
Gunther is very sorry about this, and I still don't have my 512-byte record retrieved.
Whatever it was: Only Gunther really knew where that 512-byte block of information was stored, and Gunther was the only soul that had access to the area where it was stored, and Gunther is still very much alive to retrieve it. But Gunther still fails to retrieve it. He went to exactly what he knew must be the right spot, and he just couldn't get it done.
Gunther is a failure in this matter. It may have been a stupendous task, but Gunther failed it anyway.
I place a mark on his failure record, and Gunther knows that he earned that mark of failure.
This marked failure will affect Gunther's Mean Time Between Failures, or MTBF.
---
If Gunther is a hard drive, then: The hard drive now has a recorded failure. It may have taken 200+ years, but failure is failure.
(And in a pure and just world, we can chalk that failure up on the MTBF charts for everyone to see exactly what that MTBF has now become for any given Gunther/hard drive.
Except: We also need to know the success rates in order to tabulate that MTBF. Failure alone doesn't get us there.
And in hard drives, nobody publishes good data about success rates. We don't talk about the drives that live forever; they usually get disused and eventually recycled before they properly fail.
IE, we broadly only discuss failure rates (including outliers), and not success rates (we generally send those most-surviving drives off to make new cars out of in some foundry somewhere; we don't wait for them to fail).
Without both measures, a true in-the-field MTBF cannot be established. A percentage-of-death rate without a percentage-of-ultimate-life rate doesn't mean anything in terms of MTBF.
(Which brings us back to those randomly-selected EXOS drives from Seagate, with a rating of 2.5 million hours MTBF.))
That's a lot of work to avoid accepting your previous comment was incorrect.
I then presented practical examples of how I understand failures to behave in data storage devices, such as Gunther.
Can you answer the initial question?
Or are you just here to brow-beat your version of what you believe to be correct -- without the benefit of citation, or logic, or reason?
(Remember, kind HN user: I did not dismiss you at all. I engaged on all cylinders. And I'm here to learn. If you have something to teach, then: Please teach it. If not, then: Please disengage.
I'm not interested in logical fallacies, though, like appeals to authority: Those non-constructs waste our time.)
You just need to make sure you let the electronics acclimatize before powering them on. I always let my external HD acclimatize for 24 hours before powering it on after I grab it from my shed where it's been lying for months in freezing conditions. No problem whatsoever.
It is a convention that optical discs are discs, and floppy disks or hard disks are disks. It is not a hard-and-fast spelling rule, but since there is indeed a set of official standards for Compact Disc and Digital Versatile* Disc, you can be sure that any conforming optical disc won't be spelled with a "k".
* Were you surprised that the "V" doesn't stand for "Video"?
I have so many photos of vacations and people who I've buried over the years, that I don't want to lose.
So I was thinking about using Blu-ray M-Disks as well, these last couple of months.
On the enterprise scale, the question I think everyone should be asking is: Do they have LTO tapes on a shelf? Fundamentally the same issue (but larger), in terms of needing to ensure it's physically separate, and secure, and tamper evident. =)
Anything in the cloud or connected to your network can be hacked. The backup of last resort must be on a shelf.
Most enterprise gigs I've worked sent them to Iron Mountain or equivalent.