Long time, no me, & archiving data

[Zoidberg]

Howdy do, folks.  Long time, no me.  'scuse the absence, lots of personal health and family health issues to attend to the past couplafew years.  Doing okay, though.

I have listened to the funny bands occasionally over the past year but haven't heard much.  I had to dismantle any external antennas at my apartment complex, so mostly I take a portable outdoors for occasional listening.  I did hear TCS recently but didn't log it.

I've received a few private messages over the past year or so asking about some off-air recordings.  Apparently my old mediafire account was suspended because of the word "pirate".  Their ban-bot assumed it was illegally pirated copyrighted material.  Archive.org is better for this stuff anyway.  They have a streaming player and you can download the files.

I fell way behind in uploading files to my "canklecat" archive.org account back around 2011-2012 because my old PC and slowpoke ISP couldn't cope with Flash and uploading large files.  Since then I've upgraded everything and can begin to catch up.

BTW, don't overlook Sealord's outstanding archive of off-air recordings on archive.org.  Sealord gets much, much better reception and his archive dates back to the 1990s.  Some of my off-air recordings are painful to listen to due to high local manmade static.  That's why I stopped recording off-air around 2012. It was a losing battle in my area.  But I did snag some good recordings of stations west of the Mississippi, so occasionally I got better recordings than some other folks, especially when Squishy the Sunspot was favoring Texas.

Coincidentally I've been transferring all of my audio, video and photo files from CD/DVD to external hard drives.  I noticed nearly half of the CDs/DVDs I burned 10 or more years ago are failing, so I've been busily re-archiving stuff the past month.

Lesson learned?  No older style CD or DVD type seems to be stable beyond 10 years, if that.  I've tried CD-R, CD-RW, DVD-R, DVD-RW, DVD+RW.  Doesn't matter which dye or metal film type they used.  Sony, Fuji, Maxell, etc.  All of 'em seem to have similar failure rates.

Some of my photographer friends recommend the M-Disc DVD for longterm storage.  It requires a special type of burner, but those are affordable too.

For now I'm using a couple of Western Digital Passport Ultra external drives - these are  small, lightweight, USB 2/3 powered, no external power source needed.  Only 5400 RPM but with WD Smartware running in the background at night there's no serious burden.  Unless you're a photographer you'll probably prefer Windows Backup, Acronis or drag-and-drop file saves.  WD Smartware is very specialized stuff - it's a true backup program that does not allow deleting archived files.  The only way to retrieve space on the backup drive is to start over.  Good because it's goofproof.  Not good if you inadvertently archive too many redundant copies or junk files/folders.

BTW, I just got a Western Digital My Cloud 3 TB device as well.  So far my impressions are mixed.

On the plus side:

• *Once it's loaded up and hooked to a wifi modem, it will serve as local "cloud" storage for all your compatible devices.  It's mainly intended as  a media server, but I plan to use it to store my photos, including raw files, for occasional editing on my laptop.
  
  *"Cloud" is a misnomer if you plan to use it only as local wireless network attached storage.  (But don't try to upload massive files to the My Cloud device over wifi - that's painfully slow.)
  
  *It is intended to be useful for remote access via passwords, but I probably won't bother with this.  And it is streamlined for Dropbox, but I won't use that either.  I'd prefer Amazon's S3 or Glacier service.
  
  *It uses the WD Red drive, which is rated for 24/7 server operation.
  
  *The Red drives are pricier than the Blue and Green, but when bundled with the My Cloud case (internal CPU, NAS stuff, etc.), it's reasonable as a grab-and-go Network Attached Storage device for rookies and cheapskates like me who can't afford a full blown Synology NAS rig.
  
  *The case has good passive cooling so it's quiet.  And the Red drives do run hot.
  
  *It's the size and weight of a hardcover dictionary.  If all else fails you can grab it and go in an emergency.  (But it needs time to spin down and park the heads.  Instructions say 30 seconds but in reality it's closer to 5 minutes due to delays in network disconnections.)
  
  *It includes Smartware Pro, which can be used with any brand or type of storage, not just WD brand.
  
  *Despite the instructions indicating a gigabit router is necessary, it isn't.  Direct ethernet connection between PC and My Cloud will work just fine.
  
  *Drag-and-drop transfers over ethernet cable are reasonably quick, comparable to USB 3 speed.
  
  *Western Digital's free concierge phone service is great.  I spent 30 minutes or so via Google phone with them to walk through the initial setup.

On the downside:

• *There's no direct USB 2/3 connection between PC and My Cloud.  It's gigabit ethernet or wifi only.  The single USB port on the back is for piggybacking another external drive to extend the My Cloud, not for direct connection between PC and My Cloud.
  
  *If you prefer USB 3, check out the non-NAS WD My Book drives, which use WD Green drives.  The Green drives are intended for low energy consumption and idle when not in use.  But the My Book devices aren't wireless NAS.
  
  *Smartware crawls painfully slowly between PC and My Cloud.  Like 1 GB per hour.  I have nearly 1 TB of photo files to backup, so this isn't feasible.  WD is aware of the problem but hasn't offered a solution.  And Windows 7 Home version doesn't support Backup to network devices - gotta have the pro version for that.
  
  *So for now it's drag-and-drop.  This effectively relegates the My Cloud to server status as an external drive, not true backup.  But this is nitpicking for many folks.  In photography/video archiving parlance, backup usually means one-way, with little risk of accidentally overwriting or deleting the backup.  For most folks, this isn't as significant an issue.
  
  *Warming up the My Cloud after a shutdown or idle state takes awhile.  The initial warmup was supposed to take minutes - it took over 12 hours.  Since then it takes about five minutes to be network-ready again after a shutdown (due to our many recent thunderstorms) or idling overnight.  Reportedly there are workarounds to minimize the idling and restart times, but I've been so busy transferring files to the new drive I haven't had time to explore those options.

Anyway, I'm satisfied with WD's pre-packaged My Cloud, My Book and Passport devices because they're good values and compact.  Most of all, I need grab-and-go external storage.  Unlike the WD My Cloud device, the Passport drives can be ejected and disconnected in seconds.  My apartment complex has mostly seniors and disabled folks.  We get emergency response calls - medical and fire - at least once a week, sometimes more often.  I like being able to grab a lightweight portable drive and have my data, which is also backed up to a couple of cloud servers.

But if you're in a single family home and don't have to worry about forgetful neighbors who set their microwaves for an hour instead of 10 seconds for a lemon pie, or burn their toast every dang day, you'll probably prefer a Synology or WD multi-drive NAS and/or RAID unit.

[ChrisSmolinski]

Great to see you again, Lex!

Thanks for the tips on backing up data, and the reminder about the finite lifetimes of CDs.  Lots of folks use them for backing up family photos, etc, and it would be incredibly depressing to go back 20 years later to view them, and find that the information is gone.

[Zoidberg]

Yeah, I was shocked by how badly some of my CDs and DVDs had deteriorated in normal household storage.  I was about to prep a 10th anniversary photo album for a couple whose wedding I photographed years ago.  I usually made triplicate copies of the CDs/DVDs, usually one set spanned across multiple CDs, another to DVD-R and another to DVD-RW or DVD+RW.  But in some cases every disc had some errors.  Didn't seem to matter which PC or software I used either.   

And those were the good ol' days when a 1 GB media card was considered enough for a session.  With the current crop of 16-24 or more majorpickle dSLRs even an 8 GB is only adequate, especially if you're shooting video too.  CDs/DVDs aren't even practical for that sort of backup now.

I haven't done a serious statistical analysis but my impression is that DVD-RW may be a better way to go (other than the pricier M-Discs).

Most of the errors I've noticed were in DVD-R and DVD+RW.  The latter surprised me because the advantage to +RW is supposedly in the error checking.  But it doesn't seem to matter which burn protocol I used: Nero, In-CD, drag-and-drop, others.

Some discs were unreadable on the Pioneer drive in my desktop, so I dug out my 10 year old LiteOn external USB burner/drive.   Perhaps it's the horizontal orientation (the minitower's is vertical) with less wobble, or the additional error checking capabilities between Nero and the LiteOn drive.  But in some cases I was able to recover data from balky discs by using the LiteOn drive and slowing down the read speed.

And I noticed some of my commercially prepped movie DVDs are frosting over - the discs look cloudy.  They still play.  But I'm considering Handbrake or WinX DVD to make backup copies.

Anyway, back to archiving my pirate radio audio before they turn to digital dust as well.

[ChrisSmolinski]

As the price of USB flash continues to drop, I wonder how good that would be for long term storage? One concern would be bit errors due to background radiation, cosmic ray hits and the like. I've seen figures between 10 and 100 years for USB flash drives. Whether we will still have USB ports in 100 years is another question. 

[Fansome]

Years ago I did all of my backups to digital tape, using a product called "Fastback". At the time it was supposedly the fastest backup application on the market, and I was very happy with it.

Then, one day, my hard drive failed. I patted myself on the back, as I had been religious about doing backup. I bought another hard drive, installed the OS, and then went looking to buy the latest version of Fastback to restore my files. Lo and behold, Symantec had bought the company that made Fastback, and one of their first actions with it was to stop selling Fastback, probably because it was a highly successful competing product. I was unable to restore my files. Just looking around on the net now, it looks like Symantec eventually made Fastback available on their web site, but at the time the Web was in its infancy, so that wouldn't have helped me.

So, not only do you have to worry about the permanency of your backup media, you also have to worry whether the technology used to restore the backup will still be available in the not-so-distant future; this applies to software, hardware, and data formats. A number of years ago Scientific American had an article on these topics; it raised all of these issues. Of course, the technology was a lot less advanced then, but the problems of long-term data storage remain the same.

Personally, I'm sticking to paper tape.

[Zoidberg]

Well, the photograph site I've haunted since Hector was a pup is arguing about that very issue right now - the stability of NAND memory.  Not only the read-write cycles but the long term stability of data even in fresh or little used flash memory.  I'm just observing from a safe distance while the "experts" posture, pout and pontificate.

The consensus appears to be that USB flash drives should be refreshed every few years by rewriting the data.  Presumably that would involve copying from one flash drive to another, or to the hard drive and back to the flash drive.  With USB 3 this would be pretty quick.

There's also some debate over the stability of data on unpowered SSDs.  I may get one for my laptop but I wouldn't consider it for reliable storage, just to speed things up a bit.

Anecdotes suggest some media cards typically used in digital cameras and other devices may be incredibly resistant to adverse conditions.  Some digital cameras have been recovered from months or years of immersion in ocean water and the media cards could still be read.  However, resistance to corrosion and long term data stability over a decade or so aren't necessarily the same thing.

With media cards getting so cheap there have been a few photography projects for which I've used the Compact Flash and SD cards for storage for a couple of years.  Now that I'm seeing how unreliable CD/DVD is, I may consider dedicating some media cards to permanent storage for some of my medical documentary photography projects.

[Zoidberg]

I kept a Pentium II with tape drive backup for several years, but quickly outgrew it when photo archiving became a priority.  Ditto zip drives.  Those are fine for text and number data but impractical for most media.  I still have a few 3.5" sorta-floppies that each hold one full resolution JPEG from my first digital camera, and a few zip discs that may hold one raw file.  I keep them mostly as reminders of hubris.

I suppose the photographer's equivalent to paper tape would be negatives and prints.  Those require no special equipment to read, other than eyesight.  But as the family archivist of old photographs, it's a lot to keep up with and it's all a single natural disaster away from total loss.

For awhile I was skeptical of cloud backup for my entire collection of photos and videos, but as an experiment I opted to allow syncing Picasa and Google Plus to upload all of my images to their freebie storage.  Everything is automatically resampled to 2048 pixels on the longest edge, but it's still reasonable.  Letting it run constantly in the background wasn't as much a burden on our middling speed DSL as I'd expected, and it ingested about 500 GB of images and videos over the course of a month.

So I'm seriously considering Amazon S3 or Glacier now.  They seem committed to long term cloud storage.  And if it's good enough for the CIA, it's good enough for me.  Besides, the NSA keeps backups for all of our data in their secret nest in Utah. 

Years ago I did all of my backups to digital tape, using a product called "Fastback". At the time it was supposedly the fastest backup application on the market, and I was very happy with it...

Personally, I'm sticking to paper tape.

[ChrisSmolinski]

I should look into S3/Glacier for backing up genealogy data. Not just what I've entered, which is really only several MB, but the tens of GB of photos I have of microfilms, church record books, etc. A lot of that data would nearly impossible to replace. The many GB of photos of records would be uploaded once and never touched again, except in an emergency due to data loss.

Lex, you were talking about using older optical drivers earlier to read a misbehaving disc. A short time ago, I got a freshly burned disc from a researcher who was doing some work for me in an archive in Europe, photographing old church books. The disc was slightly damaged during shipping, and I was getting a bunch of read errors. On a whim, I tried it on a much older Mac, and the disc was able to be read with zero errors. 

[Zoidberg]

Yup, I was thinking about genealogy stuff too.  My dad did a ton of that before he died in March this year.  He added some to a web based genealogy site but the site is a bit slow and cumbersome to use.   I think he did most of his stuff with Family Tree Maker.  It's a lot of work, some of it pre-web, so quite a bit of snail mail and meatspace research was involved.

That, plus some of my photos, would be suited to Amazon's Glacier, which is lower cost in exchange for somewhat restricted download access.

Incidentally Amazon Prime now includes unlimited free cloud storage, which I've been using.  It will accept camera raw files, which Google Plus free storage does not.  But there's no syncing, no automation, it's basically just drag-and-drop file transfer.  So I've already unintentionally uploaded some duplicates.  But it's handy for folks with less intensive needs who are already Prime subscribers.

Regarding the older drive, I'll give that a try.  If all else fails I did keep my old Pentium II Windows ME PC that I used to burn many of those pirate radio discs.  So far the LiteOn external drive has been able to cope with many errors, although it takes longer.  It seems to automagically know to adjust read speed and will make repeated passes in an attempt to ferret out the data.  Fingers crossed...

I should look into S3/Glacier for backing up genealogy data...

...you were talking about using older optical drivers earlier to read a misbehaving disc. A short time ago, I got a freshly burned disc from a researcher who was doing some work for me in an archive in Europe, photographing old church books. The disc was slightly damaged during shipping, and I was getting a bunch of read errors. On a whim, I tried it on a much older Mac, and the disc was able to be read with zero errors. 

[ChrisSmolinski]

Thanks for the tip about Prime and free storage! I didn't realize that, so I will certainly look into it.

[Zoidberg]

Here's a photo of my PC with the WD My Cloud for scale.  The My Cloud doodad is in white, between the minitower and old LiteOn external CD/DVD burner.

The WD Passport Ultra USB portable drive is on top of the green thing.  It's pretty small.  There are similar portable drives from Seagate and others, but I've had good luck with Western Digital so I usually get those.

For the well heeled survivalist data hound, ioSafe makes rugged portable drives in aluminum billet cases that are crush proof, waterproof and darned near bulletproof.  But pricey.

After I've finished loading up the My Cloud via the ethernet connection, I may actually put it on a shelf away from the PC, along with the modem and router.  Right now it's taking up space my Palstar, speaker and DSP noise reduction unit used to occupy.

[Sealord]

Good to see ya back man!