[Yog Sothoth]

I started this thread here as an offshoot from the thread "Thoughts on current (2022) state of the art in CD ripping, and on .flac vs .wav?
" as I was afraid i was hijacking the original OP's discussion.  Perhaps this is better positioned under the general IT circle though?  Moderator please advise!  I'm lost, lost in the wilderness!

[Yog Sothoth]

I'm a ZFS fan as well. I have a QNAP that has ZFS and two Truenas Scale servers, I use synchthing to keep them up to date. I'm paranoid about losing my music.

I've continued this here from the original thread...

Yes, I suspect there are many of us with large collections now and storage deserves some serious consideration.  I also suspect many haven't given thought to data integrity or backups of any sort, which I find concerning.  There's an old saying:  There are two types of computer users - those who have lost their data, and those who are going to lose their data.

I would think that online cloud storage would be an option too, depending on size of the music library, storage cost, and bandwidth.

It sounds like you have gone through great lengths to ensure the safety of your data!  Do you prefer either one of the QNAP  or TrueNAS over the other?  I am familiar with both but have no intimate experience with either.

[mgalusha]

I have about 5.5TB of music, losing it would really irritating. My paranoia likely stems from being an IT guy for about 30 years. As you say, there are only two types, those who have lost data and those who will. I still have a vivid memory of formatting the C: on a DOS computer in the 90's. My bacon was saved because I had an un-format program on my network and it was reachable even though I'd formatted the drive, the command shell still worked so long as I didn't shut it off. I was able to un-format the drive and didn't lose any data, but it certainly wasn't a good day.

On Truenas vs QNAP. They are different animals. QNAP is a pretty friendly interface to Linux, they have several versions, one uses the ext4 file system and the other ZFS. One nice thing about the QNAP devices is a large assortment of add ins. I have my Roon server running directly on it and the primary store of music is on that box. I have 6 x 8TB, 2 x 500GB SSD's and 2 x 500G NVME's on the QNAP box, all ZFS, the two drive pools as raid 1 and the 6 drives as Raid Z2 (raid 6).

Truenas comes in several flavors, Truenas Core uses BSD Unix, and for a pure storage solution, is probably a better choice. While it can run other services, it's kind of a pain to set up "Jails" to run them. I've done it but it takes some doing to get things just right. The other version is Truenas Scale, it's based on Debian Linux and has really good support for VM's  and docker packages. I have a Truenas Core machine at work, it has been trouble free. I was using it here until scale released a few months ago, I moved to that and killed my VMWware lab server, it was hosting a couple of Windows servers for testing, along with the odd Linux instance, those live on the Truenas server and perform very well. It's not as full featured a VM host as VMWare, but it's way more that good enough for my use and has been rock solid. It's an actual server class box, an older Dell, dual xeon's, ECC memory, dual power, etc. The second Truenas is also running scale, but not apps except for synchthing, which is pushing data from the main server to the second Truenas. It's based on a desktop pc.

I have thought about online storage, but I have about 12Mbps of outbound bandwidth, it would take months to push all the data to a cloud provider. Perhaps, if the promised gigabit fiber is actually installed, I will reconsider.

Probably way too much info.

[RipVanW]

No, not too much info at all.  Very interesting.

I don't think I've ever formatted C: before, but I've certainly worried about it back in those days.  I've lost data over the years, sometimes able to retrieve it, sometimes not.  The times you can't definitely spur one to create backups - and backups of backups.

I've not heard of QNAP before, so I'll take a look at it.  I don't do much with VMs at the moment, so that aspect isn't likely to be a big draw.  For now I'm just looking to set up a rock solid NAS to store music and general network backups from the various laptops and desktops that I have around the house.  Currently those are all just standalone backups, and it would be nice to have them centralized and protected with RAIDZ or RAIDZ2.

There's a good used PC place around here that I might go to in order to look for a server-class box.  They have a lot of stuff that comes in off of lease, etc.  Outside of the obvious things that you've cited (ECC memory, dual power, xeon processors, sufficient drive bays etc) is there anything you'd recommend as a minimum to look for that would be unique to audiophile usage vs a typical data NAS?  Particularly Dell, since I've pretty much standardized on them for anything that I don't build myself.  I'm sure the used store will put a definite premium on an enterprise server over what they charge for a used desktop system, so I'm also looking into building my own NAS server using a 8-slot case and a traditional desktop mobo - but even that will cost some $$.

My outbound bandwidth is almost identical to yours, so until they get fiber running down the street, cloud backup isn't likely to happen (not that I'm very big on that anyway - I prefer to keep my stuff all local.  I have trust issues when it comes to storing anything but email online...).   

[WGH]

Wow, a lot of heroically complicated backup ideas. I can understand a need for real time backup for business data or a stable of laptops and desktops but for music too?
I make a backup image of the C: drive every few weeks, individual folders can be backed up in the cloud and emails are on msn or Google if I ever need them but I haven't received an important email in years since I retired.

I sometimes get new music once a week but usually once or twice a month. It's easy to plug in a USB HDD to copy the one or two albums to the backup, it take 5 minutes at most. In 20 years I never had an unplugged HDD go bad. There are stories of unused, unplugged SSDs loosing data over a long period which is why I stick with HDDs.

Seems like if there is an active backup of multiple RAID drives constantly running, a drive will die every year. And most of the time the drives are not backing up anything, just humming in the background waiting to fail.

Am I missing something?

[Yog Sothoth]

Wow, a lot of heroically complicated backup ideas. I can understand a need for real time backup for business data or a stable of laptops and desktops but for music too?
I make a backup image of the C: drive every few weeks, individual folders can be backed up in the cloud and emails are on msn or Google if I ever need them but I haven't received an important email in years since I retired.

I sometimes get new music once a week but usually once or twice a month. It's easy to plug in a USB HDD to copy the one or two albums to the backup, it take 5 minutes at most. In 20 years I never had an unplugged HDD go bad. There are stories of unused, unplugged SSDs loosing data over a long period which is why I stick with HDDs.

Seems like if there is an active backup of multiple RAID drives constantly running, a drive will die every year. And most of the time the drives are not backing up anything, just humming in the background waiting to fail.

Am I missing something?

I moved to the setup I currently use for a number of reasons, but there were two prime movers:

1) I used to use a setup much like you're suggesting.  The main drive failed.  I pulled out the backup to restore from and it had corrupt data throughout.  I lost everything.  I suppose I could have had a battery of external disks that I rotated, but at that point a simple RAID/Mirror setup seemed just as easy.  Since I have my OS spin down the drives after about 10 minutes of inactivity they are frequently (mostly?) just sitting there motionless.  Or they are SSDs, which I have had very good luck with.

2) I store much more than just entertainment media, some data of which is crucial (insurance information, taxes, blah blah blah).  It's total is far too much to efficiently store in the cloud (although I expect that to change as bandwidths increase).

Also, one difference perhaps for me is I spent 30 years in IT supporting Unix/Linux servers and networks so for me setting this stuff up and automating it is like eating a stick of cotton candy.  Except I don't feel sick afterwards.  Once it was all setup, it was even more trivial to populate a small NUC, move it offsite, and then automatically run nightly syncs.  So I feel my critical data as well as music/movies has protection against catastrophe.  Until the earthquake comes.

Interesting, I retired a few years ago too (early, due to medical problems.  I miss my job.) and I still get very important email that I would not want to lose (but which, like you, is stored on a server somewhere).

[rif]

2) I store much more than just entertainment media, some data of which is crucial (insurance information, taxes, blah blah blah).  It's total is far too much to efficiently store in the cloud (although I expect that to change as bandwidths increase).

I don't think I'll ever be comfortable storing tax and other sensitive docs in the cloud.

[Yog Sothoth]

I don't think I'll ever be comfortable storing tax and other sensitive docs in the cloud.

Indeed privacy issues are a concern.  For files in cloud storage I use encfs to keep the backend file storage (in the cloud) encrypted.

[RipVanW]

Wow, a lot of heroically complicated backup ideas. I can understand a need for real time backup for business data or a stable of laptops and desktops but for music too?

Many would say the same thing about 'heroically complicated' efforts such as low noise power supplies and optically coupled signal paths to get a small incremental improvement in sound quality. 

Here's the way I'm looking at it: 
Some form of network-based centralized storage mechanism is needed in order to be able to listen to the music files on more than one device.

I already make ad-hoc data backups for multiple computers in the house.  Because of having experienced multi-level failures like Yog described, I've kept multiple backups on multiple HDDs over the years.  That works, but the reality is that those backups only get updated once every several months.  It hasn't bit me yet, but someday it could.  Setting up a centralized NAS is a way to make it simpler to update those backups on a more regular basis, and fix that problem.

Setting up a NAS and periodically backing up that NAS to an external (preferably off-site) drive allows me to kill two birds with one stone, by using the NAS for the music store, and providing a target that I can send my general system backups to.

If you're going to have a NAS, then there's no reason not to set up at least a minimal RAID as an on-server backup.  I'm familiar with mirror RAIDs from the systems I deal with at work, and striped RAIDs from past reading.  Historically, hardware RAID has been given the nod for best performance, so I started looking into what the good choices were these days for that.  While investigating that, I found that the integrated RAIDs and cheapie PCIE RAIDs weren't worth much, and even the expensive RAID cards had lost their shine, focusing more on throughput than data integrity.  That's when I read about ZFS, and how it could (with sufficient additional drives in the array) detect and fix bit errors.  So.. if you going to be setting up a new NAS with some form of RAID, why not spend an extra couple hundred bucks and use a RAID format that ensures data integrity?  Cheap cost for the added benefit.

That's how I ended up with the current plan that I have.

Oh, and btw, the idea of using an enterprise-grade server?  Probably not.  ECC?  Check.  Dual PS?  check.  Multiple hot-swap HDD bays?  check.  Array of fans that sound like a jet engine?  Check - and deal killer.  The first time it powered up would be the last time it powered up, as I would be told to get it the hell out of the house...   Back to the other idea of an 'enthusiast' NAS decent mid-tower case, mini-ITX ATX mobo and silent fans. (edit: I nixed the original case I had in mind because I couldn't find a decent ECC-compatible mobo in a mini-ITX size at a decent price)

[GeorgeAb]

dBPoweramp to rip CDs. Not free, but reasonably priced and well worth it. I rip in FLAC and another copy in AAC for use in car or phone. Approximately 700 albums which as FLAC is .4TB. Working copy is on HD in Roon Server. I keep three copies on other harddrives. No cloud storage. If the house burns down, I will have bigger things to worry about.

[Yog Sothoth]

dBPoweramp to rip CDs. Not free, but reasonably priced and well worth it. I rip in FLAC and another copy in AAC for use in car or phone. Approximately 700 albums which as FLAC is .4TB. Working copy is on HD in Roon Server. I keep three copies on other harddrives. No cloud storage. If the house burns down, I will have bigger things to worry about.

This is exactly what I initially thought too.  But then I realized time marches on.  After a year or so, having rebuilt and put together a new stereo I would again want to listen to to music.  A lot of my music would be difficult to come by again.  So, I decided on an additional remote NUC with usb drives on it to server as a remote backup.  Cheap insurance.

[Yog Sothoth]

That's how I ended up with the current plan that I have.

Oh, and btw, the idea of using an enterprise-grade server?  Probably not.  ECC?  Check.  Dual PS?  check.  Multiple hot-swap HDD bays?  check.  Array of fans that sound like a jet engine?  Check - and deal killer.  The first time it powered up would be the last time it powered up, as I would be told to get it the hell out of the house...   Back to the other idea of an 'enthusiast' NAS case, mini-ITX mobo and silent fans.

It seems like a wise plan, and I don't think ZFS will disappoint.  When my storage needs grow large enough, I'll switch from a mirrored setup to raidz.  But it does take a long time to rebuild a raidz (or resilver a mirror). In my case, doubly so since my drive are all usb!

One thing I really wish zfs supported is erasure codes.  It's raidz implementation seems pretty solid, but for larger installations (beyond what I personally would need) erasure codes would be much more efficient.

[artur9]

I don't think I'll ever be comfortable storing tax and other sensitive docs in the cloud.

I work for a company that has a solution for that.  It's getting better and better.

HST, cloud storage definitely has its issues regarding security etc.

[artur9]

It seems like a wise plan, and I don't think ZFS will disappoint.  When my storage needs grow large enough, I'll switch from a mirrored setup to raidz.  But it does take a long time to rebuild a raidz (or resilver a mirror). In my case, doubly so since my drive are all usb!

I'm using a trash can Mac with 6 attached lightning drives.  Performance much better than USB.
It's running Debian with ZFS and each dual-drive enclosure is a mirror and then RAIDed together.  Working decently so far.

One thing I really wish zfs supported is erasure codes.  It's raidz implementation seems pretty solid, but for larger installations (beyond what I personally would need) erasure codes would be much more efficient.

The company-I-work-for's cloud solution uses erasure code tech.  I think MinIO also does, if that solution otherwise works for you.

[brj]

is there anything you'd recommend as a minimum to look for that would be unique to audiophile usage vs a typical data NAS?

Not unique to audiophile usage, but - an uninterruptible power supply with enough battery capacity to allow safe, automatic shutdown of all attached devices deserves a recommendation. Redundant power supplies and enterprise grade storage hardware won't help if the power is cut in the middle of a write.  Of course, that is another excellent reason to use a copy-on-write file system.

(My 2013 Synology NAS is still going strong, though I did recently replace my 15+ year old UPS with a more advanced unit.  The Synology uses EXT4 as I was unwilling to lean forward into BTRFS back then, but when I finally upgrade, I'll likely move over, as Synology manages BTRFS well.  That said, I will say that we evaluated multiple high-reliability file systems at work for a rather esoteric deployment, including commercial options like NitroFS that is used on other programs we support, and ultimately selected ZFS.  We ultimately couldn't justify the cost of NitroFS on our program and future expansion of the file system wasn't a factor, but regardless, we're quite happy with ZFS for our use case and I would definitely consider it elsewhere going forward.)

[Yog Sothoth]

I'm using a trash can Mac with 6 attached lightning drives.  Performance much better than USB.
It's running Debian with ZFS and each dual-drive enclosure is a mirror and then RAIDed together.  Working decently so far.

The company-I-work-for's cloud solution uses erasure code tech.  I think MinIO also does, if that solution otherwise works for you.

I believe bcachefs does also, although it is not yet in the mainline kernel.  It purports to support the same functionality as zfs and btrfs; I may give it a try at some point.

Your mac solution sounds very interesting!

[artur9]

Your mac solution sounds very interesting!

More details on my Mac solution.

The Mac Pro is a MD878LL/A.  No fans in particular, no overheating.
Described on everymac.com  https://everymac.com/systems/apple/mac_pro/specs/mac-pro-six-core-3.5-xeon-e5-gray-black-cylinder-late-2013-specs.html

I replaced the OS on the Mac with Debian 10 on the internal SSD.  Meant to dual partition but, whoops!, only linux on there now.

Challenge is getting Thunderbolt 2 drives.  I ended up with 6 of these dual enclosure ones from WD
https://www.anandtech.com/print/6216/western-digital-my-book-velociraptor-duo-review

I have a variety of drive sizes in them, ranging from 2TB to 10TB.  I would upgrade some of them but the bitcoin people are driving up the cost of the larger drives.

Under linux the enclosure's two drives show up as separate drives.  I use ZFS to mirror the drives in one enclosure then aggregate all the enclosures together.  Probably there's another better way than that but this was easy for me to understand as I was learning ZFS.

This Mac also runs minimserver to server out my music and JellyFin to serve out some videos.

I do use a laptop, usb-powered fan to keep the drives cooler.

Fun thing for a propeller-head like myself to put together.

[Yog Sothoth]

More details on my Mac solution.

The Mac Pro is a MD878LL/A.  No fans in particular, no overheating.
Described on everymac.com  https://everymac.com/systems/apple/mac_pro/specs/mac-pro-six-core-3.5-xeon-e5-gray-black-cylinder-late-2013-specs.html

I replaced the OS on the Mac with Debian 10 on the internal SSD.  Meant to dual partition but, whoops!, only linux on there now.

Challenge is getting Thunderbolt 2 drives.  I ended up with 6 of these dual enclosure ones from WD
https://www.anandtech.com/print/6216/western-digital-my-book-velociraptor-duo-review

I have a variety of drive sizes in them, ranging from 2TB to 10TB.  I would upgrade some of them but the bitcoin people are driving up the cost of the larger drives.

Under linux the enclosure's two drives show up as separate drives.  I use ZFS to mirror the drives in one enclosure then aggregate all the enclosures together.  Probably there's another better way than that but this was easy for me to understand as I was learning ZFS.

This Mac also runs minimserver to server out my music and JellyFin to serve out some videos.

I do use a laptop, usb-powered fan to keep the drives cooler.

Fun thing for a propeller-head like myself to put together.

Sounds very cool; I like the idea of repurposing a Mac.  I think their hardware is very good, but I've never wanted to buy into the all encompassing Apple ecosystem.

My setup is somewhat similar. A NUC acts as a fileserver, running Xubuntu and using ZFS on USB drives.  Not very fast, but the disk bandwith needs for playing music and videos is fairly  minimal.  Rebuilding the mirrored raid after a failure is another matter.  It takes at least a day.

For actually playing I have another NUC connected to my stereophonic HiFi.  The music library from the server is mounted via NFS onto this NUC.  I play video using vlc and music using Volumio.  I control it all from a small tablet.  I'm pretty happy with the setup, although firewire would be a welcome addition.

I also use the NILFS2 filesystem for development (which these days means converting LPs and other audio work).  NILFS2 is very cool.  Continuous snapshotting with every write.  I have that filesystem on another mirrored set running md.

And then I have placed a NUC at a friend's house which I rsync to nightly.

I'm very interested in experimenting with bcachefs as it implements the same types of things as ZFS and btrfs, supports erasure codes, and is a new from the ground up codebase (no doubt with all the associated errors of new code).  It's not in the mainlline kernel yet, but I'm happy to compile a new kernel, maybe test it on a spare laptop or in a vm.

It's shocking how much you can accomplish for audio and other tasks on these ultr-small form factor systems!