Will be interesting to here what folks suggest as I'm starting up w/ a mid 2010 mac mini as a server. Based on my experience so far I'd say:
1. Storage:
For me I'd look to get whatever solid state drive for the mac you need / can afford first, but don't buy the SSD from Apple. I've seen 500GB SSD drives for as little as $300. Will be quieter / faster than a regular HD. Some places I've read that you can do 2 HD's w/ Mac Mini's. So that's something to consider as well.
500gb is a lot of storage for music. I don't even use 200gb for almost all apple lossless. If you're looking for 3TB presumably you have a LOT of content, particularly hd video (?). If this is so, I'd still look at doing SSD internally + an external HD for additional media storage.
2. Memory: max it out for max performance - it's cheap - but don't buy from Apple. Self installing is easy.
3. Audio Out : don't use optical / toslink. Optical should be good, but I think good quality transmitter / receiver / circuits were never really done, so I've yet to hear a good optical connection. Use the USB (particularly if you have the VDAC II w/ asynch) and if not that, then get a USB to SPDIF RCA converter - like a HifaceII. USB is likely to be the best + cheapest option. You probably don't even need a very high end USB cable. Belkin or at best a Wireworld Ultraviolet or Starlight at best.
4. DVD Burner : probably doesn't make much of a difference. The one on my imac died after 7 years & I got a $40 USB Lacie. Does the job. If you want to play blu-ray's, you might have to pay up to $200 for a USB one.
5. Media player software : here I'd like to know people's opinion. I use stock itunes w/ apple lossless. Some people swear media players make a difference. I'd mostly only choose another if I wanted to do FLAC. Only improvement I've heard was w/ Jriver w/ autolevel defeated & forced to play purely in memory. That sounded substantially better than w/out those options.
P.s. I wouldn't want to run any media server w/ any flavor of Windows. Been burned to often w/ windows & don't need it / trust it. Even something running linux w/ a media server software that runs on linux has to be preferable.