Having been there with my mac mini I can say that USB is an improvement vs toslink out, and that power supply upgrades also take things to a higher level. Which is not to say that a stock mac mini - even an older one - can't satisfy many if not most people.
TOSLINK : The main limitation of toslink among others is sub-par quality of the receivers & transmitters : toslink seems to always have been more directed towards convenience and low cost vs maximum performance (vs something like ST-Optical that was designed from the get-go to be a higher quality data link - albeit one which is very underutilized vs toslink). Again - not to say that toslink can't sound pretty decent - and sometimes better than poorly implemented coaxial digital or even USB when poor or improper cabling is used.
USB : This in my experience has ultimately been the most high fidelity data medium for PC based audio when done right. Which means asynchronous and isolated interface - whether via a convertor or with a DAC that does both well. There's just a lot of noise from the PC side which does and will interfere with your signal and which you don't want to pass on to your DAC. Toslink doesn't get rid of that either - it just converts the "garbage" it gets, turns it into a less than optimal optical digital signal on the other end, then does what it can less than perfectly to reconvert it into analog. At least the medium itself is not subject to RF / EMI like SPDIF coaxial or cheapo USB.
Power supply : It's been my experience that a lot of potential violence can be done to musical data by EM noisy PCs and with DACs. I found upgrading the cheap switching power supply on my DAC to a high quality linear one paid substantial dividends in a cleaner and more transparent signal and deeper darker backgrounds that allowed more of the music to come through. I've not yet experimented with the doing the same with my late 2010 mac mini, and don't know if doing so will make as substantial a difference, but I know the cost to do so is not as major as many component upgrades - and as it all starts at the source - it just seems to make sense to take things to the next level. I have tried using a C7 to IEC adaptor then using an upgraded power cord, and this did make an improvement in the sound quality, even when the power cord had already been hooked up to a line conditioner. My theory is that the power cable itself was also doing some line filtration vs a stock cable, among other potential benefits.
Which Mac Min : Would tend to agree that both (a) sometimes people think they need more than they do if they are just using it as music server, and (b) it is true that for the marginal difference it may make more sense to buy a current generation refurb or entry level new mac mini vs a used one. Best usually you're going to do on a usable mac mini is somewhere north of $250, whereas the previously mentioned are a few hundred more. In the audiophile world that cost difference is pretty minor - and there is something to be said to having some warrant & tech support if you do want to do something more with it.
