Yes streamers impact sound quality. The streaming file may have data-related timing issues (jitter) or digital signal integrity problems, I have read there is no error correction between a streamer and a DAC.
The Mac Mini was well regarded as a streamer. Search for "Mac Mini" in the GR Research Circle. Eric Hider was offering mods to make it even better.
I built a streamer/music server because I needed more computing horsepower than most commercially available streamers and I like building things (and I have building computers since 1997). My streamer has 3 separate linear power supplies to keep noise at a minimum and a JCAT Audio Grade USB card to keep jitter at a minimum. I upsample all my files to DSD256 which is why I need the Intel i7 processor but I can also upsample PCM (or not) and use different filters to compare PCM to DSD processing and generally play around with sound.
https://www.audiocircle.com/index.php?topic=177495.msg1867916#msg1867916
I think your PS Audio DAC upsamples everything to DSD128 whereas the newest DirectStream upsamples to DSD256.
In contrast, o2b's Mac Mini uses an incredibly noisy SMPS for power and has really shitty clocking on its USB outputs. Eric Hider's mods were software-based if I recall - creating a really stripped down operating system that eliminated extra processing noise internally. But I don't think he address the noisy power supply, nor the basic hardware limitations of the USB output of the Mac Mini. Other companies like Uptone Audio offer replacement power supplies that are way way quieter, but these mods cost some significant outlays, and still don't address the USB output compromises.
Eric Hider was offering these mods back in 2010 roughly. The streamer world has moved light years forward since then. At this point, I would highly recommend an Eversolo A6 Master Level with an aftermarket linear power supply, which you should be able to get for around $1k or a little more. Or the Eversolo A8, which you can probably get used for $1500-1600 and does not require an aftermarket linear power supply (it has one built in). These will have low noise power supplies, upgraded clocking on their outputs, and an incredibly versatile software package that makes streaming rather easy and flexible. It's way, way more user-friendly than setting up an Mac Mini and will open you up to the world of controlling your streamer with a great app on your phone instead of having to try and bootstrap a solution with your Mac Mini for remote control.
I have the Eversolo A6 Master with a Teddy Pardo power supply, and I've found the coax output to be better than the USB for my purposes - feeding a tube DAC to get some warm and seductive feels. I used to have a stock Mac Mini with some software package, then moved to a first gen Auralic Aries, then to a Sonore Ultrarendu with an excellent power supply, and then about a year ago to the Eversolo. The Eversolo is not quite in the league of the Ultrarendu in the sound quality department, but it's leagues better in the software department. Eversolo is probably better than the Auralic Aries was too. The Aries was a pretty sizeable jump up in sound quality from my old Mac Mini, so the Eversolo is an even bigger jump, and the Ultrarendu (with a good enough power supply and investments in network power and filtration) is better yet. For the money, Eversolo takes the prize for me, hands down.
Someone else in this thread mentioned that you might go with a Holo Audio Red instead of an Eversolo if all you want is a streamer (you don't want or need to try out the Eversolo's built in DAC). I don't disagree in principle, but I haven't heard the Holo Red to compare on sound quality (thought I read somewhere that it isn't quite up to the Eversolo level when using a linear power supply for the Eversolo, but not sure that's reliable). However, the Eversolo gives you maximal versatility for not that much more money. If you ever have problems with your PS Audio DAC and need to do without it for a while, you can use the Eversolo instead. Or you might find that the Eversolo DAC is good enough for you (a lot of people think it's rather good), and offload your PS Audio. With the Eversolo, you have the ability to try it out in various configurations and have flexibility for the future.
I also owned the Spatial Audio M1's, which were the immediate predecessor to the M3's, so I have a good sense of what they are capable of. IMO there is a reasonable chance that the Eversolo's DAC could have better synergy with your M3's than the PS Audio DAC, which is known as being a bit warmer and smoother in character, whereas the Eversolo DAC is crystal clear and airy, if a bit incisive. Different flavors. I'm sure the PS Audio is more refined on the whole, but maybe not by THAT much, and overall system synergy can be really important.
Ok, I've rambled enough.