[JLM]

Start by educating your sense of hearing.  Music appreciation classes are a good place to begin.  Listen to unamplified music in places with good acoustics.  Get your hearing checked.  No point in spending gobs of money if you can't hear.

Set a budget.  Good systems that serve medium sized rooms can be had for $1000 retail, but won't go super loud or deep.  Avoid DIY that leads to proud papa bias which all audiophiles are guilty of anyway. 

Setup your listening space.  As I mentioned above the room shouldn't be squarish, small, or uninsulated.  Ideally it will be dedicated to listening.  My room is 8ft x 13ft x 21ft and well insulated.  It has a fiberglass exterior door with weather seal and flexible ductwork (can't hear the furnace).  It is treated with ten GIK 244 2ft x 4ft absorption panels plus three side wall tall bookcases (diffusors) that I can vary the "design" to suit.

Spend considerable time auditioning (months/years).  In many cases this would involve traveling, make a vacation out of it.  Bring 10 or more varied selections of what you love, including an audiophile guilty pleasure (poor recording) as the system needs to serve your desires.  Take written notes.  Only audition in a space that's similar to your room, not an open/noisy space.  Audition complete systems.  Insist on in-home trials. 

[JohnR]

It would probably be helpful to begin by defining what a purist audio system is.

I can think of two definitions which lead to completely opposite approaches:
1. Focus on the hardware. This is FRM's approach. No crossover, simple circuits (except for the later comment about Class D, which surprised me), no boxes.
2. Focus on the result. This means measurements and (in my opinion and experience) active+DSP. But it's a longer road. Still, if I were starting again, that's where I'd start.

[FullRangeMan]

The summer is too hot for a tube amp, I forgot to consider that.

[artur9]

Start by educating your sense of hearing.  ... Listen to unamplified music in places with good acoustics. ...

This, a thousand times this.  Otherwise, it's akin to making a space travel movie in the forties, when no one knew what the Earth looked like from orbit. 
https://www.thespacereview.com/article/3470/1

HST, electronica runs afoul of the "unamplified music" touchstone.

[MC275V]

As a longtime audiophile and music enthusiast, I feel that a “purist” system can be quite different for each person.  I also believe that your audio listening preferences can and will change over time.  If you listen to your system (or systems) often and enjoy it, you shouldn’t worry about it being purist.  I started my audio journey in the late 1960s with a Fisher tube receiver, ElectroVoice speakers and a Garrard record changer (which I only used manually for 33s).  By the mid 1970s I had moved to a solid state Sherwood receiver, Technics direct drive turntable and Sansui speakers.  By the early 1980s I had moved to Stereophile approved equipment, including HarmanKardon Citation power and preamp, Rega turntable, and Fried speakers.  By the 1990s I had B&K power amp, Musical Design tube preamp and Energy Pro22 Connoisseur speakers.  Today it’s McIntosh 275 amp, McIntosh C2200 pre, Clearaudio table, Harbeth speakers etc.  I’ve enjoyed each system throughout the years.  It’s been a fun experience for the last 50 plus years and I still listen every day. So, how to start a purist audio system; listen to live music, listen to as much equipment as you can, then purchase what you enjoy can afford.