I guess "progressive rock" is the best label. I'd also loosely classify them as heavy metal as well. Since Rush is the great granddaddy of all prog rock bands, you may not have ever listened to that genre, but Magnitude 9, Dream Theater, and Symphony X are other bands in a similar vein.
Some of Rushes biggest hits are "Tom Sawyer", "Time Stand Still" and "Big Money."
For the new Rush listener, I'd recommend Chronicals as a good place to begin, or The Spirit of Radio for a little cheaper, single disc.