I am in the middle of my NX Studio build and I too have a small well treated room at 10ft X 12ft with a vaulted ceiling.
I will be going with one of these sub options and I am seeking guidance as well.
I can get the H frame out 3 feet in from the front wall but that is about the limit.
So, which is best?
It depends. The problem with the open baffle subs in a small room is getting the baffle at least 3ft into the room. If you can't do that, the sealed sub is the way to go.
If you can get the baffle at least 3ft into the room, you can use either type of sub. Which one you choose comes down to what you want from the sub. Both are quick, clean and musical so that won't be the deciding factor. The biggest difference comes down to how the two types of subs load the room and the presentation the different types of loading give.
If you want heavy gut punching, chest thumping, window rattling bass down to 10Hz, the sealed sub is the choice. However, if you are OK with a less forceful bass presentation and ultimate speed , finesse, tone, timber, spaciousness, smoothness, and tight bass that is still very impactful, the OB subs are the way to go as long as you keep in mind the caveat that the baffle has to be at least 3ft off the front wall.
I am not saying the OB subs are not impactful, they just have a different type of impact.