Yes, you are wise when you suggest to invert your cost proportion between DAC/amp and speakers. Better speakers give you more bang for the buck and less diminishing returns than do better electronics.
For my TEAC/KEF combo, which is working well on my desk, see this thread: