It isn’t a single “thing” you are some distance away from. It’s photons remaining from the early universe that can be found everywhere without direction. Pick “one” of them and you can track your speed relative to it. It’s the closest thing we have to a universal reference frame.

Also see the later questions on https://www.astro.ubc.ca/people/scott/faq_basic.html

Edit: I’m stupid, photons move at light speed of course. But you can detect a colder and hotter side of the CMB and use that as a reference frame.