I had to take my wheels off. I tried using trim removal tools to avoid scratching the caps and wheels but ended up ruining the edge of the trim tools. I think if you have a hot glue gun you can remove them like pulling a dent out: glue a stick to the cap that you can pull. Alternatively, if you don't care about scratching, a screw driver might have better l luck. If you're really impatient and don't care about damage, you could drill a hole through the cap and stick something in to yank the cap out.
Edit: I wasn't swapping caps, I was checking axle nut torque.