I'm pretty sure it's a connector up in the headliner as pictured below. Not sure how much slack there is to pull it out and disconnect. You might need to take that corner of the headliner down and over to the center. I'm pretty sure the map lights and dome light work off the same wiring.
Question - I can't figure out how to thread the connection through the swivel without removing the connector first. Is that the right way to do it? Also, it feels like the swivel pin is hanging up on something inside the visor when I try to pull it out. Any thoughts what it might be?