adobe illustrator - Figure-ground double exposure on negative

I have a logo of a man's face with the double exposure effect, it will be printed on a t-shirt. I applied the effect in illustrator using clipping mask. I used a night illustration. The effect works on white background as in the left part of the image, but when I change the background to black the man's face doesn't appear anymore as in the right part of the image

Large image on white background

Large image on black background

My question is what can I do to make the logo successful on any t-shirt color?