adobe illustrator - How do I move a pattern swatch once I've applied it to a shape?

I want to create patterns/images within an image in Illustrator E.g. smaller triangles within a large triangle.

I’ve tried to create a swatch with the smaller images and then chosen the larger image and selected draw inside tool but once this has taken effect you can’t move the swatch inside without it moving the larger image, could someone please advise the best way to achieve what I need to do?