Use the outfit as a reference, not a requirement for an exact match. A related colour or a restrained contrast can both work.
Choose one reference point
Start with the main fabric, an accessory or a small pattern detail. Matching all of them at once can create a crowded palette.
Compare in the same light
Bring the swatch and outfit together when possible. Do not assume that two photographs with the same colour name will match in person.