Creativity assessment can be influenced by social group membership, including gender. This study analyzed implicit biases in the Consensual Assessment Technique (CAT) by examining gender differences in creativity assessment. We asked male (n = 26) and female (n = 39) judges to rate the creativity of fashion outfits presented on Instagram and then examined gender differences in mean creativity ratings and rater consistency (inter-rater reliability). We found no systematic support for gender differences in the level of creativity ratings, but observed that rating consistency was significantly higher for female than for male judges. In an additional qualitative analysis of the implicit criteria that raters applied when assessing creativity, we found that female and male raters attached different relative importance to various assessment criteria. Our study suggests that rater panel composition can indeed affect aspects of creativity assessment, although we do not obtain strong support for a gender-related bias in the CAT methodology.