This study investigated brain bases of sexual and gender diversity. Heterosexual men and women, gay men, and transgender women underwent structural MRI. Regional volumes and cortical thickness were examined. Heterosexual sex differences were replicated. Men had greater gray and white matter volumes and women had thicker cortices. Overall, transgender women had a brain pattern that more closely reflected that of heterosexual women, whereas gay men were more similar to heterosexual men. Yet, transgender women also showed some volume and cortical thickness similarities to heterosexual men. These findings help identify brain regions related to gender identity.