I'm 5'5.5" 125 lbs, 34B, sz 4, and wears Patagonia small, and the xs and small fits beautifully. The pix in red below shows the fit of xs on me. If just wearing very thin layers, the xs fits fantastic, fitted and tailored.

I wear an XS in most Patagonia things and an XS fits me fine.
Hope this helps.

I'm 5'5" and 120lbs as well and have been wearing this jacket for a year in the small size and love it. It's neither tight nor loose; very contoured and I can nicely layer up under it.