XS would be a more trim fit, and a Small would be a little looser, longer in the body, and maybe give you room to layer underneath. My wife is similar size to you, and she prefers the XS. hope that helps!

Unfortunately, you don't get much length through the torso going from an XS to a S in this item :( Just gets wider. Patagonia seems to be running increasingly wider these last couple years though their sizing is still all over the place :( If you are a 32B go with the XS; closer to a 34C go with the S. It is trim but not fitted. (The Mountain Hardwear take on this is VERY fitted and runs small!)