I'd give medium a shot. At your height, a small will be too short in the sleeves and a large will hang loose unless you're barrel chested.

I am 6'0" athletic build and the large is perfect for me.

I am 6'3" 175 pounds, broad shoulders, skinny waist and neck, long arms. The medium did not fit well, the length on the midsection was too short for me, tight in the shoulders, but the arms were plenty long. Please folks, suggest base layers for similar builds!