No reason to leave it at home
I'm 6'3", 180lbs and I ordered the large. They fit nicely over my R1 pants and hiking pants. I love the pants. I did a winter overnighter in northern Japan and loved having these on when the sun went down. They're lightweight, pack down to the size of a nalgene big mouth, and are super cozy. If you're not sure whether you'll need it, bring it anyway. Along with a good top layer, you could forego the sleeping bag should you need to bivy.