Deer I would go no less than 40 and use good broad heads and shot placement. Elk I would go no less than 55. I consider those the bottom line numbers and prefer deer up over 55 and 70 for elk. But one deer was successfully taken with my wife's first bow on 38 pounds.
Really depends on bow. Some shoot faster for with less weight. With that said my father shoots a Mathews q2 Older bow. He pulls 52lbs because of his shoulders, he has killed everything in North America and most things in Africa. I would say as long as you can shoot well I would feel comfortable down to 40. I say this cause I've killed many large Michigan whitetail growing up with that. I would recommend a good cut on contact Broadhead such as montech. Those seem to cut very easy compared to a muzzy which I shoot for the bone smashing tip. Lol. hope this helps