Muscle Slips First
You lose a little lean muscle every year after your 30s, and muscle is the tissue that burns energy while you sit still. What replaces it isn't nothing fat settles into the space the muscle left. Same body, same number on the scale, less of the tissue doing the burning. That's why the breakfast that was fine at 30 does more to you at 50, even though your effort looks identical. A diet doesn't put that muscle back. It just takes away the food.
Johnston, A.P.W. & Peterson, C.M., "Skeletal muscle aging and the mitochondrion," Trends in Endocrinology & Metabolism, 2013.