The youngest was Chinese skateboarder Zheng Haohao at 11 years and 11 months.
The oldest shown in this diagram is a little misleading. From what I can tell, he was a reserve member of the Australian Dressage team. The oldest athlete who actually competed was Spain's Juan Antonio Jimenez Cobo at 65. Also in dressage.
The oldest who didn't rely on a horse to do their work was Luxembourgish table tennis player Ni Xialian, at 61.