Well, that’s because every single other body part has a bone in it. The penis does not have a bone, so growth would be more related to their endocrine systems, nutrition and individual genetics.
The height difference is not statistically relevant here
Noses and ears also have an underlying cartridge scaffold, something also not present in the pee pee structure.
Those structures also don’t become engorged with blood to increase their firmness and size.