Consider where you look when you're talking to someone in person. Do you look at them continuously? We generally don't do that, it ends up being weird and creepy. Our eyes wander as we talk, so it's natural that we'd do the same during a video call.

And consider this: when your own face is visible to you, when aren't you looking at yourself? Typically when I see my face, it's in a bathroom mirror, and it's natural to look at yourself (whether to take note of your appearance or otherwise).

So I think it's pretty natural, and I'm guessing lots of people do it.