Let's say a site like the New York Times supports ActivityPub, so you can comment e.g. from Mastodon on an article but also from the article's page on nytimes.com, where all comments are displayed.
Is this article/nytimes.com part of the Fediverse? Yes and no. It should be but if we speak about the Fediverse we mostly mean social networks like Mastodon and Lemmy.
I think the reality is and will be slightly more complicated. I think we should think of the Fediverse as being a part of the web, be it a traditional website or a social network. That's why I propose a new term: the fungal web, based on how mushrooms act as a dezentral connector over which plants communicate.
For me the most annoying thing about the term Fediverse is that it is too general to be useful, not that it doesn't include enough.