Where do you live? Whether you can use your own modem or not may differ. What the isp can or must do differs too.
I'll interpret "privacy at risk" as normal user privacy, with responses reasonable for normal citizens in a western/EU region (I can't confidently speak for others).
A modem is usually a "stupid" device or component. It is configured for the adequate transmission settings. It's not a concern.
The router is often rented and managed (and updated) by the isp. Replacing it with your own, a bought product not from the isp, and managing it yourself is a reasonable and relatively simple thing to do. I wouldn't call it necessary. It's the extra with extra effort. Installing your own open firmware is extra extra.
The simplest, most effective thing you can do for privacy is change the dns server of your devices. Instead of using your default routers isp provided one, use a privacy focused/mindful one. You can use one that does not resolve ad hostnames for additional significant benefit.
When you don't use the isp dns and use secure connections the isp already has no open protocol to snoop through. If they or another party at their endpoint wanted to snoop they can only use IP addresses which may vary in usefulness or attempt other more sophisticated tracking and analysis. A VPN would hide even the IP addressing - which is usually not necessary.