Generally trial and error until I get the feel I want. Set up a build one way, try it out. If I find myself running out of of ammo, not doing enough damage, etc, I'll try to remember that and adjust for next time. Although by this point I've played so much all of my guns are pretty locked in and I think I would get confused if I changed OCs now.
I did greatly enjoy the new set of OCs they released. Using most of those now.
You would get the intended effect, but layering multiple coats of varnish risks making the whole thing a little cloudy and obscuring details. If you really wanted to do this I would hand paint both matt and glossy areas. But I also think that messing with sheen doesn't look very nice. I like to matt varnish everything.