If you want to optimize your speed, use the Advanced Page Cache and use the asset pipeline to combine and minify all of your CSS into a single file. Even if you add a few hundred CSS rules, the load difference will be negligible.
If you still want to load those CSS rules just on specific pages, then you can include them inside tags in your page markdown. This way you don’t generate an additional file request by either having a new or additional CSS file for that page.