Math is fundamental in many aspects. Math essentially leads to physics, not the other way around.

Look at General Relativity. Einstein spent many years with mathematicians before he could even write out GR. Before that came Special Relativity, based entirely on 2 simple ideas and math. Even SR wouldn't have been possible if it weren't for Maxwell, an excellent mathematician with a keen sense of discovery.

Have you done calculus to the point of 2nd order and higher calculus as well as partial derivatives? All those years of working on math leads to a simple understanding of many things.