There is a clear fundamental difference.

The "greatest scientific minds" are basing their speculations on known evidence, mathematics, principles etc.

Idle speculation that does not account for the math involved, that discounts the known or that does not have a solid foundation- can, indeed, be wrong.

In order to create a theory or model- you start with the known.

You do not speculate to the point where you are basing assumptions upon assumptions upon assumptions.

This enables us to come up with ideas- so we know what direction to look in.

If you lost your wallet- then you don't know where it is- What will you do?
You start with the known- You retrace your steps...

You don't go swimming in the ocean speculating about a pickpocket flying fish.

ETA: Being human, we tend to place almost emotional value on our theories. Having them thrown back as wrong can almost be 'hurtful.'
Take the moment to appreciate that people are helping to guide you toward where you need to be in order to speculate successfully, instead of feeling that your pride has been wounded