That's not really what I mean by a "simulation", I am actually only referring to a situation that one might characterize as "a priori modeling of a certain system based on making input assumptions and applying all the known laws of physics that we think could possibly be relevant". Such a system is a "black box" that takes input parameters and spits out output parameters, and what happened in between is that (hopefully) a certain set of equations were correctly solved by a typically long and exhausting process of generating computer code with that intention. Such a process essentially always generates "bugs" (or at least oversights and shortcuts), but hopefully the endeavor reaches a point where none of the bugs introduce any significant errors. This is a description of pretty much everything that gets reported under the heading of "theoretical modeling" or "theoretical simulations" at an astronomy meeting.

No, it's not a simulation of some physical system with the intent of understanding that system, it's merely an application of known physical details to the process of data reduction. I would classify that endeavor as "observational" rather than "theoretical", though I'll admit such distinctions can be blurred and are sometimes more arbitrary than we imagine.

Yes, that's more what I have in mind.

But the problem is the "completeness" of the simulation invariable rests on whether or not it agrees with the observations, rather than on how much we have learned from it. This is my point. I have seen many examples of the types of simulations that say, in effect, "here's what we got that agrees with observations, and here's the part that shows some discrepancies. We're working of finding modifications to the physics that will bring the discrepancies into line as well." As if the work was needed in the area of the discrepancies! I would instead say the more immediate need for work is in the area of the agreement-- for there we actually have the potential to make theoretical progress in understanding what we are looking at.

But this is not what you usually find, generally the observers and theorists seem to simply crave some level of reassurance that theory can recover the observations, and simply substitute the cartoons when anyone asks them why. Then they move on to the next problem! I see a huge hole there, around the question, "what is really going on there, and how can we understand it in a better-than-cartoon way without simply referring to the full simulation?" To me, stopping short of that is like saying, "I don't know but my computer does, so that's good enough".
Exactly. I won't say I don't use google, nor that science cannot benefit from a similar approach to making predictions, but I do feel that it leaves a large gap in what science is capable of accomplishing.

Fair enough. But you can literally pick any theoretical investigation you like, the principle plays out almost everywhere. Instead of going over all the same ground three times, let me just pick one relatively common example and explore it in greater detail: mass-luminosity relationships in stars.

Certainly we have detailed stellar models that can generate mass-luminosity relationships expected in various types of stars. These models contain a vast array of physics, from nuclear physics to radiative transfer to gas dynamics and convection. They make predictions that observers can use for detailed comparisons. And I have seen many examples of observers comparing to the theory and worrying about systematic trends in the theory that are not seen in the observations. But when the theory agrees with the observations, they all go home, that's the end of it! They just say, "now we understand stellar interiors".

The problems with this are myriad. One is that it isn't them who understand, it is the computer. An even worse problem is the process tends to end when agreement is achieved, even though the agreement can certainly have serendipitous aspects that are not physically correct. Two recent developments in stellar interiors of massive stars are the inclusion of magnetic fields and differential rotation. One group recently found that including rotation gives better agreement with the observations-- so everyone is happy. But it turns out they also find that when they include the magnetic fields they expect to be there, the agreement gets worse! What if they just hadn't bothered? Is it that the magnetic fields aren't there, or was the excellent agreement they got from rotation just a complete coincidence?

A second problem is that we often encounter terms like "we included rotation", or "we included magnetic fields". I see it all the time:
observer: "But did you include magnetic fields?"
theorist: "Yes, they are in there too."
Now you may instantly recognize that this kind of thinking ignores the important fact that theorists always make a host of assumptions about how they will choose to "include" something, so it's not like they turned on a switch and God automatically entered the phenomenon in question into their code. How did they include it? Again there's not a need to write out a set of fundamental equations, there's a need to analyze the basic action of the field as it is playing out in that simulation, and what is it really doing to the results. If I don't see that, I basically don't believe anything.

A third problem is, even if the code has no important bugs, and even if the physics that is actually occuring is faithfully represented in the code, oftentimes the cartoon that is invented to explain it is either substantially incomplete or completely wrong. The mass-luminosity relationship of stars is a classic example of this-- it is amazing how many seemingly authoritative websites (again no examples needed, it's almost impossible to find a counterexample even if you try) will give bad or false explanations as to why massive main-sequence stars are so much more luminous than low-mass main-sequence stars.

What you will find, I assure you, are cartoons that include reasoning like "the higher mass raises the core pressure, which cause fusion to occur faster", or "the temperature in the core is raised by the strong gravity, causing fusion to occur faster". The faster fusion then results in higher luminosity, so they say-- the only trouble is, that cartoon is completely wrong! First of all, high mass stars are low pressure objects, not high pressure, and secondly, the temperature of the core is set by the luminosity, not the other way around!

The actual reason that high-mass main sequence stars, when you sit down and do the kind of analysis I'm saying is so often lacking, is that a high-mass star does not need to contract as much to reach core fusion temperatures. Thus you end up with a larger "leaky bucket of light", which is what a star is-- and a larger leaky bucket is a more luminous one. That's the reason, there is no need to mention any details about fusion because fusion does not control the luminosity of a main-sequence star (the luminosity controls the fusion), once you have an estimate of the core temperature of the star (fusion simply acts as a thermostat that maintains a relatively fixed core temperature, it is not necessary to know it accurately to understand the luminosity of a star).

If you find yourself having a skeptical reaction to that assertion, ask yourself two things:
1) How did Eddington understand stellar mass-luminosity relations long before anyone even knew there was such a thing as fusion, and
2) Does that reaction not prove my point that this analysis is widely missing, at least in this example?
But it usually does complete the interaction, that's what I'm saying. That interaction is the one that says to the observers "time to move on to new observations" and to the theorists "time to start working on new simulations, we're done here". The grant money is liable to dry up in both areas, as grant agencies begin to focus on newer "unsolved" problems.

Forgive me then, for I cannot discern from this description exactly what was accomplished by the Millennium Simulation, impressive label notwithstanding? I'm partly kidding-- I'm sure it will end up being useful, but what I'm saying is, I predict what will happen is that the physics will be tweaked and bugs will be corrected, until the agreement with observed numbers are achieved. Then an interesting thing will happen-- everyone will label the issue "solved", "paradox resolved", or some such thing, and will immediately move on to something else, after their various prizes and congratulations are accepted.

Maybe someone will also take the time to figure out what simple aspects of the simulation were actually at work in getting the number right, or maybe they won't, it will never be viewed as crucial-- it will just rely on "how good of a talk" the PI is capable of giving at the AAS meeting when it comes up. In the mean time, the "cartoon" will satisfy everyone, even if it's completely wrong (as in the case of mass-luminosity relationships).

__________________
Logic is the grammar of truth.

Meaning and absolute certainty are incompatible, and profound meaning and absolute certainty are profoundly incompatible.

The only thing intelligence is capable of is recognizing itself.