Sorry if I'm repeating anyone here but...

To put it simply, the argument behind the Carter Catastrophy is a logical fallacy because someone has to draw the short straw.

I don't understand, sorry. Can you elaborate?

Grant Hutchison

If there's a future human civilization, it has to have a history. Some of the total number of people born in its history would have to be born relatively early during that history - they'd "draw the short straw", as it were.

I agree that it is a logical fallacy, because logic has to be 100% correct. But I think Grant is right that depending on how the "game" is set up, there are situations where the Carter catastrophe analysis is correct-- the majority will be correct that the end is relatively near. So the question is still, do we have any additional knowledge that allows us to infer we are not a part of the "majority"? I would say, certainly, there's no need to argue from a standpoint of zero information, which is what the Carter hypothesis does. If anything, there is plenty of empirical evidence that we could be near the end times even if human population had been 10 trillion in the past! (And of course the Carter hypothesis would indicate that any time you have a population of 10 trillion, and it winnows down to 5 billion at some point, those living would conclude that the end times could not be anywhere near. Would that have been a reasonable conclusion for the dinosaurs?)

This isn't true. The fact that you chose ball number 7 doesn't say anything about which urn is which.

It is true that before you choose a ball, you are more likely to pick 7 if you pick from the urn with 10 balls. Reasoning the other way doesn't work, though.

Look at it this way: before you chose a ball, you had to choose an urn. At that point, you had a 50% chance of choosing the urn with 10 balls. Then you pull out ball #7. That gives you no new information about which urn you have selected! Sure, it is unlikely a priori that you would choose #7 from the million urn, but the fact is that you did pull out number 7. If you'd pulled out numbers 11-1000000, you'd have new information, but you did not. Your odds of having chosen the ten ball urn are still only 50%.

Answers like this always depend on the "rules". I'm reminded of the great Monte Hall puzzle, in which you are shown three doors, one of which is a grand prize and the other two are goats. After you choose one door, Monte reveals another door has a goat behind it. Then he asks you if you want to change your choice to the remaining door. Is there a reason to do it?

Depending on the "rules" governing what Monte does, the chance of the new door being the grand prize will either be 1/2 or 2/3. It's pointless to argue about which is correct until you know Monte's rules. (And given the way Monte normally works, the rules indicate the answer will be 2/3).

It's a similar sort of problem, yes, but not the same. In the Monty Hall problem, Monty has extra information that he indirectly gives you by ALWAYS opening a door without the grand prize. In the choosing balls from a tub problem, no such decision is made -- it's purely random. You don't get any extra information from having chosen the number 7. Your odds of having chosen 7 were small, sure... but once you've picked it, you have pruned your decision tree:

prior to selection:
Root (1.0)
---ten ball urn (0.5)
------1 (0.5 * 0.1)
------2 (0.5 * 0.1)
------3 (0.5 * 0.1)
------etc
---million ball urn (0.5)
------1 (0.5 * 10^-6)
------2 (0.5 * 10^-6)
------3 (0.5 * 10^-6)
------etc

post-selection:
Root (1.0)
---ten ball urn (0.5)
------7 (1.0)
---million ball urn (0.5)
------7 (1.0)

Actually, it might help to think about it this way. The problem is formulated in a deceptive way. They could pick any number from 1-10 and still make their point. What they could not do is pick 11-1 000 000. That means that the selection was not truly random, and given a random choice of numbers which would seem to make their point, there are ten options from urn #1 and ten options from urn #2. Even odds for each possibility, no matter which number is picked.

Or, this way: there is an urn with one ball, and an urn with 9 balls. You reach in and pull out a number... any number. The chance that it is a 1 is 20%. However, given that it is going to be a 1, the chance that it is a 1 is 100%, and you can flip a coin to determine which urn you're going to pull it from: it's equally likely that each urn is the one with the lone ball.

If you still doubt, try it! Put nine pennies and a dime on a table. Turn all the pennies but one to heads, and the dime and the other penny to tails. Now, mix them about with your eyes shut and poke a coin at random. If it's a heads coin, discard the measurement (you are not interested in selections that are not the 1 ball!), and if it's tails, record whether it's a dime or a penny. Except for the difference in coin sizes, you should end up with about 50% of each.

The issue is, if you set up the experiment as described, and repeated it millions of times, you'll find that the vast majority of the times that a 7 was chosen, the selection came from the small urn, if you have a 50/50 chance of choosing either urn. Of course, if you have a much greater chance of choosing the bigger urn, say proportional to its size, then you can have the 7 come equally often from either urn. It's all a question of the rules.

That's not the same thing, though. Saying you perform the experiment and then check the value is not the same as fixing the value and then performing the experiment!

Exactly. So the question is, which set of rules, of the two you contrast, is more appropriately applied to the situation of birth order of an intelligent species? Here's a way to set up the rules where the Carter hypothesis seems at first to be correct, in the absence of any other information. Imagine one morning you wake up and are visited by an alien time traveller. The time traveller tells you that he has observed the extinction of a million intelligent species on a million worlds. He amuses himself by picking lives completely at random, and paying them a visit, and he chose you. And he mentions, by the way, that of course by these rules, 90% of the time he is talking to someone whose species does not outnumber that individual's birth order by more than a factor of 10. Why would you not conclude that your chances of being in that group are 90%, if you know nothing that distinguishes the species in question? You may have a hard time sleeping that night as you ponder the Carter catastrophe.

But you wake up feeling better, because it has occurred to you that he did not tell you the 90% figure applied to the subgroup of beings who were in around the 10 billionth in birth order! And indeed, there's no way to know that the 90% figure would apply to that subgroup without more information. (Bringing us back to Fram's original point.) So you ask yourself, what more information do I have? And you begin to wonder if the 90% overall average goes up or down among the 10 billionth beings from species similar to your own....

That's not the question, really. Well, perhaps it is. The answer is "the correct set of rules." In their argument, they make a false assumption about the probability of which urn is more likely...

...and then they use that false assumption. And that's the problem. You are reasoning with a false assumption, so you can't logically say anything about the correctness of your conclusion.

The truth in this case is that, reasoning in the same way that I do with urns (which I hope is correct), there is an equal chance of the urn containing any number of balls greater or equal to x, provided that I pulled x out of an urn. So the distribution is that there is 0 chance of the species going extinct before now (which makes sense!) because the ball did not come from an urn with fewer than x balls, and equal (and, as it turns out, infinitely small, if there is no upper bound to time/population) chances for any two times in the future, represented by any two urns with more than x balls.

Then, of course, even that reasoning is overly simplistic, because it doesn't apply any weighting to the urns based on social factors, when our star is due to blow up, science, etc. But it's the correct logical conclusion to an unbiased version of the urn example.

You have used a different set of rules to derive the "snarkophilus anti-catastrophe", namely, that humanity will never become extinct, because each total number of humans greater than our present birth number is equally likely, and there are a virtually infinite number of possibilities! Interesting. This seems as justifiable as the Carter catastrophe if yours are in fact the right rules, but really neither set of rules are very plausible, just as the "everything has a 50/50 chance, it either happens or it doesn't" is also not a very useful rule. What do you say to the alien time traveller argument, in relation to your picture of the right rules?

I'm not entirely sure what you mean by saying that there are different sets of rules. That idea came up in response to

The question posed by Carter is, "given that you fix the result of the experiment beforehand, then perform the experiment and discover that the answer is 7, what can you say about which urn was selected?"

My answer is "nothing," because the experiment is fixed. The probabilities of having chosen each urn are the same as they were before the experiment. (I arbitrarily decided that there was a 50/50 chance of picking a particular urn. It doesn't have to be that way. That's partly why you get the "anti-catastrophe" scenario. Choosing a different distribution of urns gives a different set of probabilities.)

I suppose that another set of rules is not fixing the experiment, but performing it and seeing what happens. I've already explained what conclusion you can draw from that experiment, what you call the anti-catastrophe. And, as I mentioned, that conclusion isn't really valid in real life because there's no reason to think that each urn carries equal weight. But it's still better than the other version, because it is based upon sound logic.

To resolve the anti-catastrophe, do the following: pick a number. An integer. Any positive integer. Now, amongst the integers, the probability that you chose that number is essentially 0. (There is a discussion about surreal numbers kicking around here somewhere that is almost pertinent...) The fact is, however, that you chose an integer. That integer is finite, even though there are an infinite number of possible choices. And so it is with the anti-catastrophe. Even though you have an infinite number of choices of urns, in the end you have to pick one, and it has a finite number of balls in it.

There's still the possibility that the anti-catastrophe occurs, of course, but you can't decide how probable that is compared to an integral count (catastrophe after n people) with additional information.

*pause for breath*

And that leads in to the alien time traveller.

He has chosen a life at random from the species, so you have a 90% chance of being one of the last 90% of people to be alive. That makes perfect sense, and doesn't worry me in the slightest. That's because you have to add up all the possibilities. If there are only ever going to be 10 billion people, and I'm number 10 billion, that's bad. Same for if there are only going to be 11 billion. But what about if there are going to be a trillion? Two trillion? There are a lot of numbers up there, and in each of those scenarios, you're in the first 10%. It's increasingly improbable that he would choose you in particular as species number increases, but the fact is that he had to choose someone. In fact, if there's truly an equal chance of your species number being any number, your birth number is statistically so small (compared to infinity) that there's no point in worrying: you're most likely at 0%.

You need a realistic distribution as to the probabilities of each species number being correct to get any information out of your extraterrestrial visitor.

And, of course, there's the possibility that you'll never be extinct. He's witnessed a lot of extinctions, sure. But he didn't say how many species go on forever. You have no data whatsoever about that (except the laws of thermodynamics, but I digress).

This is what I am saying as well. The 90% number applies over the full sample, not necessarily the subsample of comparable beings (i.e., a subsample of species in comparable situations where the time traveller happened to choose the ten billionth being, or so). There might be a correlation between overall longevity and making it to ten billion at all, and there might be elements in place that make a particular species more or less vulnerable to extinction by the time they get to the 10 billionth being (which are we, do you suppose?). Thus the "rules" you need to make the Carter hypothesis work categorically is that the total number of beings before extinction must be a random variable that receives no inputs from anything about the species, and no correlations with making it to 10 billion in the first place (beyond the obvious). The rules to make the snarkophilus anti-catastrophe I'm not sure I can think of, but there probably are some that could be applied in the context of this thought experiment. But I would argue that in any event, neither of those rules are plausible. As soon as you look at anything that relates to extinction potential, the odds change dramatically from the Carter "default" argument. The real question is, which way?

It also strikes me that the Carter hypothesis assumes that there must be a finite number of humans born a priori. In the version we're talking about here, it also assumes that there are only two options, the small urn and the big one. Even if we limited ourselves to a finite number of humans, why just these two choices? What happens to the odds if you have a billion urns, with a number of balls ranging from one to a billion? What if you have a trillion urns? Sure, by Bayesian reasoning, drawing number 7 means that the probability you drew it from the urn with 10 balls is greater than the probability you drew it from the urn with a million balls. But the overall probability that it was from a small urn goes steadily down as you increase the number of urns. And how do you know when to stop adding urns, unless you make an arbitrary decision ahead of time?

That speaks to limitations in the urn analogy, which is why I prefer the alien time traveller approach. Since the alien chose you at random, it would not seem to alter your situation vis a vis the validity of the Carter catastrophe.

Ah, OK. This doesn't work, because (as Ken says) Carter's argument is probabilistic: it states that there's a 95% chance of extinction after x number of lives. Such a prediction accepts, indeed stipulates, that it will be wrong on 5% of occasions (ie, for those born early in human history).

Unfortunately, this "illustration" seems to have been written by someone who doesn't understand Carter, or doesn't understand how to construct a good illustrative analogy: it's utterly misleading about how Carter's argument works.
There's no choice of urns. There's a single urn, which contains an unspecified quantity of consecutively numbered balls - anything from 10 to a million. We blindly select a ball, and find it's number 7. This is more likely to occur if the urn contains a small number of balls than a large number of balls. In fact, we can deduce a 95% confidence interval for how many balls the urn contains. Implicit in that calculation is the prediction that 5% of the time the urn will actually contain more balls than the upper limit of our confidence interval (assuming I've constructed a one-tailed confidence interval).

Grant Hutchison

Yes. This was my point, early in the thread, when I pointed out that the possibility of infinite human lives blows Carter out of the water. Whether we draw the 7th ball or the googleplexth ball, we're equally unsurprised to find ourselves close to the start of human existence.
Freeman Dyson certainly did some work in which he suggested that the number of "processor cycles" possible in a finite, expanding, cooling Universe approached infinity as time went to infinity. This would suggest that there would be room for an infinite number of human consciousnesses (implemented in some suitable form left as an exercise for the student) in the future Universe.
A Big Rip scenario might well undermine his calculations, however: I don't know.

Grant Hutchison

Yes, I think that everything Grant has been arguing has been correct, except the basic assumption that every number of balls in the urn must be equally likely. Imagine instead that the urns are stuffed using an algorithm that makes it 1% probable that there are 10 balls, and 99% probably that there are 1000 balls. If you pick a 7, what do you conclude is now the probability that the urn contains 1000 balls? The relative probability for 10 balls is .01 times .1, or .001, compared to the relative probability for 1000 balls, which would be .99 times .001, or about .001. This is the same relative probability-- choosing a 7 in this case gives you a 50/50 chance the urn was the 1000 ball or the 10 ball variety! The key point is that to make a meaningful probability argument, you need to know something about how the probabilities are distributed. Probability is about information, it's not something absolute (except in quantum mechanics), and the assumptions you make about the things you don't know (the "rules") are everything.

If this is still unclear, realize that the Carter argument is made from a position of no information, outside of birth order. It is like a person who has just learned how to play chess, entering a chess tournament. This person has no knowledge of how chess tournaments work, other than that there will be 1000 competitors. They don't even know the level of anyone else in the tournament. But they figure, hey, out of 1000, I'll probably end up no worse than 900th place, with 90% confidence. But that's only true if all 1000 competitors also just learned, and are a random cross section from the same population. If instead, this tournament happens to be the World Championships, then it might be quite likely that the newcomer will place dead last, despite their probabilistic argument based on no information. When making a probability argument, the meaningfulness of the result depends on the reliability of the assumptions. Thus the Carter argument is correct as far as it goes, but is of very limited meaning, like the chess entrant's conclusion that he/she will not finish worse than 900th place.

The BIG Problem Is, How Do you Determine the Confidence Interval?

Without ANY Starting Information, It'd Be Impossible!!!!

__________________
If you Ignore YOUR Rights, they Will go away.

Right. It's like the problems with the "Drake equation"-- what do these probabilities really mean when the numbers change every time a scientist sneezes?

A ball chosen at random has only a 5% chance of coming from the lowest-numbered 5% of balls, and a 95% chance of coming from the other, higher-numbered balls. We are therefore 95% certain that our ball, number 7, has a higher number than the lowest 5% of balls. So we are 95% certain that the lowest-numbered 5% consists of fewer than 7 balls. If there are only 6 or fewer balls in a 5% sample, then the total number of balls must be (6*20)=120 or fewer. When we draw ball number 7, we are therefore immediately 95% confident that there are 120 or fewer balls in the urn.

Grant Hutchison

Well, I don't think I've ever argued in favour of that - Carter just assumes it. To make my position clear, here, I've simply been trying to explain why it's rather harder to make Carter go away than some folk imagine - he's been vexatiously discussed for 20 years, after all, and that's not a common characteristic for an easily-dismissed logical error.

So you're saying that information external to Carter's assumption may constrain the lifetime of humanity in some way. By analogy, we might imagine a community of human liver cells trying to use Carter's reasoning to predict their community's future existence. Cells in a 7-year-old child would predict a relatively short future, while cells in a centenarian would predict a long future. Oops: the information they lack is that there is a characteristic lifespan for a human.
However, to use this argument to properly blow Carter away, you'd have to tell us which specific bit of information you have that gives you a more reliable estimate of humanity's future existence than Carter does.

Grant Hutchison

Okay, I'll put things very simply and bluntly:

From a point in the history of any civilization, you can't predict with any accuracy how long that civilization will last. Period. For all we know, we could all be wiped out by a NEO in a few decades... Or we could be at the beginning of a civilization that survives billions of years into the future, and colonizes the whole galaxy. We don't know, and furthermore cannot find out until something actually happens. So y'all quit yer catastrophizing and do something useful, and we might actually have a chance of surviving whatever comes up!

Ah, the Argument from Repeated Protestations of Increasing Loudness. One of my favourites.

Sorry. Couldn't resist. Just teasing.

Seriously: Carter's argument would say that your NEO is just the sort of catastrophe needed to explain our apparent location near the End of Days. And that the billion-year civilization is of course possible ... just unlikely, given how close to the start of it all we find ourselves.

Grant Hutchison

No, because you don't know the size of the single urn. In order to properly predict which urn you have (which is the point of the whole exercise) you need to consider all possibilities for urns.

In the case of 7, there's a 1/7 chance of picking the 7 ball if n=7, 1/8 if n=8, 1/9 if n=9, etc. Add up all those possibilities, as n tends to infinity. It's a divergent series, which means your 1/7 chance is singularly unlikely: you will never be the last person in your species. The only ways to resolve this are to change the series by weighting the odds to get a convergent series or to accept that the species will never die.

But why are you adding up these probabilities? What does this sum measure? By summing probabilities, you're producing a probability greater than one - what does that mean?

Grant Hutchison

So a NEO or other global disasters would be more unlikely if fewer people had lived? Personally Im more confident about humanitys chances of a continued long existence with six billion individuals than if there were only, say, a hundred thousand. The more people, the more likely someone will pull through a major disaster and be able to continue reproducing.

Hmm, I don’t get it. Maybe I’m just too stupid to understand the puzzle, but it seems to me that based on the basic premise of the story, every generation would be just as much “near the end”, if we think that “this” generation is “near the end.” Or to put it another way, every generation would have just as much a chance of being “near the end” as any other.

Why is there any reason to think that we are more “near the end” than Europeans in the middle-ages who were dying by the millions of the plague, or Africans 50,000 years ago who lost whole tribes during gigantic floods, earthquakes, tsunamis, and massive volcanic eruptions?

Seems to me that humans would have been closer to becoming extinct half a million years ago, when there were so few of them and they had no medical protection from diseases. It wouldn’t have taken much to wipe them all out, just as thousands of other species were wiped out in the past. As a matter of fact, maybe a really superior species of being -- above humans -- did exist on earth for a while, but did die out because of some natural disaster, and we are merely the lowly intellectually-inferior survivors.

The problem ultimately reduces to this: given a chosen ball X, find the probability of each urn with n balls being the one that represents the universe. If you had a limited number of urns, you'd add up the probabilities and multiply by a normalization factor to account for the fact that at least one urn must be chosen. Even with an infinite number of urns, if you have a finite sum, you can still normalize with a finite factor. However, because the series is divergent as the upper bound of n tends to infinity, the probability of each urn being chosen is essentially zero.

Carter claims that a 10 ball urn is more likely than a million ball urn. That may be true, but it is not more likely than an urn with between a million and fifty million balls. And so it goes. You can always find a range of urns that is more likely than the last. And that's where his error lies.