OK, I'll look at that.

Grant Hutchison

Back from that thread, with a quote lifted from there:
I disagree. I successfully used both bits of information in your envelopes puzzle to come up with a betting strategy which is successful over repeated trials. The fair price is struck if the buyer offers a price equal to the value revealed in the first envelope, whatever that price may be.
Carter uses his bits of information to come up with a probabilistic betting strategy which is successful over repeated trials (5% are wrong, 95% are right). This success is attained if the "bettor" bets on a total population equal to 20 times his birth number, whatever that birth number may be.
Both approaches go around any concern about the shape of the distribution. (And of course both approaches can also immediately be improved if we are provided with additional information about the distribution.)
By singling out an inappropriate subset of cases which share a specific value (of revealed content, of birth number) I think you're actually creating an apparent problem that doesn't exist if the whole picture is observed.

Grant Hutchison

Actually, your betting strategy never used the $10 number in determining your formula. What you really did was use logic to show that a betting strategy that uses the first number (whatever it is, as you say) to buy the second envelope will work out in the long run. There's no $10 in that, except that your formula tells you that when you encounter the $10 subset, you should pay $10. But as soon as it does so, you no longer have any expectation of breaking even in that particular subset of trials! This last remark is crucial.

I see the issue. The key point is, you do not expect to have a breakeven strategy in any given subset, such as, 10 billion births!

Here's a better way to see it. Imagine that every intelligent being that has ever lived meets in the restaurant at the end of the universe. Someone says, "let's have all the people who were among the first 5% born to one corner of the room." Of course there's going to 1/20 of the attendees in the corner. Now they say "OK, can I have every being that was the 10 billionth born to raise their hand". Why would you expect 1/20 of those people to be in the corner? Generally, that would not be the case! The reason is, if you imagine that total populations are generated by some probability algorithm, then that algorithm must have some scale, a "median" birth number. The number 10 billion has some unknown relation to that scale, which generates the unknown correlations, just as the number $10 has some relationship to the envelope-stuffers scale! We used to think that because the relationship was unknown, Carter was an OK probabilistic argument. But what I'm now saying is, just because you don't know the correlation does not make the argument meaningful. You still don't pay $12.50 for the other envelope, and you still don't expect there to be 1/20 of the 10-billiionth-borns in the corner of the room! So Carter is just plain wrong, since it is an argument that completely involves that bunch of 10-billionth-borns. This is the subset I've been referrering to-- it's us!

But only if you believe that this subset of trials has any relevance to the real world. I say it hasn't.

I'm very happy with that. I can't see why there should be any correlation between these two groups. If most populations are very long-lived, then a lot of those with the 10-billionth birth number are going to be in the corner. Carter doesn't care, so long as 5% of each population was wrong in its assertion that it was in the last 95% of all births.
I think you're chasing some sort of symmetry that doesn't have to be there for Carter to work.

Grant Hutchison

I think you are missing the key point that Carter says more than just that we have a 5% chance of being in the first 5%, it also tries to use the fact that we are the 10 billionth born to say that this says something about our total populations. I'm saying that you cannot use both of those numbers to generate a third one, like an "expected longevity" of humanity. That is exactly the same mistake as using $10 to get an expected value for the other envelope, by any method. You're saying you don't care about the 10 billionth-born subset, all you care about is the 1/20 of the total attendees who are in the corner. But if that were true, you would have to be able to state Carter's claims without applying it to us, because we're in that subset of 10 billion borns, and there are not 1/20 of us in the corner. Carter's error is in the number 200 billion, not in the number 5%. We are 5% likely to be in the first 5% until you use the information that we are 10 billionth born. You can't use both those numbers and apply it to us.

No, I understand that.

And I'm saying we can, and that the absence of correlation between the "10-billionth group" and the "first 5% group" is entirely to be expected.

And that's fine, because we don't care about where we're situated among the superset of all populations everywhere, only about where we're situated in our own population. And we accept the possibility we might be wrong in our prediction.

Grant Hutchison

I see that your approach is to restrict to the human in the "restaurant". But here's the problem with doing that. There's only one 10 billionth born, let's say it's you. You're either in the corner, or you're not. But whether you are or not, on what are you going to base your conclusion that there was a 5% that you would be? What criterion can you use other than looking at the other 10 billionth borns in their own similar intelligent species? Let me go through the steps of Carter's argument. First all, we assert that you and I have only a 5% chance of being in the first 5% of humans. We've talked about potential difficulties with that, but those are irrelevant compared to this more central issue, so let's assume that this much is true. OK, now we do some research and find we are the 10 billionth born. The last step is to combine these tidbits as if the second did not alter the first, and conclude that there is only a 5% chance that humans will outnumber 200 billion. That is the wrong step. A probability is a fraction out of many trials that are equally likely. It is fine if those trials are idealized, they don't ever have to actually take place. But what is the set of trials that you are using to say that there is a 5% chance humanity will outnumber 200 billion, if it is not the trials represented by all the species in the "restaurant"? I can't see any basis for using a 5% probability in concert with a 200 billion total population. Can you explain any way whereby you have a set of (idealized) populations distributed such that 5% are more than 200 billlion and 95% are less? And if you can't, what does a 5% probability mean?

The fact that I am declaring with 95% confidence that I'm in the latter 95% of all humans who'll ever live: that requires that there's a 5% chance I'm in the corner.
OK, that's fine, but that's bringing in a bit of external evidence. I suggested this was what you were doing when you first introduced this argument, and you denied it at that time.
I am not performing these trials. The 200 billion result comes only from the 10-billionth human. Other humans perform the same calculation and produce different results. 5% of those various different human results will be wrong, not 5% of every 10-billionth soul who makes the 200 billion calculation. I've no idea how many of those will be wrong.

If we go back to your restaurant, with the first 5% of every species cluster in one corner: the 10-billionth soul of some species will be in the corner, the 10-billionth soul of other species will still be in the general room. All the folk in the corner will be saying "Oops, we got our calculations wrong - that's 95% confidence for you." All the folk in the general room will be saying "Yep, we were right to assert we were in the last 95%." So some 10-billionth souls will have been right, and some will have been wrong. We can't say what proportion without introducing external information. It doesn't make the calculation invalid at the 95% level for any given population.

Grant Hutchison

There's the rub. I feel Carter's argument is information that does not inform. What can we do with it? How should we behave? What should we do different? What can we do with it? What does it mean?

__________________
0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 ...

I dunno.

Grant Hutchison

What external information? All the species are identical. Think of them as all the possible ways humanity could have gone, if a butterfly had flapped its wings. How is this not exactly the issue involved in a confidence interval? You are clinging to the 95% confidence without recognizing that being the 10 billionth changes that confidence irreparably, even if you don't know how it changes it. You may only maintain that confidence if you never use the information that you are the ten billionth. That is the extra information.
But that's not how confidence intervals work. Where is your idealized set, all who conclude their species won't outlive 200 million, and only 5% are wrong? That is the set you have not produced.

Well, you originally suggested we look at:That certainly seemed like you were looking for external information about the likely lifespan of intelligent races.
"Clinging"? Nice choice of word.
But 95% confidence here doesn't require all those other imaginary species in their crowded restaurant - just us humans.
Let's say everyone who ever lived announced at some time in their lives: "I'm one of the last 95% of people who'll ever lived." 95% of them would be right. If they were statistically astute, they'd have better phrased it "I'm 95% confident I'm one of the last 95% of people who'll ever lived." The 10-billionth person might be one of those in the top 95% or one of those in the bottom 5%: it doesn't matter, his/her statement is still valid.


There is no such set. The 200 million figure is the product of a single person, the 10-billionth, who may be right (with 95% probability) or may be wrong (with 5% probability). His rightness or wrongness is not a function of the figure 200 million, but of his/her actual position in the total number of humans, something which (s)he cannot know, but can certainly estimate with 95% confidence.
In some of your collection of species, the person who predicts 200 million will be correct; in others, that person will be wrong. We have no way of estimating the proportion of rights and wrongs in that set, but that proportion has no mathematical linkage to the first 5% of humans who ever lived, or whether there were more or less than 10 billion of them.
I think you're confusing within species probability and across species probability.

Grant Hutchison

No offense intended, your insights have been razor sharp. Indeed, on several occasions I've thought, "wait, he's right... no wait, I'm right..." But I see it clearly now, and there is only one point that I am having trouble making to you. It looks like you and I are all that remain in this debate for the time being, so it's up to us to get to the bottom of it.

Agreed.
Also agreed. As long as they never use the information that they are the 10 billionth, your statement is completely correct. But there's something we must agree on. The only meaning to the statement that "I am 95% confident" of anything is that I have in mind a collection of cases, or trials, or individuals, that are effectively indistinguishable, except for some random stirring which is being examined in regard to the confidence interval. How are you going to even define your confidence interval, if you don't use this definition? You have in mind a set of all humans, and the stirring is the difference between you and someone else. So yes, of that set, 5% are in the first 5%, it's well defined. But the same must be true of any statement of confidence. Therefore, if you claim that "I am 95% confident that humanity will not outlive 200 billion", you must also produce such a set (even if idealized) or it is a meaningless statement. Note this statement sounds very similar to your statement above, but it is not the same, because it explicitly uses additional information-- I am the 10 billionth. The first statement has a set where 5% are wrong, the second set does not, not even in principle. This makes it an incorrect probability argument, much like the set of people who saw a $10 envelope and figured it would be the cheaper envelope half the time. How do you prove that claim wrong? By asking for a set of people seeing $10 and having it be the lower envelope half the time. Could the supporters of that probability view simply say...
as if it saved their argument? The fact that there is no such set is exactly why the argument is wrong.

Exactly. This is why you simply cannot be 95% confident that the 200 billion limit applies. The within species probability is not enough, because it is not a probability that tests a confidence interval about a population that makes it to 200 billion, it only tests a confidence interval about being in the first 5%. These are different tests! What do you mean by "I am 95% confident our species won't outlive 200 billion." Where is the 95% who are right about this exact statement, right from their own mouths, and the 5% who are wrong? Carter must change the statement to "I am 95% confident our species won't outlive 20*x, where x is my birth number". That statement only checks out if you range over all x, but then where is the 200 billion? This is just like you had to range over all y to say that the expectation of the second envelope is y, and you realized that isn't true for a given y=10. It is just the same thing here.

Or to abandon it as an argument that's going nowhere, to be frank.
The same set provides confidence for both claims. The mathematical linkage between the first claim and the second claim is trivial.
(A paediatrician tells you that your son's height is in the 98th centile for his age. You go home and measure your son's height - he's 120cm tall. That immediately tells you that 2% of kids your son's age are over 120cm tall. You don't need to doubt that proportion just because you've now measured the height. It's the same proportion, with or without the numerical knowledge.)
Or this imaginary set has no relevance to the argument, which is my position. The set of relevance is the set of humans who make a claim about the likely total number of human lives, using their birth-number to make the calculation. 5% of them are wrong.
The set of people who make the specific claim that there will be 200 million humans, of which 5% are wrong, doesn't exist, as I said, for exactly the reasons you've set out. But it's just a non-issue for this problem.
Are you going to claim that there is a set of children 120cm high who are not in the 98th centile for their age, and so measuring your child's height invalidates the paediatrician's assessment?

Here's another way of looking at it. Saying you're 95% percent certain you're not in the first 5% of human lives is just shorthand for saying "If only 100 humans lived, I'm 95% certain I wouldn't be in the first 5." Inserting your birthnumber merely introduces a constant of proportionality into that claim. Each individual inserts their own birth number, and comes up with their own estimate of the total number. The claim stays the same because the proportion is the same, just as it would have done if you'd decided to express your claim as a fraction or a per-mil, rather than a percent.

Grant Hutchison

I hope we won't have to do that, the answer is out there!

Your example is perfect, it shows us exactly what we need! But you have to change it into a probability argument. Let's say my son's height is 120cm. Can I assert with 95% confidence that he is above the 5% rank with no additional information? Yes I can, because I never used the 120cm, I can always make that assertion. The real question is, can I put limits on the median height at this same confidence, by using that 120cm? No, that is exactly what I can not do! Here is how it would go. I'd assume height was a Gaussian variable, and I'd say I was 95% confident my son was above the 5% point. Let's say that means he is above half the median height (we'd need the standard deviation to allow this to be done, I'm picking numbers to come out simple). Can I now put a 95% confidence that the median height is below 240cm? There is a testable prediction, a calculation we can actually do and see who is right. I will wager anything reasonable that if we do this calculation, we will find that if we randomly select heights, and range over all possibilities, we will not find that 95% of the parents in our hypothetical study will be correct in the 95% limit they place on the median height! I don't know if it will be above or below 95%, but there's no reason to expect 95% as you must be claiming. (Unless, that is, there is something very special about Gaussian distributions, so I wouldn't wager too much until I'd thought that part out!)

Here you are ranging over all the people who make the calculation, so you are correct that 5% will be wrong. But we are not using the 5% number after ranging over all humans, we are using it just for ourselves, in saying that humanity is 5% likely to outlive 200 billion. The argument cannot be stated without the 200 billion number, so it has not ranged over all humans.

No, I'm not bringing in another parameter like age. I'm saying that the median height limit, for that age, cannot be calculated to 95% confidence using the single 120cm measurement (and the standard deviation information), because humans have an average height, and 120cm has some unkwown relationship to that height, so I can't make any confidence calculation about that average height.

The point is, since the number 200 billion appears in the Carter catastrophe argument, you have not ranged over all humans in presenting that argument, and so the 95% confidence does not apply.

OK.
We're now well past the stage at which you're just repeating your stuff and I'm just repeating my stuff.
Time to give up, I think.

Grant Hutchison

Wait! I've had a revelation. The reason that Carter's 95% confidence is in general not correct is the correlation between 10 billion and the actual "average" number of intelligent beings in any species. I've said that already. The only way Carter works is if there is no such correlation,and although you may not agree with that statement right now, perhaps you will when I point out that there is a way to eliminate that correlation, and that is if the population distribution is scale free. This gibes with your last point about being able to rescale the numbers, so that's why I think this might be a useful insight to this discussion. So what kind of distribution is scale free? A power law! So Carter only works if populations are distributed according to a power law. So it's not the Gaussian distribution, it's power laws, that are special. So that's good news and bad news for Carter. The good news is, there are a lot of distributions in life that do come out power laws. The bad news is, they always break down at some point. If the true population-distribution power law is too small, then you have zero population expectation, and if it's too large, then you have infinite population expectation. You have to introduce a scale to avoid these problems, and as soon as you do, bingo, the correlation appears and Carter breaks down. I maintain that your intuition is correct, but it's working under the unrevealed assumption of a scale-free population distribution, which in practice is not possible. But if you can resuscitate the scale-free idea, then you have resuscitated Carter. Otherwise, RIP.

It looks like this thread has reached its end, so I'll summarize what was learned for those who might still peruse it, or be curious about the validity of the Carter catastrophe conjecture. The idea behind the catastrophe is that if each person has x humans born up to their own birth, then they can be 95% certain they are not in the first 5% of humans, in the absence of any other information. This would seem to suggest that the 95% confidence extended to the idea that humanity would not outlive a count of 20 times x, but the primary debate centered on whether or not it was allowed to set x=10 billion (or so, for us), and still expect the 95% confidence to apply to the number 20x = 200 billion. To have this work, most felt it would have to be argued that setting x=10 billion did not constitute any extra information. But since 10 billion must have some relationship to the actual expected number of humans (i.e. high or low in relation to it), in a situation of complete knowledge of all the contributing factors, so most felt that using any actual value of x (such as 10 billion) did constitute extra information, invalidating the Carter confidence interval. This was not a unanimous view. No doubt the Carter conjecture will continue to be debated, but probably not on this forum!

Sorry if I'm resurrecting a dead thread and only using new words to reiterate arguments that have already been put forward, but it seems to me that the problem with the Carter Catastrophe is that it implies that the future or present probability of something is somehow altered by the probability of the chain of events leading up to it.
If I roll three six sided dice in a row, the chance of getting a 6 the last roll is still 1 in 6, not 1 in 216, no matter if I already rolled 6 twice in a row before.
Really... isn't any attempt to calculate the probability of any event or outcome based on anything but the probabilites of the direct physical factors that matter to the specific outcome/event in question inherently meaningless, and a matter of "confusing the map with the territory"?

I read er... listened to "time" by stephen baxter, i found the "Carter Catastrophe" pretty interesting, you see it happen all the time in nature, except maybe for cockroaches, which apparently never die out. I'm just going to forget about the hysteria and note a couple possible holes in the theory, first of all, it just cites population, not other important factors like technology level, area occupied etc. Secondly, you can say you have sufficient data, but really, is looking at a few samples of data over 100 or so years going to suffice, we may have accurate data for humans at least, but for the animals you're comparing us to you may not have enough samples for the comparison.
For the theory, if u suspended ur disbelief long enough, it does make sense if you think about it a certain way, first of all, all species are terminated on a distinct timeline, no exceptions even cockroaches, mosquitos, and fruit flies, which we wish were all dead now.

Otherwise the theory makes sense, but as history has shown us, the worst thing that happens with isolated populations is they all die (easter island) which we have a lot of, Or the government collapses and a lot of people die, but not all.

Before starting on this again, please note that this has been a resurrection of a thread that died 2 years ago when the discussion reached it's natural end.
The resurrection hasn't added anything new to the discussion.

__________________
‘To those who regard “crime fiction” as some sacred icon which must follow a rigid formula, I will always be the man who writes 18-syllable haiku.’Trying to make sense of computers, The Error Log.

The probability of drawing a high numbered ball from the million ball urn is zero if the person filling the urn has had only enough time to put in the first ten balls. Drawing a seven tells us nothing about how many balls will eventually be placed in the urn. For the experiment to work, I'd have to wait until all of the balls are placed.

Even if the human race will be much larger, we haven't been around long enough to produce trillions of members so of course my birth order is low. No matter how large the population will become, everyone doing the experiment now is going to get a low birth number. It has nothing to do with how large the population will become just as the early drawing of a ball from an urn doesn't tell us how many balls will eventually be placed in it.

It may be true that most of the people from among everyone who will ever live will be correct in guessing that the human race will become extinct soon, but I have no reason to believe that I'm in that majority because we're not done producing more people. A ball can't be drawn from an urn if it hasn't been put in yet.

The problem with your argument is that it gives a result that even though 90% of humans will be in the last 90% born, by your argument none of them will ever be able to conclude that they are likely to be in the last 90% born, on the simple grounds that they have no way of knowing if we are done producing more people. That doesn't make your argument wrong, it means that it still leaves something to explain.

If all I knew was that there were going to be a finite number of humans and I was one of them then I'd have to say that there's a 90% chance that I'm in the last 90%. But I have made other observations. As stated in the problem, I know that the population is increasing geometrically. With the possibility of truly vast numbers of new people being added, how is it possible for anyone to seriously think that he's now probably in the last 90%?

It would be different if the earth really had standing room only or a killer asteroid were already spotted. But as it is, the same geometric progression that suggests that most of the people who will ever live will be in the last few generations also suggests that we aren't in those generations.

__________________
Life is like a box of chocolates. All of your choices are bad for you.

Just goes to show how little number games can actually mean sometimes.

__________________
"If this were play'd upon a stage now, I could condemn it as an improbable fiction."
Shakespeare, Twelfth Night
"The Mayan symbol for "book" looks a lot like a triple hamburger, but I've never seen them claiming it as proof the Mayans had Big Macs." - KaiYeves
"Distance doesn’t matter much in space, where if you just start a thing off with the right kind of shove, sooner or later it will get where you want it to go." -Frederik Pohl, Mining the Oort

Bayesian statistics (i.e. "Carter Catastrophe") only works when applied to finite sets. When applied to potentially infinite sets, such as total number of humans that ever will live, it produces meaningless results. In particular, it produces same result (we are nearing the end of the set) no matter when it is applied -- today, 1000 years ago, or 10,000 years ago.

__________________
Fiction has to be plausible. Reality is under no such constraint.

Yes, this is also my position on this-- that we are applying additional information that can compromise the idea that we are in the last 90% born. I feel that one can never use the number of people that have lived in the same argument as one that asserts we are 90% likely to be in the last 90%. So it's true that anyone can conclude they are 90% likely to be in the last 90% in the absence of all information, but they cannot look at how many people have been born (be it 10 or 20 billion or whatever it is), as then they are no longer acting in the absence of all information.
The Carter hypothesis would require that the exponential phase is quite likely to be coming to a close, on the grounds that any random person is likely to be born near the close of such a phase, not near its inception. But as soon as we look at how many people have been born, so where we are in that exponential process, we can no longer use that argument because there can be correlations that we are simply unaware of that spoil the Carter argument. In other words, if you have no information that can interact with correlations, then you have a truly random sample, but as soon as you use information that might involve correlations, even if you don't know what the correlations are, you can no longer assume your sample is random. That's also the conclusion from the "envelope puzzle" described in a thread spawned by this one, I forget its title. (I never reached rapprochement with Grant on this issue, I wonder what his opinion is at this point.)

But the number of humans that will be born is almost certainly a finite set. You're right though, this is all about Bayesian statistics.

I wrote potentially infinite set. It is undefined.

__________________
Fiction has to be plausible. Reality is under no such constraint.

But I'm saying it's not potentially infinite, the expansion of the universe would seem to preclude that. I don't think the weakest aspect of the Carter catastrophe conjecture is the assumption of a finite total human population, it is the assumption of a lack of correlation between the current birth number and the expectation of future longevity. But perhaps one could merge the arguments and say that if the total number may be assumed to be finite, then correlations exist between current birth number and the possible totals, and if the total is potentially infinite, then no such correlation exists but then the Carter conjecture is invalid on the grounds you raise.