It's not that you're stupid. It's that you have good intuition. It's pretty clear that the conclusion can not be correct. The reason is that the premise of the puzzle is flawed, so the conclusion is flawed. You've stated in words what I've stated mathematically above.

Two problems:
1) If your purpose is to find a 7, then of course sampling from a large number of large populations will give you more chance than sampling from a single small population. But that's irrelevant to our problem: it's one ball, and one bite at the cherry, whereas your summing of independent probabilities implies that you're sitting with a big range of urns, pulling a ball out of each, and counting success if any of those balls is a 7.
2) Carter's not interested in the unlikelihood of pulling a particular number, but in the unlikelihood of our selected number (whatever it is) coming from the lowest 5% of the numbered population.
I've already rehearsed Carter's calculation earlier in the thread:Grant Hutchison

Edit: Slight expansion for clarity.

That's exactly right. If the population curve were exponential from the very start, each generation would assume, with 95% certainty, that they were close to the Carter Catastrophe. That level of certainty implies that 5% of those who made that deduction would be wrong, and 95% would be right. But because of the exponential growth in population, the first 5% (the ones who're wrong) are strung out over most of history, and the last 95% crowd into the last few generations before the extinction.
So (like everyone else who's ever lived) we can use Carter's argument to deduce with 95% certainty that we're close to the End of Days. But there's a 5% chance that we're wrong, and in fact ahead of us is a vast bulge containing more than 95% of all the people who'll ever live. They'll be the 95% who're right, and we'll be in the 5% who're wrong, along with everyone else in history so far.

Grant Hutchison

Sorry, I missed this one on the first run through.
Not at all. If a small population has existed for a small period of time, only a few people have ever lived, and Carter's reasoning predicts that that population is likely to die out after only a few more people have lived. A "small disaster" might be sufficient to achieve that, given that the population is small. If a large population has survived for a very long time, then Carter predicts a large number of future lives before that population dies out. Probably a "large disaster" would be required. Since small disasters are more frequent than large disasters, this is all internally consistent: frequent small disaster eliminating small, short lived populations, and occasional large disasters wiping out the infrequent large populations who've been lucky enough to slip through the "small disaster" winnowing. (There's certainly genetic evidence that humans have suffered a near-extinction event when our population was small: we're genetically much more similar than we should be, given the length of time we've been around on the planet as a species.)

I think the exponential growth of the human population is what makes Carter's argument counterintuitive to some folk. If the human population were stable, then Carter's reasoning would reduce to:
"Hey, we've been around for a couple of million years without being wiped out. Chances are we can survive another couple of million. It's a 50:50 chance whether we last for a longer or shorter time than that."
Probably most people would find this an unexceptional bit of informal reasoning.
But Carter's calculation applies to human lives, rather than elapsed time, and if the distribution of human lives is skewed rightwards along the time axis, giving us a long pastward tail and a short futureward tail, people begin to become uneasy.

Well, it's been fun. I'm about to disappear into the Scottish mountains for a few days. I'll try to pick up this thread when I get back (assuming we're all still here).

Grant Hutchison

I still think my argument kills Carter; in order for a statistical argument to apply, the sample has to be taken at random; but we are not a random sample, and so statistical arguments cannot apply. We exist now, when an apparent exponential growth is occurring in the population; we did not exist in the Palaeolithic, when slow growth would indicate a distant doomsday, and we do not exist in the future, when a slowly rising interplanetary or interstellar population would indicate a distant doomsday.

We can only sample the population now, when the rapid rise in population seems to indicate a near-future doomsday, because we have only just developed the right statistical tools.
We did not have the statistical tools in the Palaolithic, so we were not concerned about Carter (and it would have indicated a distant doomsday back then anyway) and we do not yet exist in the future.
If we did exist in the future we would no longer be worried about Carter, because the population would be growing slowly again and this would extend the Doomsday ever further into the far future; additionally we would have a historical record of the period long ago when the Carter argument was first discovered, and would know that it loses its accuracy during periods of rapid growth.
In most cases Carter's Argument predicts a long, long existence for the human species; only now for a relatively brief and anomalous period, does the argument seem to indicate a near-future doomsday.

And what do you know? That is the exact same period that we discover the argument itself! This is not a coincidence; we discovered the benefits of civilisation, of science, mathematics and statistics all within a short period - this explosion in technology and knowledge has produced an explosion in population numbers, and also has produced the Carter-Leslie argument.


To recap; this statistical method requires that we are chosen from random from all humans that ever lived. That is not the case- we are self-selected and could not exist at any other period of time, not in the past or in the future.

__________________
New Orion's Arm Site . The Starlark . Against a Diamond Sky (OA Novella Collection) . OA Flickr set

And what is the chance that every generation of humans belonged to that 5 percent instead of the 95 percent? Well, that must be way smaller than the 5 % chance that we will be a long living species.
To say it in another way: the chance that we will be a long existing species is much bigger than the chance that we have made it thus far yet.
Just to put all this statistic spielerei into perspective...

__________________
Knowledge is a curse, but ignorance is worse

Folks, there's been a lot said on this thread to dispute Grant's claims, but you've been missing the target (myself included, early on) because all Grant is saying is that only 5% of any species will live in the first 5% of birth order, and it's unlikely that we are in that 5%. That is completely correct in the absence of any other information. The mistake is simply that this 5% probability is being treated as some kind of absolute probability, like the chance a coin will flip "heads". But it isn't, it's a conditional probability, conditional on your knowledge. Here's an example of the difference. If you sit down to a game of flipping coins, you have a 50% chance of winning, no matter who you are. That's absolute probability. But if you sit down to a game of chess, you can't say, there are 2 players so I have a 50% chance. That's true, however, in the complete absence of any other information! But if you have any information at all, the chances change. If, for example, you know that your opponent is the World Champion, and you just learned how to play yesterday, obviously your chances are less than 1 in a million. But if you also know that your opponent wants to lose because he or she is very generous, then your odds could reverse. It's a conditional probability, it's all about what you know. So the real question that should be debated is, do we really have no other knowledge, in which case Carter is the best we can do no matter how many "urn analogies" we trot out, or do we have knowledge that completely changes the probabilities? This kind of amounts to saying, are you an optimist or a pessimist? But the optimists can certainly take comfort in the fact that the Carter argument is not an absolute probability.

I think there's a problem with this. None of the components of the "privileged viewpoint" you invoke have any influence on Carter's argument: it therefore doesn't matter whether or not sampling is random with regard to these components.
1) Carter's argument applies whatever the shape of the population curve (flat, exponential, declining, bell-shaped).
2) Carter's argument applies whether or not the population under consideration knows about it (we could apply it to a population of lemmings, for instance).
3) Carter's argument applies whether or not the population under consideration is technological.

You may well be correct that it's no accident we find ourselves understanding Carter at a time when population growth turns his prediction into a "Catastrophe". But Carter's argument is merely a statistical observation, and it is no less true during periods of of stable population, when it predicts a long future life-span for the population.
So I think you're indicating why the "Catastrophe" aspect maybe shouldn't surprise us, but I don't think you're undermining Carter's reasoning.

Grant Hutchison

No, the low probabilities don't multiply as you're suggesting.
Each new generation has a new piece of information (their own existence) and therefore redoes the calculation, coming up with a new 5% and 95% confidence interval.

Grant Hutchison

But for the Carter catastrophe to be correct, you could take an imaginary 1 million species to start with. After 1 generation, only X (let's say 100,000) would still exist. Of those, 20,000 gets to a third generation, and perhaps 5,000 to a fourth one, and so on. So the chance that we have reached here is of course 100%, but the chance that a first generation species has survived as long as we did is 1 in 1,000,000 (random number, again). So it is much more amazing that we exist, according to the same hypothesis, than it is to suppose that we will exist for a very long time afterwards...


Another criticism / point of view. An alien visits the earth, and sees just one generation of people. The chance that it sees us is bigger if only a few more generations will survive than if we will live for millioàns of generations (thus far, standard Carter). Now take the opposite view. An alien comes to the earth, and sees another generation than us. The chance of this happening is much bigger the more generations there are, and much smaller if only a few generations are to come. So when you take the starting position that the alien has picked another generation (and this one is equally valid as the supposition that he has picked us), the chance of a long survival is actually bigger than the chance of a short survival.
A nice statistical paradox!

__________________
Knowledge is a curse, but ignorance is worse

There is no point in arguing that the Carter hypothesis is wrong, unless you are bringing in additional information that bears on whether we are more or less likely to face extinction than purely random chance with no information at all. The only criticism of the Carter thinking that makes any sense, to me anyway, is the criticism that it draws on so little information that its conclusions are meaningless. It's like betting a poker hand before anyone else has bet, and you haven't looked at your cards.

Carter's calculation necessarily applies to a single species, the species of the observer. Otherwise the observer would not be a random sample from all possible individuals in the population.

Carter doesn't say anything about this, and it doesn't seem to make sense, probabilistically.
Carter says only that an alien who did encounter us might be able to make some prediction about how long we will survive in the future, based on how many individuals of our species have lived so far. An alien passing at random is always more likely to encounter a long-lived species than a short-lived one, and to encounter a species somewhere in the middle 90% of its existence rather than at its start or finish.

Note: I say "might be able to make some prediction", because for the alien's random sampling in time to strictly correspond to Carter's random sampling in lives, the population would have to be stable across time, with the same number of individuals living at any given moment.
I've realized I didn't think this through properly the first time, when I introduced my visiting alien in discussion with eburacum. The same proviso would pertain when I suggested we might be able to apply Carter's reasoning to lemmings ... lemmings were a very bad example indeed, since their population is notoriously unstable.

Grant Hutchison

The lemmings problem can be rectified by using the Carter reasoning on the number that will likely yet live, rather than on the time the species will remain. The time always requires further reasoning, but the number comes right from the probability analysis. But I reiterate, knowing what we know about species here on Earth, it would be silly to not try and do better than the estimate that comes from the Carter reasoning from zero information (which goes back to Fram's initial post).

I don't think so. Carter requires that any life is as likely as any other life to be sampled. Since we take a random snapshot of our lemmings in time, we're as likely to sample a bulge with lots of lemmings as a trough with few lemmings; but a random lemming life is more likely to sample a bulge than a trough. So we can't sample the population in a "lemming-life" way without knowing how it's distributed in time - but if we know that, we don't need Carter.
I completely agree that Carter is best undermined by examining his simple assumption, rather than his simple reasoning. Simple assumptions are very often wrong, whereas simple reasoning is very often right. Nevertheless, it's my experience that people are often convinced there's a flaw in his reasoning, if they just pick away at it for long enough.

Grant Hutchison

Ironic that I now find myself to be the Defender of Carter! The point is, it makes no difference what the current population is, only the total birth number. If you think there might be correlations between extinction and present population, join the crowd. My point is, to use information of any type (outside of birth number) will break you from the Carter zero-information analysis, it makes no difference what the information is.

Yes, it is the assumption that we have no information to go on, when in fact it's not hard to find something that could correlate with extinction. The problem is, I can't help wondering if Carter's assumption is optimistic!

Ah, but if we sample at a random point in time with respect to an exponentially growing population (like lemmings or humans), then we are more than 5% likely to sample from the initial 5% of lives, simply because those lives are spread over a long time period.
We therefore can't move on to state that the birth number we find on sampling has only a 5% chance of existing in the first 5% of lives, and we've crippled Carter's reasoning. In fact, if we sample at a random time, we can't deduce a 95% confidence interval without knowledge of the shape of the population curve ... and if we have that, Carter is superfluous.

The elegance of Carter's insight is that sampling a random life (simply by living), allows you to forget the shape of the curve and derive a confidence interval with no other information but your birth number. Random sampling in time will only give equivalent information if lives are uniformly distributed along the time axis.

Grant Hutchison

But why are you discussing sampling in time? We are sampling a life, under the assumption that all lives are equally likely to be sampled. Other constructions are certainly possible, such as, always selecting the first life! But I agree with your main point, that the Carter approach assumes the life we lead is a randomly sampled life from all of human lives, equally weighted. Again, that is what you would assume in the lack of all information about what makes you, you.

Because I introduced the topic of external sampling earlier in the thread (of ourselves by aliens, of lemmings by humans). In both cases these samples are necessarily randomized with respect to time, rather than lives. (Because we can't choose a life at random as observers outside the population unless we have a knowledge of the population profile against time, in which case we don't need Carter, because we have external information.)
Given that I'd inadvertently introduced a flawed example, I felt obliged to point out it was flawed and to explain the flaw. I can't get sniffy about other people's rotten explanatory analogies while allowing my own to persist!

Grant Hutchison

I see. Yes, I think a lot of the problems in this thread involved invoking analogies to help understand the point, but the point is so delicate that analogies are not likely to work! I think it's more important to get away from the informationless Carter argument and into areas where we can actually assess, and possibly alter, our survival chances. For example, I'd say that natural catastrophe is no longer a serious issue, as it would probably take a major extinction event to wipe out humanity, and those happen on timescales of tens of millions of years. But humanity's technological development is happening on timescales that are up to a million times shorter! So it is clear that our survival depends entirely on our responsible use of our own technological advancements. No timescales embedded in human history are relevant at all any more, nor is the Carter argument, because in point of fact our fate is in our own hands now.

Yes ridiculous. ,and dangerous. To sagest the domsday of humanity becouse the probabilaty points to it is a nonsence. Its likened to a religiouse balief. Based on speculative prediction, and probabilaty calculations. I dont see any science here.

Fortunately I wasn't attempting to construct an argument from analogy - just trying to think of some ways Carter's reasoning could be applied to a population that didn't understand Carter.

Grant Hutchison

I think I see what you're saying, you're saying that if you wanted to try and apply Carter to other species, perhaps go out and take a census of all the living species on Earth for example, right now, then you would not be able to conclude that only 5% of the species were in their first 5% in birth order. In your example, you say what if all the populations have exponential growth prior to total extinction, then the vast majority of the species you would encounter would be early in their growth because we are sampling at a random time. The only time you can know the population growth behavior but still apply the Carter conjecture is if you are applying it to your own species, such that you are a randomly chosen life no matter what information you have about what the distribution is doing. I think you are right about that.

Here is another wrinkle though, if we want to restrict ourselves to the thread that we have no useful information about our survival so we may as well apply Carter, probabilistically. To what extent can we count ourselves as a random sampling from all of humanity? Could our genes come at any point along the way? Indeed, what if humans start doing genetic engineering on the genome, such that you or I would be impossible 10 billion humans from now? So the Carter conjecture might not hold simply for extinction, in the case of intelligent life it may only hold for the time it will take to alter the genome such that you are I are no longer a randomly chosen life over all humanity. Put differently, there has to be some criterion for constraining what the selection is occuring over. If we can count future human progeny that is even minutely different from you and I, then where do we draw the line? How much similarity to us is required for it to count in the Carter selection process?

And here's yet another wrinkle. If the elegance of the Carter conjecture involves the sentience of the being doing the reasoning, which seems to be a necessary part of "selecting a life", then did you have to be a man to count? In other words, was your life selected from the lives of all humans, or just human men? This speaks to the question, could you really have been any human, one life selected at random, or did you have to be exactly you, a single individual who won an unbelievably unlikely lottery to even be here at all?

Yep, that's what I was saying. But here's an interesting paradox that seems to arise from that.
1) Suppose a visiting alien had observed humanity at the time of Aristotle. Seeing in us the potential for exponential growth, the alien realises that he cannot use Carter's reasoning to predict our extinction, because his visit is random in time, rather than in lives.
2) The alien explains Carter's reasoning to Aristotle. By the same token, Aristotle is not entitled to use it to predict human extinction, because the alien has delivered knowledge at a random time, rather than to a random life.
3) But suppose Aristotle had come up with Carter's reasoning for himself, without alien intervention? He would then assume himself to be a random sample from human lives, and could reason according to Carter.

It's the difference between 2) and 3) I find paradoxical, and it seems like there is some germ in there related to eburacum's earlier "privileged viewpoint" discussion.

Yes, there's a lot of discussion about when you start counting, when you stop counting, and what you count.
Does the "Carter Catastrophe" merely mark a transition to some trans-human condition, as you suggest?
Do we count other species of early human, or do we start from what seems to be some sort of extinction bottleneck in early homo sap existence? I can't now remember exactly how Baxter came up with the very tight 150-200 years cut-off quoted at the start of this thread, but I seem to recall he used some argument to eliminate many humans from the count: perhaps using only post-industrial humans, or those who have existed since Carter's argument was first proposed.

Grant Hutchison

I think I have the answer that will clear a lot of this up. As we've discussed, the Carter conjecture requires using no information, or the probabilities could change. But you do have to use one bit of information to make it fly-- you have to know what your birth number is, say 10 billionth. But beware, as soon as you input any information it will change the odds, even if you are not sure how. Ultimately, the validity of the Carter argument therefore relies on an unjustifiable assumption about how the total population numbers of intelligent species are distributed.

Let me clarify. Lets assume we have an immortal alien, keeping a census on all intelligent populations in our galaxy, from the beginning. No matter how the populations are distributed, only 5% of those beings will live in the first 5% of their populations. But is this still true of the subclass of 10 billionth born? Very likely not! For example, what if the total number of an intelligent species is a random number evenly distributed from 1 to a trillion. Then it is clear that if we restrict to the subclass of 10-billionth borns, we have populations that range evenly from 10 billion to a trillion. Hence in that model, the 10 billionth born has about an 80% chance of being in the first 5%! The answer depends on the population distributions. So the Carter conjecture is internally inconsistent-- you are not allowed to use both your birth number and the 5% chance of being in the first 5% in the same calculation! The only time this would be valid is if the population distribution worked out such that 10 billionth borns really did work out to be in the first 5% of their populations in 5% of all species, which would require an amazing coincidence and we certainly have no reason to expect it.

So here's what I'm saying. It is correct (modulo all the other difficulties about counting) to say "as a randomly chosen life, I have only a 5% chance of being in the first 5% of my species". What you can not say is "as the 10 billionth born, this means there's only a 5% chance our species will outnumber 200 billion". You can't use the number from the second sentence in concert with the number in the first sentence, because you would definitely expect the information in the second sentence to screw up the truth of the first sentence, even if you don't know how!

Isn't this a variety of the sort of "external information" we've agreed would undermine Carter if it were reliable?
Carter says nothing about how a number of different populations will behave, only about your own population.
In your scenario, each of the 10-billionth souls would conclude, according to Carter, that there was only a 5% chance they were living in the first 5% of their population.
It would turn out that 80% of them were wrong. In their own populations, that simply means that they are in the 5% pastward tail, and that the 95% of their population who come after them are correct in using Carter's reasoning.
Carter would therefore still be entirely correct and consistent within each population, although the 10-billionth souls could have made a better estimate if they'd been party to the external information your immortal alien has collected about population numbers.

Grant Hutchison

Most of us wouldn't have come up with the Doomsday Argument.

Therefore, we are not a random selection of the human race. We are part of the subset of the human race that existed after the Doomsday Argument was raised.

Thus the initial assumption of the argument is incorrect.

__________________
"I worry that, especially as the Millennium edges nearer, pseudo-science and superstition will seem year by year more tempting, the siren song of unreason more sonorous and attractive." - Carl Sagan, 1995

Trouble is, Carter's argument applies to you whether or not you know about it.
So the non-random distribution of "Doomsday knowledge" doesn't seem like it should have any more of an effect on Carter's "random life" stipulation than the non-random distribution of, say, knowledge of the rules of cricket.

Grant Hutchison

The trouble is that the argument doesn't apply at all.

Look at it this way: if you can end up with a paradox, then you didn't state the rules of the problem correctly, or you've made a bad assumption. The world is consistent. So regardless of whether or not you accept that a specific criticism of Carter's idea is valid, the fact that you are coming up with paradoxes should lead you to believe that the idea is flawed somewhere.

No, it's not external information, it's internal information. The point is, you cannot use the number 10 billion anywhere in your longevity calculation, and also say that there is a 5% chance you are in the first 5% of your population. The distribution I gave as an example proves this. There has to be some probability distribution, it's not external information until you specify it. But no matter what it is, Carter would not be correct in his 5% probability, except under specially chosen conditions.

You understand what I'm saying, yes, but follow up your reasoning. As a function of birth number, given the distribution I assumed, the actual probability of being in the first 5% is completely a variable. For some it's less than 5%, for some more. This is my point-- there is automatically correlation between birth number and longevity probability. Carter thinks that because you don't know this correlation, you can ignore it and still use both the 10 billion number and the 5% number in the same calculation. In fact, you may only use one or the other, not both, even if you don't know the correlation between them. There are many other examples of this phenomenon in probability. I'll start a new thread about two envelopes containing money to demonstrate the point. It is very subtle, and I think it is the real reason that the Carter reasoning is not just of limited value, as I argued before, but it's downright wrong. The other arguments are working from an intuitive place where Carter can't be right, but they haven't really hit the mark because correlations in probability have a very subtle effect sometimes.

For sure. Are you referring to any specific paradox?

Grant Hutchison