The entertaining aspect of this problem is that the theory predicts the opposite of what would seem like common sense. If a population is thriving then the theory predicts an early die off. If a population is declining then the theory predicts that they'll be around for a long time.

Since there's nothing special about me we can estimate that there will be about as many people born after me as there were born before me. Since population is increasing, those born after me will live in far less time so we face early extinction. Back in the sixties I bought myself a slide rule. I didn't really need one but wanted something to play with. There was nothing special about that slide rule so we can estimate that there will be about as many manufactured after it as there were manufactured before it. Since the calculator made the slide rule obsolete they're now being made only as novelty items so it will take a long time to manufacture as many as already existed when I bought mine. This reasoning leads to the conclusion that we'll still be making slide rules for millennia after we've become extinct.

I think that's another fallacy in the Carter argument. All you can say is that 90% of humans will be born in the last 90% of humans, so if you and I are randomly selected humans, then we are 90% likely to be in the last 90%. But that will no longer be true if we specify our birth number (say 15 billionth, I don't know), nor will it be true if we specify that the population is growing exponentially (and like you say, Carter used the exponential growth to convert birth totals to a timescale, which involves additional information that cannot be used if one assumes we are randomly selected humans unless one thinks a population grows exponentially right until it goes extinct).
Yes, that exposes the fallacy.

I think it exposes a fallacy, but not one in Carter's reasoning.
The fallacy here is in doing a post hoc analysis once the results are in and the distribution is known. "There was only one chance in 14 million that my lottery number would come up, and yet it came up: that's so amazing!"
At the time Chuck bought his slide-rule there was a 90% chance it would be in the last 90% manufactured, and a 1% chance it would be in the last 1% manufactured. Turns out, with hindsight, that it was unusual, in that it was bought at the cusp of the Carter catastrophe for slide-rules.

Grant Hutchison

I used the slide rule as an example to get the result that I wanted just as Carter or whoever used the increasing human population to get the result that he wanted. The fact that he knows that the population is increasing is similar to my knowledge the the slide rule is obsolete.

The difference is that you are observing the situation post hoc, with a strong indication that the bulge of sliderule manufacture has passed. You have excellent reasons to believe that our current situation is unusually, and that we have lived through the End of Days for sliderulekind.

Carter's argument holds only in the absence of a priori reasons to believe we are unusual with regard to our birth order. Either we are unusual in our birth order, or the current exponential growth in population will be strongly modified in the near future.
It's such a banal statement that I've never been quite sure why people get exercised about it. (I'm not suggesting that you are getting exercised, by the way; simply observing that emotions do seem to run high on this topic, quite often.) It's probably twenty years since I read Carter's original paper, and I can't seem to find it in the files, but I don't recall him saying much else apart from inviting us to come up with a convincing argument that we are in a privileged position with regard to birth order. It has since been overegged as some sort of inevitable, unavoidable doomsday scenario, but I don't recall that being Carter's stance.

Grant Hutchison

I do have reason to believe that my birth number is relatively low. Population has been increasing for millennia so it seems likely that there will be more people born after me than before me. It's certainly not a sure thing, but I do know more than just my birth number, just like in the slide rule situation. I'm not even certain about the slide rules. Some religious sect could take over the planet and ban electronics. Then slide rules would make a comeback. I don't expect that to happen just as I don't expect humanity suddenly die out.

I see little difference between the slide rule situation and the population situation.

In one case you have knowledge; in the other case you have an assumption.

Grant Hutchison

I can only guess that slide rules won't make a comeback. I can't see the future in either case. I see slide rules decreasing and population increasing. I see no immediate reason for either trend to end. The situations are nearly identical.

Just musing about some factors here:

1) The human race is not a distinct set - it is a continuum with all ancestors that went before it way back to the first replicator (and there are probably some parallel near-human life branches that could interbreed with humans that are already extinct). "Species" is a term of convenience (endearment?). So shouldn't we apply the argument to all life leading up to this point rather than to a fuzzy set like humans?

2) The Earth's human population is predicted to peak by the end of the century anyway (source UN).

__________________
Always challenge the assumptions

The similarity isn't really jumping out at me, I have to confess. So I'll call it a day on this one.

Grant Hutchison

Yes, that's one of the big debates among people who take Carter seriously. What is the population we're dealing with? What is the "start" and what is the "end"?
Do we start the clock with the last speciation event, the last near-extinction bottleneck, or just in 1983 when Carter first raised the idea?
At the other end, some of the arguments that "undo" Carter bring their own implicit catastrophe. What if the vast majority of humanity will exist in a form different from the one we currently have (as software, cyborgs, or something unimaginable)? Then we certainly occupy a privileged position in the first 10% of "humanity", and Carter is undone. But shouldn't Carter's argument then apply to "humans like us"? In which case, Carter may be both correct and wrong: "humans like us" disappear, while "humanity" in a different form persists. Similar reasoning applies for the "infinite lifespan" argument, in which we must necessarily stop being "us" in order to continue as "humanity".

Grant Hutchison

I see a trend in each of the quantities of two sets of items. I see no reason for either trend to change in spite of the U.N. prediction. If the population does peak I'd expect that to work against the doomsday prediction anyway.

Actually, there's two separate issues there, relating to whether one is talking about time or just birth order. If one is only speaking about birth number, then the exponential character of current growth plays no role. The "fallacy" I was referring to above was the connection with time, the idea that we only have a few more e-folds to go at the current growth rate. In my view, that confounds the initial fallacy by also connecting it to a temporal growth rate. But the real issue continues to be the initial mistake of connecting birth number to the generic character of our personal standing. Either may exist alone, but the two together are incompatible, they violate basic assumptions needed.
The statement that either we are in the last 90%, or we are unusual, is not controversial. It is the statement that the Earth's population has only a 10% chance of exceeding its current birth number by a factor of 10. That statement is true generically, but it is not true if one ever actually uses the current birth number. That number has unknown correlations that violate the basic assumption that we are generic. So what I mean is, anyone can say "I'm probably in the last 90% of the Earth's population", but if they actually look at their birth number, they cannot use the number 90% and the birth number in the same probability argument. It's just wrong to do so.

And yet the whole of inferential statistics is based on taking actual numbers and plugging them in to normalized sampling distributions, in order to derive other actual numbers.

Grant Hutchison

I am not saying one must avoid actual numbers when doing probability and statistics, I'm saying one must keep careful track of one's assumptions to make sure they are not invalidated by those numbers. Here we have the assumption that we are "generic" humans, sampled randomly from the total (eventual) population. We also have our birth number. There are correlations there which we cannot assume are absent, or we are doing incorrect probability. The fact that we do not know the correlations does not allow us to ignore them.

Let's look at the situation with birth number. I don't remember what I said above, but this argument seems pretty direct. Let's imagine that at the end of our galaxy, a super-intelligent species looked at all the intelligent beings that ever lived in that galaxy (perhaps from careful archeology), and took stock of the total number of beings that lived in those species. There'll be some kind of distribution over total birth number. (Let's define "intelligent species" as "one that considered the Carter argument" at some point.) Can we say that 90% of those beings lived in the last 90% of their kind? Certainly yes. Can we say that 90% of those who, individually, considered the Carter argument, were in the last 90% of their kind? Who knows, but very probably not. It is quite possible that either this argument comes up long before the end of a species, maybe because it shows enough self-awareness to stave off extinction, and it is also entirely possible that it comes up about the same time as the species wipes itself out, out of suicidal angst of some kind. So it already violates the "generic" assumption, needed for 90% to live in the last 90%.

But even without that argument, the Carter hypothesis still fails, because even if we do not ask if those species asked the Carter question, if we instead ask if they discovered the wheel, or fire, there will still be some kind of probability distribution of total birth number per species. So true enough, 90% will live in the last 90%, but can they use their individual birth number as a predictor of the the total birth number? Let's look at some examples.

Imagine a game where a number is selected at random and with equal likelihood from a distribution from 1 to N, but you do not know N. Furthermore, you are told that first another distribution, that you know nothing about, is used to choose N, and then you get your number from 1 to N. Now you are asked, before you look, what is the probability that your number will be larger than N/10, whatever N is? Answer, 90%. We agree there. Now you look at your number, and it is 100. Then you are asked, what is the probability that N is less than 1000? Not 90%, that is the wrong answer, pure and simple. You simply have no way of answering the question meaningfully. If you doubt me, try using various distributions to choose N. Unless you choose a "rigged" distribution, you will see what I mean. (It suffices to choose a bimodal distribution of just two possible N, so that you are basically playing my game with the envelopes, so that's why I introduced that other thread.)

Let's suppose that the human race does continue to increase exponentially and then comes to a sudden end. Then a time traveling alien comes back and asks one of us if the end is near. If the alien chooses one of us at random with equal probability then the correct answer is probably yes. If the alien chooses a random year from among those in which we existed and then asks someone in that year then the correct answer is probably no.

If I'm asked that question now, how do I answer? Do I assume that I'm a typical human being or do I assume that this is a typical year? If the alien asks everyone and everyone guesses no, the end is not near, then most of the individual answers will be wrong but for most of the time it will be right. If I want to maximize my chance of being right, what should my answer be? If I assume that I'm a typical human being and answer yes, the end is near, then I'm assuming that this is not a typical year, it's a year very close to the end. Can I justify assuming that there's anything special about this year any more than I can assume that there's something special about me? It seems that I'd have to do one or the other.

Or maybe this statistical method can't be used to predict the future. Not that abusing a statistics isn't fun, of course.

That is indeed a big problem, relating to "Bayesian statistics". How much do we already know about ourselves that we are allowed to use? In other words, one set you could take would be humans, as we are, but that does lead to all kinds of uncertainties. Or, it would be equally valid to look at the sum total of all beings in the entire universe who are capable of asking the Carter conjecture. Is that not also a valid set? We should conclude that we are most likely to be among the last 90%, and the first, of that set, as long as we don't use anything about ourselves other than that we asked the Carter question. As soon as we use one other single thing, anything about humanity, anything about birth number, anything about how we think-- we can not use that other thing and the 90% idea at the same time in any valid probabilistic calculation.

But the "castastrophe" in Carter's reasoning isn't driven by the value of my birth number, or the total number of births. It's dependent on the observed shape of the curve (exponential), and the "true enough" reasoning that 90% will live in the last 90%. We need no other numbers at all.

There is no such thing as a "typical year", since each year contains more humans than the previous year. So you are constrained to reason that you're a typical human. (Of course, you may produce arguments to suggest that you are not a typical human, as Ken G does above, and eburacum45 has done earlier; that's another matter. But I'd suggest that "typical year" doesn't fly at all in this context.)

Grant Hutchison

It's not the number itself, it's the scale that matters. We do need the scale, or there's no "catastrophe". If all we say is that we are 90% likely to be among the last 90%, that's all well and good. What we cannot do is say that the total population N is 90% likely to be less than 10*M, where M is our current birth number. That does invoke that scale, even if we personally have no idea what M actually is.

Let's take an example. Let's say that intelligent species face a critical moment when they develop nuclear weapons. Half wipe themselves out at birth number around 10 billion, and half figure out how to deal with it and typically make it to birth number 10 trillion (just make believe this, as an example). Now, it is still true that over both those subsets, 90% of the people will be in the last 90%. But if you go to someone who has birth number 5 billion, that sets a scale for the "catastrophe". Now ask them, what is the chance their species' birth number will exceed 50 billion, Carter would say only 10%, but the correct answer is 50%. Now you might say that the additional information about the N distribution has enabled a more precise determination of the probability, but that's tantamount to saying that "everything that can either happen or not has a 50% chance if we know nothing else about it". What is actually true is that if we know nothing about it, we cannot assert a meaningful probability-- you have to be able to make assumptions about what you don't know or you cannot use probabilities, they mean nothing.

Note that we don't need to know our own birth number to apply the above logic. It is still just plain false to say that there's a 90% chance the total number of humans that will live is less than 10*M, if our birth number is M and we don't happen to know M (indeed, we don't). You still have to use M even if you don't know what it is (the scale must appear in the answer, or there's no "catastrophe" to worry about). Try some distributions for N and you can verify that it is not true that 90% of the total numbers will always be less than 10*M-- that will only be true if you average over M, which is very much begging the question of the Carter catastrophe. This is the point, if we say Carter is right only when we average over M, the "catastrophe" is gone-- there's no scale any more. It all says a lot about what probability is-- and what it isn't.

It's the same with the timescale of course-- one must invoke what the e-folding timescale is before one gets a sense of "catastrophe" in the time domain-- there always has to be a scale in your mind, even if you don't numerically specify it. Carter can say "we are 90% likely to be in the last 90%" of any set from which we are generically chosen, after averaging over all distinguishing subfeatures of that set (even in the entire universe as a whole, not just human). But that is all-- there can be no other information, no scale in time or number, that is not being averaged over and must not appear in the answer.

And that's true separately from the other big problem-- it's far from clear that "humanity" is the set from which we are generically chosen, such that "M" for humanity is what has been averaged over. What personal attribute of ours is really the thing that identifies our generic set, all the rest having been averaged over? Our height, age, expected lifespan? If 100 years from now, the life expectancy is 1000 years (somehow), does it mean that people who think about the Carter conjecture would be weighted toward those, as they have more opportunity to have this conversation? What if tomorrow you write a very influential book on the Carter conjecture, such that it becomes a household word for billions of people (I expect some royalties there). Are we generically sampled from the set of people who have ever thought about this, even though probably up to now only a few million people at most have? Either of these two main objections put the lie to the Carter catastrophe idea, it's simply an example of how probability can not be used. You have to know more about what you don't know, about what is being averaged over, before probability has any meaning.

But if every year contains more people than the previous year then they're all typical. I don't see how pointing out such an extreme similarity in any way shows that something is not typical. The only nontypical year would be one that has fewer people, such as just after a war or a plague.

Well, if you are a year, they're typical. But you're not a year: you're a person, sampled from the Sea of Souls, who is more likely to have been born in a year when lots of people were born than in a year when fewer people were born.
So what's a "typical" sort of year for people to be born in? 10000BCE? 2000CE?

Grant Hutchison

OK. That's us back to where we ended up the last time.

Grant Hutchison

The fact that I was born recently instead of later doesn't mean there's less chance of there actually being a lot more people later. It's like drawing a ball from a 100 ball urn or a million ball urn and throwing it back if its number is higher than 100 and drawing again. If very few higher birth numbers than mine can be chosen now then it tells us nothing about how long humanity will last.

I'm pretty sure I don't understand what you're saying, here.
But I notice "typical year" is no longer mentioned. Have we reached a resolution on that?

Grant Hutchison

If I knew that human population would increase geometrically and then end suddenly and that a time traveling alien chose a year at random and asked the first person he met if the end were probably near, I think the correct answer would be no because most years aren't near the end. If the alien picked a human at random with equal probability then the answer would be yes because most people lived near the end. A typical year would be one in which I had no additional information concerning our probably extinction. If I were in a typical year and an alien appeared and asked me if the end were near but didn't mention how I'd been selected, how should I answer? He might have chosen me at random, chosen this year at random, or used some other method of selection. The best I could do is say that I don't know.

There is no alien here that I can see, so how was I selected? It appears that I chose myself, but that hardly seems random. Was I chosen by happening to read the question? That's somewhat random but I can't have been chosen at random from the set of everyone who will ever live. People who haven't been born yet could not have been selected. That's like not being able to draw a high numbered ball from the large urn. From my point of view I'm in front of the 100 ball urn or the million ball urn but the only balls I can reach are numbered 90 to 100 no matter which urn I'm drawing from. Drawing ball 95 doesn't tell me which urn I drew from.

If I consider myself to have already been selected at random by circumstances beyond my control and I want to answer the question accurately, which answer should I give? "The end is near" will ultimately be the correct answer for most people but it will have been the wrong answer most of the time. Knowing that the end will be near for most people some day doesn't seem to help me now since I might still be in the time in which the end is not near, which is most of the time. Since we're assuming that I have no indication other than my birth order, I don't see how I can make an accurate prediction.

I'm not sure what it means to think that I'd have had a greater chance to be born in the future because more people will be born then. It's unlikely that my genetic duplicate will ever exist again and even if one did, he'd grow up in a different environment and be a different person. I don't think I could exist anywhen else but now. My being here has nothing at all to do with how many people will exist in the future unless it's possible that I could have been one of them instead of being myself, in which case my being here means a lesser chance that they exist. But the existence or absence of anyone in the future did not change the probability of me being here in the slightest.

Is is nonsense because in the argument you use a guess for the total number of people which will live over the lifetime of the human race plus how many have already lived, to get a statistically expected number of people over the lifetime of the human race.

Essentially he's deriving a number from a guess at its value.

__________________
‘To those who regard “crime fiction” as some sacred icon which must follow a rigid formula, I will always be the man who writes 18-syllable haiku.’Trying to make sense of computers, The Error Log.

I believe so. But did you see my post about the game where first the gamemaster invokes some unknown distribution to set N, and then hands us a random number from 1 to N? Surely this is a canned enough problem with obvious similarities to the Carter conjecture, so my question is, will you agree that if you look at your number, and it is M, that it is false that there's a 50% chance that N < 2*M? The correct answer depends on the distribution over N, call it p(N), since
prob that N < 2*M = (sum over all N > M-1 and N < 2*M of p(N)/N) divided by the (sum over all N > M-1 of p(N)/N )
Agreed? So what's clear is that the probability is not 50% if p(N) is known, the question is, is it 50% if p(N) is not known? The answer depends on how the data is "sliced". If you look at every time the game is played, is there a way to slice the data such that it is not true that 50% of the M are > N/2? The answer is yes-- slice based on M. In other words, look at every trial where the same number M was chosen, and ask, is M > N/2 in half those trials? The answer will certainly be "no". It is only "yes" if you slice based on N (so average over M), or take all trials (so average over N and M). So the question boils down to, does the Carter catastrophe idea slice the data based on N or on M? It is sliced based on M, clearly. In other words, if you and I are birth number 15 billion or so, then we must group ourselves with all the other beings in the history and future of this galaxy whose birth number is also 15 billion. Then we must ask, will 50% of that group lie in the last 50% or their species? Answer: no. Agreed?

Or put differently, the flaw is already apparent in the association:
What is bogus here is imagining that the urns are similar to the population situation, because with the urns we have a clearly valid assumption that each urn choice is equally likely. Why can we assume this for the populations? Imagine that the urn containing a million balls is a billion times more likely to be chosen, for whatever reason. Can we still say that if we see the ball "7" that it is a "strong indication" the urn with ten balls was chosen? We cannot say that, without the implicit assumption about the urn likelihood-- and we cannot make that assumption for populations, we simply do not have the necessary information to do so. We can expect that our selection of "7" argues against the million-urn being vastly the more likely choice, but we cannot assess the likelihood that we have the 10-urn, without knowing more about the mechanics of the choice process. For example, let's say instead of choosing an urn, we mix all the balls together, and the balls from one urn are blue and from the other are green. Then we choose a blue "7" at random-- my question is, was it the 10-urn or the million-urn that had the blue balls? We have no way to say. So how is my specification not the valid way to look at the population issue-- why is the urn approach more valid?

I do not understand your argument - you first state that one selects a "random number from 1 to N" and then you argue that the distribution of such numbers is not simple or continuous but may be some unknown function. That would seem to conflict with the usual expectation of the meaning of selecting a "random number."

The strength of the Carter hypothesis is that its statistical conclusion is correct when you know nothing about the likely future distribution of a population and assume that one is taking a "random" sample.
The problem with the Carter hypothesis is that it is only valid if you know nothing about the likely future distribution of a population and therefore assume that your sample is a random selection.
So, if you have some valid argument or evidence regarding the likely future distribution of a population, then the simple Carter statistical argument no longer applies.

Let me clarify-- first N is selected by the gamemaster, and that selection is what has an unknown distribution for us. After N is chosen, then we get a random number from 1 to N. The latter is evenly distributed, so the whole game exactly mimics the Carter situation, but is more conducive to mathematical scrutiny.
There is no "strength" to the Carter hypothesis. It can only be one of two things depending on how far you take it: 1) the plainly obvious statement that 90% of beings live in the last 90% of any set from which they are generically chosen, and 2) the false probability argument that this says anything about the number of humans that will be born, given the number that have been. So the problem is not in the "random sampling", it is in the incorrect use of probability concepts-- and the mathematical game I described shows this, just take any interesting distribution over N that you like and ask if 50% of the times that a given specified M is chosen, will that M be > N/2. It will not-- unless you average over M, but then you can't use M to create the whole catastrophe concept.
It is a common misconception about probability that it has something to do with your knowledge. The choice to include knowledge is yours when you do a probability calculation, there is nothing "automatic" about it. All probability calculations are subject to the assumptions you put in-- knowledge is irrelevant, except that it is normally assumed that you will use all your knowledge in building your assumptions. When people think that probabilities depend on knowledge, they get all confused about under what situations do the probabilities change, like if you forget what cards have been shown in poker, does your probability of winning change? Answer: there is no unique concept of a probability of winning, it all depends on the calculation you choose to make.
It still applies, if you choose not to use any argument or evidence-- on the grounds that it would be of suspect reliability (which it would). The problem with Carter is much more fundamental-- it is wrong probability if you are selecting on the basis of the value of M.