uh, in the end, i think i'm saying that yes, method 1 is flawed but also yes, they may be using method 2.

both concepts, i think, result in less than honest data usage. using the statistics you're attempting to uncover to rule out tree-rings is not much different than simply looking for your known single. particularly given that they know a-priori what type of correlation they're looking for.

taks

__________________
goodbye richard pryor :(...

Taks, your link in post 83 is incorrect, and I'll ignore post 84. Back to parts of post 82.
I don't notice any of them showing a divergence, a flattening, or a lack of correlation. Perhaps some studies do have that, but some of those clearly don't.
Hantemirov (2002) has e.g. 14 years with a lower than average temperature between 1907 and 1936, 15 between 1937 and 1966, and 11 between 1967 and 1996 (the last year in the set). For those three sets, I come at an average of 0.127 for 1907-1936, 0.132 for 1937-1966 and 0.138 for 1967-1996. But if I only take the period 1981-1996 (the "flattening"), I get 0.477.
When I look at Osborne/Briffa, the unprocessed data, I get (for the columns with recent enough data):
Column 2: 1901-20: -0.164; 1921-1940: 0.117; 1941-1960; 0.121; 1961-1980: -0.073; and 1981-1994: 0.264.
Column 5 (other base number): 1901-1920: 25.090 ; 1921-1940: 26.009; 1941-1960: 28.459; 1961-1980: 24.868; and 1981-1995: 22.836 (this one looks to be a reverse number, i.e. the highest number gives the coolest temperature).
Column 7: 1901-1920: 9.237; 1921-1940: 9.315; 1941-1960: 9.290; 1961-1980: 9.257; 1981-1995: 9.910
Column 10: 1901-1920: 1046.5; 1921-1940: 1428; 1941-1960: 1198.5; 1961-1980; 1334.05: 1981-1995: 1928.53
I haven't bothered calculating the others anymore
This are all fast calculations, so I hope I haven't made some error... But to me, from this very simple execrcise, this does not look like there is something about the data after 1981 which would make them better (for AGW) supporters to be left out of the studies. On the contrary, I would say...
According to the NSF, Mann e.a. had published all their data as they were supposed to do long before the Barton Committee made their demands. See e.g. this letter from the NSF in a reply to McIntyre.
For the way Moberg publishes his data, see e.g. this Word document. It looks to me that apart from two data sets which he received from the researchers by personal communication (and for which the names of the scientists involved are given), all his data are clearly given. I don't see a problem here (if the two "missing" datasets would make a huge difference to the result, that would be clear very fast for anyone trying to reconstruct his study).
As for Jones, I don't know exactly which of his papers you mean. Mann & Jones 2003 data is available via the link here, while Jones & Mann 2004 can be found here.
Jones (1998), probably the one you are talking about, can be seen here., and were there since december 1998, apparently. Here they are as text or Excel file. The same site has e.g. as well the Mann 1999 data and the Mann 1998 data.

__________________
Knowledge is a curse, but ignorance is worse

Fram and Tak- did you guys work for Stats, Inc.? The scientists who publish material use the best data they can find to promote their agenda. No need to see that as inappropraite as the art of persuasion is all science really has to support its costs. Which of you two are the "Real" defender of Trees, since my office shreds one at the rate of 1 per week (abuse of paper) a most unaccptable pace. That's what I want to know.

Emphasis mine. I don't think you'll find many people here who'll share such a blanket statement about the practices of scientists. Even though this kind of thing happens, it is not the rule, and when discovered is highly negative for the scientist involved. Scientists may look for a topic where there is interest (and grants) or which may have practical applications afterwards, but the study itself has to be done (and is normally done) objective, based on all relevant data, not on the data you like. If the result is not what you expected or hoped for, that's too bad, but that's how science works.

__________________
Knowledge is a curse, but ignorance is worse

What would be the agenda of climatologists, in your opinion? Putting themselves in the way of powerful, hostile corporate interests?

__________________
"A witty saying proves nothing" Voltaire.
"All your bias are belong to us" Ara Pacis.

which corporate interests would you be referring to, disinfo?

or, are you trying to smear skeptics by saying they are corporate shills?

reported.

taks

__________________
goodbye richard pryor :(...

i do not work for a statistics company. i do, however, use statistics with what i do: statistical signal processing. the methods employed in most of the proxy studies, principal components analysis, are also originally known as the karhunen-loeve transformations (or expansions). a search at the ieee (www.ieee.org) yields a few thousand hits on each (combined). standard fare in a detection theory class (page 180 of van trees "detection, estimation and modulation theory, part I", though he did not devote much more than a couple pages to it). such methods are seemingly used in the image processing/recognition field.

i do not specifically use it, but the method is similar to the things i have done: generate a correlation matrix across a set of vectors, determine the eigenvalues (basically a matrix inversion), weight the original data with said eigenvalues to remove noise.

taks

__________________
goodbye richard pryor :(...

they mostly stop at 1980. this was discussed at the NAS panel and is called the "divergence problem." post 1980 tree-rings no longer correlate with current temperatures.

no, sorry, but he did not publish all of his data. phil jones still has not, and you reference him just as often as mann.

which is the problem... who has been able to reconstruct his study?

posting some of your data is not sufficient. neither is posting a method that does not allow complete reproduction. this latter part, contrary to the AAAS letter, is particularly important when studies are either funded by the public or used to drive public policy.

and, again, i refer you to the open letter (posted in this thread) sent by dyson, et. al. regarding publication of data.

taks

__________________
goodbye richard pryor :(...

But method 1 doesn't involve selecting tree rings to match "the correlation they're looking for". method 1 involves selecting tree rings which match known temperature data, then using those same tree rings to extrapolate backward.

Perhaps method one is flawed (e.g., maybe these trees had fairly constant rain for the past century, and therefore matched the temp record, but didn't have consant rain previously, so the extrapolation is flawed.), but use of method one clearly isn't a case of cherrypicking data to match a theory. The data could still have contradicted the theory.

method one, by itself, uses the output to justify the input.

what you're looking for is a correlation with temperature, which you've assumed is rising. you then use this criteria as a selection for the input. a-priori knowledge of the expected output is biasing the data.

method one and method two are variations on the same theme. one is just openly disingenuous, the other is just bad statistical analysis.

if the correlation exists, it should exist across all trees barring, as i said, valid reasons for exclusion. the correlation (or lack thereof) is not a valid reason. and, to date, nobody has outlined any valid reason for exclusion other than the output itself. they make claims at realclimate, apparently, but not in the published work.

taks

__________________
goodbye richard pryor :(...

Did I say anything about skeptics? I don't think so.

If it's O.K. to suggest that global warming is just a huge hoax perpetrated by scientists for personal gain, then I don't see the problem with pointing out that astronomical personal gains are in fact unlikely for them.

__________________
"A witty saying proves nothing" Voltaire.
"All your bias are belong to us" Ara Pacis.

here's steve's discussion on the problems he's had with data archiving... based on everything else, i can only assume he is not lying.

taks

__________________
goodbye richard pryor :(...

Is that really what they did? The key difference between what you describe and what I posted is whether the short term correlation is with actual temperature data, or with assumed temperature data.

I still think it would be good science to
a) compare tree rings over X years versus actual temp data over those same X years, a
b) throw out trees which don't correlate,
c) use the remaining trees to estimate temps for Y years, where no actual temp data exists.

You seem to be claiming that Mann, et al, did not first compare tree rings to actual temp data, but instead compared tree rings to assumed temperature data. Is that really what occurred?

I trust the NSF and the AAAS more than I trust him. His correspondence with the NSF gives me no reason to think he is correct. But in the end, I'm also assuming, and giving precedence to the words of one group against the words of another.

__________________
Knowledge is a curse, but ignorance is worse

Taks, please don't go round in circles. I have given you a load of references to recent papers where the data are more recent than 1980, after your previous claim that most stopped there. Then you started about the divergence. I have shown you that as far as I can tell, there is no such divergence in the data. Now you go back to the supposed lack of data post 1980. Can you tell me what was not good about the counterevidence I gave?
I can't find no better info on the divergence than that some people claim that it exists. As far as I could tell, they were talking about a divergence between the tree rings and the troposphere temperature, not the surface based temperature. More (better) info on this is welcome.

I reference all kinds of studies, including Jones, Mann, Briffa, ... but also many others. I have shown were to find the data. I have given references to huge data set collections as well. I can find references that Jones and Mann both have refused to post their data to some people, but that does not mean that it isn't available, just that they don't want to help people that are clearly antagonistic and predisposed against them. I have given references from the NSF that they feel that Mann has given all the data that is needed. I have also given you the reason why they aren't oblged at all to give away their software. Furthermore, you don't need their method: use their data and use what you consider a good method. If the result is markedly different from their results, then it is the time to criticize the method.

About the open letter: the part starting "we understand" is rather vague in its source. And it doesn't mention Jones, Mann, or Moberg, which are the three you accused of not publishing their data. I wonder why these authors wouldn't refer the same high profile cases...

__________________
Knowledge is a curse, but ignorance is worse

(For those who are just joining us.) I love these exchanges. Wish I knew what Tak and Fram were talking about really. Can't tell which one is the true defender of trees though. Stay tuned for the climax. Both are very adept in their fields of expertise. I like both sides so far. Antarctica will be watching.

It's nice to know you don't care.

__________________
"A witty saying proves nothing" Voltaire.
"All your bias are belong to us" Ara Pacis.

I care for the trees. IMHO the source of changes come from beyond us: http://www.rense.com/general70/d3vet.htm no matter which set of data you prefer.

Just want to make clear that climate studies are not my area of expertise at all.

And I guess Taks is the true defender of trees, as he doesn't want to see those poor trees being killed for treering counts...
Just kidding!

__________________
Knowledge is a curse, but ignorance is worse

most, fram, most. i did not say all. please don't try to misrepresent this.

then why did d'arrigo bring it up at the NAS panel?

no kidding. it's a pretty embarrasing point. at least some are willing to stick their necks out and attempt an explanation.

really, now who would that be? is that your argument regarding M&M? they are antagonistic? how many times do mann and jones get to call them shills or absurd before you see them as antagonistic? M&M have put forth very professional correspondence and analysis, which has to date resulted in several publications, yet they may be antagonistic based on your view?

just because the NSF and mann "feel" their efforts suffice does not make it so. full disclosure should not require congressional force. it should also clearly follow the rules set out by the magazines they publish in (science and nature should be ashamed for allowing this).

and you are wrong. your reasoning is flawed. they have a duty to provide the proper means for reproducibility. reproducibility is required for falsifiability. it exists in nearly every other science except this one.

if their method is so well explained, why is it nearly impossible to replicate their results? where is their data exclusion (tree-rings) policy published?

no, but i've shown you exactly where serious scientists performing serious analysis are continually blocked. that they eventually release some of their data and methods is not sufficient.

taks

__________________
goodbye richard pryor :(...

mine either.

yes, it is true, i'm a closet environmentalist.

taks

__________________
goodbye richard pryor :(...

either he is lying or he is not. he continually publishes all of his correspondence while mann, et. al. only publish that which make them look good. they have not yet been able to get a corrigendum issued against M&M, yet he has had two issued against the hockey team, both times w.r.t., among other things, data availability.

sorry, but the notion that full disclosure is not required, for any reason, is ridiculous.

i should also point out that if they are using proprietary methods, they need to file for a patent. the K-L methodology is not patentable, but they modify it for their own purposes. nor are general wavelet filtering algorithms (ala moberg). if it is not patentable, then what is the beef? particularly since this stuff is being used to drive public policy.

taks

__________________
goodbye richard pryor :(...

barring any supporting documentation, yes, that is what they do (esper's comments, linked to earlier, also flatly state they use outcome to determine inclusion).

throwing out those which don't correlate still puts the cart before the horse. the hypothesis is warming climate AND correlation with tree-rings. using anything BUT known biological (et. al.) causes for exclusion is circular.

no, i didn't say that. i said, paraphrased, they choose only those tree rings which do correlate. most are now looking like the correlation stops around the 1980s.

the problem is this. say you find a tree-ring that does not correlate and you throw it out. answer these two questions: 1) why did it not correlate and 2) if we cannot find a true biological bias, does the relationship truly exist?

to date, they have issued some comments regarding causes over at realclimate, but those comments aren't in the real papers, perhaps for a reason... since we cannot know what reason they tossed certain tree-rings without a proper analysis, the reason "it must be biased" is, by itself, introducing bias to the results.

even the link fram provided regarding using tree rings to determine temperature does not provide any results. they're using statistics to show the correlation, so citing them is, again, putting the cart before the horse. how can you just a-priori decide to use tree-rings correlated against known temperatures when you don't even know a cause-effect relationship exists?

i should also point out, when you take the tree-rings out of most of these studies, the hockey stick suddenly looks a lot less like a hockey stick.

taks

__________________
goodbye richard pryor :(...

oh, finally, once amman and wahl actually get their recent admission of a failed r2 statistic published, the entire tree-ring argument may become moot. they'll have a much harder time convincing anyone their claims have merit. mann originally claimed skill with r2 (among others). the r2 allows them to sort of benchmark their results against out of calibration periods. it provides a measure of how likely the results are from mere chance. david stockwell's reproduction, using noise instead of proxies, verifies that random chance can produce the same results (in support of a poor r2).

taks

__________________
goodbye richard pryor :(...

??? You said most, I said you said most. What is this about "trying to misrepresent you"? What, exactly, in my statement here was trying to misrepresent you? Don't make such accusations lightly, Taks, especially when you are the one not providing any evidence at all. I have shown that there are a lot of recent (i.e. since 2000 or so) studies with data post 1980. You still have to show one which doesn't include it, let alone show that most don't.
Just like you still need to show how your QED statement that restarted this "debate" is in any way "demonstrated" and not just claimed.
Well, because he noticed how some blogs like climateaudit make a big deal of it, while
HE brought it up to show that it wasn't really a big problem at all.
Well, of course, if the NSF, the AAAS, Science, and Nature, are all wrong, then there is nothing much left, is there. "Congressional force" is very relative, as my quote of another congressman (quoting the NSF) in a previous post showed. One committee was influenced by M&M, another not. I don't think the Congress has taken a position. I furthermore don't think the Congress knows how to judge this situation, and I rather stick to the experts (the NSF, the AAAS, and Science and Nature) here. If they are happy with what is done and how it is done, then it is good enough for me, and I'll continue using their data until new scientific research in relevant peer reviewed journals has corrected them. For now, the majority of that kind of research is clearly in line with them.
Ammann et al have done just that, a replication of their results, and find that they are basically correct.

__________________
Knowledge is a curse, but ignorance is worse

Care to back up this statement? I have already shown papers which use stalagmites, ice cores, ... and have basically the same result as tree ring or multi proxy studies (for the latter you still have to show how you know the tree rings dominate them as well).

__________________
Knowledge is a curse, but ignorance is worse

Fram: 2 (for data refs and for multi proxy defense) Taks: 2 (for reclaiming hockey stick to the NHL and for environmentalist credo: tree don't lie.) My humble appraisal of argument so far. Did I miss something?

that's the entire argument that has been made all along by M&M, fram. i don't know how many times i have to point to their work. they clearly showed removing even a few of the tree-ring series and the results changed significantly.

stalagmites that you've pointed to are a) single proxy and b) clearly state that they measure CO2, which is known to be rising.

ice cores don't show temperature, they show CO2 only.

taks

__________________
goodbye richard pryor :(...

oh, here's d'arrigo's paper on the "divergence problem."

taks

__________________
goodbye richard pryor :(...

from MBH98:
so, where, then, is this linear relationship proved? also, since there is seemingly no proof, how can anyone justify the results when they clearly state the part i've bolded above?

this is the point i was trying to make with pghnative. without a clear exclusion policy based on proven biases, the claim that tree-rings accurately represent temperature is false.

also, you can show all the multi-proxy studies you want fram, but their methodology clearly weights the tree-rings heavier than the other proxies. keep in mind, when we see the "pretty graphs" in all these studies, they are merely the output of the algorithms. the algorithms take the proxies and filter out the noise, weighting some more than others. this is why the tree-rings dominate, and this is the crux of the M&M argument (among other things).

amman and wahl attempted to criticize this but both attempts over at GRL got rejected (known as the "little whopper" at climateaudit). they also reference the LW 11 times in their criticism that was "provisionally accepted" by Climatic Change... we're curious to know if this recent rejection will mean anything.

still, fram, you are unable to explain away the failed r2, btw.
taks

__________________
goodbye richard pryor :(...