Now that I've gotten your attention, no, this thread isn't about some recently published paper that supports astrology. It is, however, about a test that we're doing right here on the BABB. In a thread discussing astrology, gzhpcu made a statement that he's made before, that he investigated a particular type of astrology based on Jung's work, and based on the descriptions of people it provided, felt like there was something to it. He's acknowledged that that isn't exactly a scientifically supportable claim, but still feels that the personalist descriptions fit remarkably well. For myself, I think it's more likely a case of preferentially noticing the parts of such a description that match, and neglecting the parts that don't match as well, but I think we can all agree that judging how well a single description fits someone is a pretty subjective matter.

Therefore, it seemed to me that the best way to explore the issue would be to simply test whether the descriptions created match better than descriptions for someone else's birth date, so I suggested such a test here, and gzhpcu graciously agreed to do the work for such a reasonable test.

So, here's the way it works. I'm acting as a moderator for the test, and while I fully expect a null result, I'll be acting strictly as a neutral party in conducting it. My actual interaction with the participants will be minimal. The participants will send me, via private message, their birth details, including time and place of birth. Both of these should be as accurate as possible. Time of birth should be at worst within five minutes, and within a minute would be better. Place of birth should be to the arcminute or better in both latitude and longitude (I'm willing to do the looking up if you just have the name of a hospital, for example). I will randomly assign each participant to a group of four (if there happen to be two people with extremely close birth times and places, I'll make certain that they are in different groups, though). All of the birth information will be passed on to gzhpcu, without names attached. He's agreed to do a natal chart for each person, and then providing a description for that person. Each participant will then receive all four of the descriptions for the people in their group, in random order, and will then choose the description that they feel fits them best. Once everyone has made that determination, we'll see how many choose the one that was actually based on their own birth information, and I'll post the full results here.

One would expect that, just by chance, one in four people will choose the description that was actually created for them. If, however, gzhpcu's technique actually creates descriptions that match people better than average, we should see a significantly larger number of matches. I'm hoping for a total of twenty participants, to have at least a moderately reasonable sample size. So that would mean an expected five matches. Six or seven would be hard to justify as anything other than a statistical fluke, while eight or nine might be arguable (my personal recommendation would be for further tests, though that would clearly involve quite a bit of work for gzhpcu). If there were ten or more matches (half the participants), I'd acknowledge that such a result seems unlikely to be mere chance.

So, how can you participate? So far, I've had a total of four people respond with their birth details, and two more say they'd be happy to participate, though they haven't yet forwarded the information. You're welcome to participate, whether you think astrology is valid or not. The requirements are: send me a private message with your birth time and place, as specified above; once you receive a set of personality descriptions, make an honest effort to choose the one which you feel best suits you; don't make any effort to discuss the specific details with myself or gzhpcu in any way during the test (for example, don't send a private message telling gzhpcu your birthday, and suggesting that he slip the word "asparagus" into the description so you can recognize it ). I'll likewise make certain not to provide any outside information that might give a hint as to which description goes with which person, and of course gzhpcu won't actually know who the participants are until the test is complete, so he won't be able to base anything on his knowledge of people from their posts here.

If anyone has recommendations for improvements to this protocol, please let me know, and I'll do what I can to accomodate them, if everyone agrees that they seem reasonable changes.

Oh, and if we can, I'd like to keep this thread limited to discussing this test, details thereof, significance of the results, once they're in, and so forth. If you want to discuss astrology in general, and its good or bad points, I'd suggest posting them in the thread I linked to at the beginning of this post. Since this is purely an empirical study, I'd prefer not to get bogged down in a discussion of what the proposed mechanism for astrological influence might be, or anything of that nature. We can discuss whether there needs to be a mechanism after we see if there's any phenomenon that needs to be explained!

What about a double-blind test of astrology?

__________________
papageno


"Why waste time learning, when ignorance is instantaneous?" - Hobbes (Calvin and Hobbes)

"It's all about context!" - Vince Noir (The Mighty Boosh)

"I've never heard of such a brutal and shocking injustice that I cared so little about!" - Zapp Brannigan (Futurama)

Looks like a pretty reasonable study. Of course, those results weren't necessarily using the particular astrological technique that gzhpcu espouses, so he could, if he wished, claim that not everyone who claims to be an astrologer is using valid methods (in fact, I'm fairly certain that he does claim that). They had a much larger set of participants, which would be nice, but I don't think it would be fair to ask gzhpcu to do quite that many.

I was thinking that the communication via the web actually helped establish the equivalent of a double-blind study. That is, if I were sitting across the table from the participants while they read over the descriptions and made their choices, I'd feel compelled to hand that task over to someone who didn't know which description was the one for each person's birth date, to avoid providing any unconscious clues one way or the other. However, since we'll be communicating strictly via mail, I'm confident that I can provide the four descriptions to each participant in an unbiased manner.

By the way, I mentioned that they would be presented to each participant in random order. I will determine the order randomly, but I think each group of four will see them in the same order. That way, if there's any selection bias (say, people tend to be more likely to pick the first one they read), it should cancel out. One person might be slightly more likely to pick the right one because of the order, but the others would have an equal additional likelihood to pick the wrong one. I don't think such selection biases would be that strong anyway, but it seems best to minimize such effects where possible.

You should also establish ahead of time what results you would need in order to say that there is a positive correlation.

__________________
"We can't test an elixer of immortality! It'd take forever!"

That's a good plan. I obviously gave a rough idea above, but that was certainly a pretty casual estimate. A good place to start would probably be to work out the odds of getting various numbers purely by chance, or actually use some of the typical statistical methods for determining correlation. I can do that, but if there are any statistical wizards out there that would like to take a crack at it, I'd be happy to have your advice.

Grey,

I PM'd you my info. However I have a question.

Most Horoscopes are so vague that they can really loosly apply to everyone that reads them. How are we going to be sure these descriptions are going to be specific enough? What I am getting at is that you are battling two seperate issues here. First, your attempting to see if a reading can specifically desribe a person, but the other obsticle you have to overcome is how the person choosing the description views themselves.

IE:
Reading 1 - Your a great Guy!
Reading 2- Your a jerk
Reading 3 - You're happy
Reading 4 you're sad

Bob - Hey #1 is mine, I'm a great guy
Rest of the world - Bob's a sad jerk

__________________
The More I learn, The More Ignorant I Become

Eagle Eye Observatory

What about testees (edit for clarification: the volunteers, the data providers -- that are in a way being tested, too) who might want gzhpcu-brand astrology to fail this test? Won't they choose the description that fits them worst?

__________________
0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0....

You're worried you might have to kick some biased testees?

It's true that there will be the possible bias that how someone views themselves may not be how everyone else views them. In a private message to someone who volunteered to participate, I mentioned that ideally, you'd have a team of psychiatrists extensively interview each participant and come up with a personality description, and then you'd have a second group of psychiatrists try to match the psychiatric descriptions to the astrological ones. That said, even a reading that is more like what the participant thinks they're like than anyone else woould be significant, I think.

Remember, though, that these exact same four readings will also be given to the other three people in the group (the people for whom readings 2 through 4 were created). If Bob correctly chooses reading 1 as his own because it sounds more flattering than the others, Sue, Carol, and Fred will all tend to do likewise, so they'll all get it wrong, and we'd get 1 out of 4, exactly what we'd expect randomly. We only see better than random if Bob is more likely to pick reading 1 than the other readings, but Sue is more likely to pick reading 2 (say), Carol is more likely to pick reading 3, and Fred is more likely to pick reading 4. Does that make sense?

Sort of, but then gzhpcu has an easy post-facto objection to the test when it fails to support his belief: the astrological descriptions, though correct (hah), might not have been selected by the people described.

Is there evidence out there that these volunteers, say, who read several science-based desciptions of personalities will accurately identify their own?

__________________
0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0....

That's the one way I can see to bias the test. I've specifically asked all participants to do their best to choose the one that actually describes them best, and I'm forced to trust the honesty of the board members, but given the population fo the board, I think that's actually a reasonably safe bet. If this were GLP, I'd want to make sure the controls were a bit more rigorous. From the list of participants so far (and there are about a dozen at this point!), I think there are both some that are probably sympathetic to astrology, and others that are skeptical, so that's nice.

Note, though, that there's something interesting about the strategy you describe. If you think astrology is nonsense, than you should expect that whichever one describes you best would be more or less random, and you'd expect that playing honestly should result in a null result for the experiment. For you to think that choosing a description that fits you poorly will actually make the overall results worse, you'd have to think that the one that describes you best actually is probably the one prepared using your birth information. If you don't think that gzhpcu's descriptions are accurate, then you should expect that it won't make the slightest difference which one you choose. That is, for someone to think the strategy you describe was viable, they would have to both believe that gzhpcu's methodology works, and want to sabotage the experiment for some reason. I think it's safe to assume that to be a fairly unlikely attitude.

That is in fact true. However, gzhpcu has mentioned several times that one of the things that led him to think there was "something to this" was that the descriptions he created were often well accepted by the participant (there was also his own assessment that the description fit the subject well). If, on average, a typical group of people can't actually select their own description from a group of others, I do think that's reasonable evidence that the tendency of people to approve of the readings he provided was simply one of noticing the matches and ignoring the misses.

I can't think of an easy one offhand. Perhaps there's a respectable online personality test someone could recommend to me via private mail? If the volunteers were amenable, I could forward the questions, the volunteers could provide me with the answers, and I could then give those results to each group of four, and see if the correlation is higher. That's sounding like significantly more work for the participants, though. If anyone has any other recommendations, I'm open to them.

Would you like for some of us to participate? Also, what's GLP?

__________________
here we go, the world is spinning
when it stops, it's just beginning

sun comes up, we laugh and we cry
sun goes down, and then we all die

Touched by his noodly appendage

I don't. I am most worried about the excuses after the test yields negative results. Yes, it would be remarkable if the volunteers select their astrological descriptions. But it will mean nothing if the volunteers do not select their description -- because people just don't or because people actively tried not to. You had better eliminate the possibility of excuses before the test.

(By the way, if (just some of) the volunteers want to make sure the test fails -- whether they believe in astrology or not, all they have to due is make their selection randomly. Choosing the worst description -- if they were accurate -- would yield a negative correlation, which would be significant.)

__________________
0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0 0 1 1 0 1 0 0 1 0 1 1 0 1 0 0 1 1 0 0 1 0 1 1 0....

I don't know enough about astrology really to know whether this is a good suggestion or not, but anyway.

I can think of two possible sources of bias here. One is the possibility that people will deliberately lie. But I think that's not really that significant a problem -- presumably those who volunteer to participate are actually interested in the results.

The more problematic thing, I think, is the danger that people don't really understand their own personalities. It may be that all four people in a group will subconsciously choose the personality that they would like to have rather than the one they really have.

The only way I can think of to overcome these things is to provide information that is not so subjective. For example, you are left handed, you have such-and-such color hair, your father was a firefighter, whatever. If the descriptions are things like, you are stubborn, then it's really hard to know whether this applies to oneself or not. Is it possible for the readings to have that kind of specificity?

__________________
As above, so below

If the scoring is a simple matter of yes, someone selected the correct description, or no, someone did not select the correct description (i.e., there is no partial credit for choosing a description which is similar to the correct one), then it is not so difficult to calculate the probability distribution of the number of correct scores under an assumption that the whole thing is just so much guff. But by frequentist statistical methods (which are what most statisticians use), the result is only in one direction, i.e., the test may "prove" with a pre-determined level of confidence that the results are not due to random chance, but it cannot "prove" that the results are due to chance. So I don't think a test like this can actually disprove astrology, it can only fail to prove it. If the result is a failure to prove, then your two possible interpretations are (a) it is all guff, or (b) there is something to it but we are not able to show this convincingly with the data in hand.

Of course, that's just the statistics part. If people deliberately try to sabotage the test, e.g., that's a whole other kettle of fish. But if there is no validity to astrology (in case anyone is wondering, that's my personal opinion), then sabotage will not affect the outcome. If there is no relationship between the people and personality descriptions, then the statistical test should still fail to find a relationship except for X% of the time (where X is chosen in advance), even in the presence of sabotage. However, if there is a relationship (i.e., astrology works), then sabotage could prevent the statistical test from finding it.

Also, I just did a quick calculation, and with 20 participants and a 25% success rate just by random chance, I think you can be 95% sure that a result in favor of astrology has been found when you receive ten correct answers. There are certain approximations involved here which tend to overstate slightly the critical value for a given level of confidence. The exact probability of each number of correct answers by random chance can be calculated exactly for a small sample, but I'm too tired to do it right now. I did this very quickly, it's late at night, as I just said, I'm tired, and I don't do this sort of calculation every day, so I wouldn't accept my figure of ten as gospel until it is confirmed, by me or someone else.

To make things clear: I will just go through the work of interpreting the natal horoscopes, but will certainly not nit-pick or look for excuses if the test turns out negative as is expected by the majority here.

I am also not the typical astrologer. My impression is that there is something to it. Should the test come out negative, then my conclusion will be that somehow I have fallen into a trap of seeing something which is not there. Just that simple. Otherwise, my old statement of "there might be something to it" would still apply.