"Parse Tree" <parsetree at hotmail.com> wrote in message news:q_0%8.8020
> >
> > A. gets selected 25,000 times.
> >
> > B. gets selected 25,000 times.
> >
> > C. gets selected 25,000 times.
> >
> > D. gets selected 25,000 times.
> >
> > Unless the correct answers are not distributed evenly across A., B., C.
> and
> > D. (i.e., unless the test designers didn't assign the correct answers
> > randomly), then there is NO other possible outcome (except for an
initial
> > minor variation from these figures which would eventually even out over
> > time).
>> You're pretty ill informed. Firstly, when guessing, the correct answer
has
> no bearing on what is guessed. Thus if the answer is really A, that has
no
> effect on me guessing (since by guessing, we assume that they do not know
> the answer). Thus the distribution of the answers is irrelevant.
>
You misunderstood the point. If the test designers intentionally loaded up
A. as the correct answer, this could skew the distribution of correct
answers. If you look carefully at the correct answers, they obviously
didn't do that, so this isn't a factor anyway.
> Additionally, while the expected value for A is 25000, that doesn't mean
> that A will be picked 25000.
>
bzzzzzt, you just flunked TIMSS. This is PRECISELY what American 12th
graders must have taught wrong that most other 12th graders seem to have
been taught CORRECTLY.
If the sample size is large enough, as TIMSS was, then this IS what will
inevitibly happen. There is NO other alternative.
>>> > If the correct answer is B., and this answer is selected by 25,000
girls,
> > then you have zero evidence that they properly applied the theories to
> > resolving the problem. If they selected this answer 25,750 times, you
> still
> > have no evidence that they understood the principles, or could apply
them,
> > because such a score would be lower than the 3% standard error. If they
> > selected this answer 30,000 times, you are just barely higher than the
> > combination of the 25% multiple choice guesses and the 3% standard
error,
> > which starts to make the score meaningful.
>> What 3% standard error? There isn't some percentage that denotes a
standard
> error.
>
The test developers provided an extremely clear definition and calculation
for standard errors. If you disagree, why don't you talk to them?
> There exists a probability that even while guessing, every single person
> picks A. The probability is slim, but it still exists.
>>
Theoretically, yes, but again, each answer will be selected by 25% of the
students, IF the sample size is large enough, IF the guesses are truly
random, and IF the test designers assigned equal weight to A., B., C., and
D. [read: the test designers evenly distributed the correct answers across
A., B., C., and D].
Even if you had only 170 students taking the test, the probability of them
all selecting A. is 0.000000000000000000000576, which isn't even worth
mentioning (it's even smaller than the probability that ONE American female
teacher scores higher than the median score of her male students majoring in
engineering, physics, chemistry, or even business). The probability when
you get to thousands or tens of thousands of students are off the map.
So it is only with a very small sample size that it's possible to have a
scenario different than that above: 25% for each answer.
TIMSS was not a small enough sample size to be able to make that argument,
though.
John Knight
