"Parse Tree" <parsetree at hotmail.com> wrote in message
news:40B%8.10382$sb5.1038870 at news20.bellglobal.com...
> "John Knight" <johnknight at usa.com> wrote in message
> news:vkA%8.17597$Fq6.2105618 at news2.west.cox.net...> > "Parse Tree" <parsetree at hotmail.com> wrote in message news:q_0%8.8020
> > > >
> > > > A. gets selected 25,000 times.
> > > >
> > > > B. gets selected 25,000 times.
> > > >
> > > > C. gets selected 25,000 times.
> > > >
> > > > D. gets selected 25,000 times.
> > > >
> > > > Unless the correct answers are not distributed evenly across A., B.,
> C.
> > > and
> > > > D. (i.e., unless the test designers didn't assign the correct
answers
> > > > randomly), then there is NO other possible outcome (except for an
> > initial
> > > > minor variation from these figures which would eventually even out
> over
> > > > time).
> > >
> > > You're pretty ill informed. Firstly, when guessing, the correct
answer
> > has
> > > no bearing on what is guessed. Thus if the answer is really A, that
has
> > no
> > > effect on me guessing (since by guessing, we assume that they do not
> know
> > > the answer). Thus the distribution of the answers is irrelevant.
> >
> > You misunderstood the point. If the test designers intentionally loaded
> up
> > A. as the correct answer, this could skew the distribution of correct
> > answers. If you look carefully at the correct answers, they obviously
> > didn't do that, so this isn't a factor anyway.
>> You're talking about guesses anyway. The correct answer is irrelevant.
>> > > Additionally, while the expected value for A is 25000, that doesn't
mean
> > > that A will be picked 25000.
> >
> > bzzzzzt, you just flunked TIMSS. This is PRECISELY what American 12th
> > graders must have taught wrong that most other 12th graders seem to have
> > been taught CORRECTLY.
>> If my answer for such a simple statistics question differs from TIMSS,
then
> TIMSS is quite wrong.
>> > If the sample size is large enough, as TIMSS was, then this IS what will
> > inevitibly happen. There is NO other alternative.
>> This is only true when the sample size approaches infinity.
>> TIMSS could easily have 30% of people guess A.
This is either a poor choice of words, or you really don't know what you're
talking about. Even if you had only 85 students taking the test, which is
orders of magnitude fewer than participated in TIMSS, the probability that
"30% of people guess A" is 0.01, which is a far cry from "easily". With 170
students, the probability drops to 0.006. By the time you get to the sample
size of TIMSS, the probability is much less than 0.00006
>> > > > If the correct answer is B., and this answer is selected by 25,000
> > girls,
> > > > then you have zero evidence that they properly applied the theories
to
> > > > resolving the problem. If they selected this answer 25,750 times,
you
> > > still
> > > > have no evidence that they understood the principles, or could apply
> > them,
> > > > because such a score would be lower than the 3% standard error. If
> they
> > > > selected this answer 30,000 times, you are just barely higher than
the
> > > > combination of the 25% multiple choice guesses and the 3% standard
> > error,
> > > > which starts to make the score meaningful.
> > >
> > > What 3% standard error? There isn't some percentage that denotes a
> > standard
> > > error.
> >
> > The test developers provided an extremely clear definition and
calculation
> > for standard errors. If you disagree, why don't you talk to them?
>> The 3% error is only for this specific test. It isn't standard across all
> such things. Thus for your example above, 3% is not true.
>
True, but for some tests, it was greater than 3%, and for others it was
less. This is just an average, since we don't need pinpoint accuracy to
define the problem. The error is only used when the score is close to zero
(after correcting for guesses). It's used only to satisfy those who believe
the test is in error. They are correct that there IS a standard error, but
it's almost meaningless for most of the answers.
> > > There exists a probability that even while guessing, every single
person
> > > picks A. The probability is slim, but it still exists.
> > >
> > >
> >
> > Theoretically, yes, but again, each answer will be selected by 25% of
the
> > students, IF the sample size is large enough, IF the guesses are truly
> > random, and IF the test designers assigned equal weight to A., B., C.,
and
> > D. [read: the test designers evenly distributed the correct answers
across
> > A., B., C., and D].
>> The correct answers are irrelevant. We're talking about guesses here.
>> Also, the 25% thing is quite an approximation.
>
For 10,000 students, the furthest observed variations from the median of 25%
is 3% x 25%, or 0.75%, which means it will almost never be lower than 24.25%
and almost never higher than 25.75%. That's well within reasonable limits.
> > Even if you had only 170 students taking the test, the probability of
them
> > all selecting A. is 0.000000000000000000000576, which isn't even worth
> > mentioning (it's even smaller than the probability that ONE American
> female
> > teacher scores higher than the median score of her male students
majoring
> in
> > engineering, physics, chemistry, or even business). The probability
when
> > you get to thousands or tens of thousands of students are off the map.
>> But we're not only talking about that probability. We're talking about
the
> probability that each question does not have 25% of people guessing it.
>> That includes a question having 0 - 24% of people guessing it, as well as
> 26 -100% of people guessing it. These things add up to a noticeable
number.
>
Yes, they do, but the bottom line is that a completely random selection of
these questions will result in each answer being selected by 25% + or -
0.75% of the students.
The times that the answers fall out of those boundaries are very rare--and
wouldn't throw off the results by that much. iow, this could not possibly
begin to explain why ONE THIRD of the questions had scores *lower* than if
it had been a completely random selection.
> > So it is only with a very small sample size that it's possible to have a
> > scenario different than that above: 25% for each answer.
> >
> > TIMSS was not a small enough sample size to be able to make that
argument,
> > though.
>> TIMSS is still small enough that you cannot say that the probability a
> quarter will guess each question is 1.
>>
If 25% of the students select each of the four possible answers, it's
inevitible that the students guessed. It's impossible for one quarter of
the students to be taught that answer A) was correct, for another quarter
that B) was correct, etc.
John Knight
