"John Knight" <johnknight at usa.com> wrote in message
news:vkA%8.17597$Fq6.2105618 at news2.west.cox.net...
> "Parse Tree" <parsetree at hotmail.com> wrote in message news:q_0%8.8020
> > >
> > > A. gets selected 25,000 times.
> > >
> > > B. gets selected 25,000 times.
> > >
> > > C. gets selected 25,000 times.
> > >
> > > D. gets selected 25,000 times.
> > >
> > > Unless the correct answers are not distributed evenly across A., B.,
C.
> > and
> > > D. (i.e., unless the test designers didn't assign the correct answers
> > > randomly), then there is NO other possible outcome (except for an
> initial
> > > minor variation from these figures which would eventually even out
over
> > > time).
> >
> > You're pretty ill informed. Firstly, when guessing, the correct answer
> has
> > no bearing on what is guessed. Thus if the answer is really A, that has
> no
> > effect on me guessing (since by guessing, we assume that they do not
know
> > the answer). Thus the distribution of the answers is irrelevant.
>> You misunderstood the point. If the test designers intentionally loaded
up
> A. as the correct answer, this could skew the distribution of correct
> answers. If you look carefully at the correct answers, they obviously
> didn't do that, so this isn't a factor anyway.
You're talking about guesses anyway. The correct answer is irrelevant.
> > Additionally, while the expected value for A is 25000, that doesn't mean
> > that A will be picked 25000.
>> bzzzzzt, you just flunked TIMSS. This is PRECISELY what American 12th
> graders must have taught wrong that most other 12th graders seem to have
> been taught CORRECTLY.
If my answer for such a simple statistics question differs from TIMSS, then
TIMSS is quite wrong.
> If the sample size is large enough, as TIMSS was, then this IS what will
> inevitibly happen. There is NO other alternative.
This is only true when the sample size approaches infinity.
TIMSS could easily have 30% of people guess A.
> > > If the correct answer is B., and this answer is selected by 25,000
> girls,
> > > then you have zero evidence that they properly applied the theories to
> > > resolving the problem. If they selected this answer 25,750 times, you
> > still
> > > have no evidence that they understood the principles, or could apply
> them,
> > > because such a score would be lower than the 3% standard error. If
they
> > > selected this answer 30,000 times, you are just barely higher than the
> > > combination of the 25% multiple choice guesses and the 3% standard
> error,
> > > which starts to make the score meaningful.
> >
> > What 3% standard error? There isn't some percentage that denotes a
> standard
> > error.
>> The test developers provided an extremely clear definition and calculation
> for standard errors. If you disagree, why don't you talk to them?
The 3% error is only for this specific test. It isn't standard across all
such things. Thus for your example above, 3% is not true.
> > There exists a probability that even while guessing, every single person
> > picks A. The probability is slim, but it still exists.
> >
> >
>> Theoretically, yes, but again, each answer will be selected by 25% of the
> students, IF the sample size is large enough, IF the guesses are truly
> random, and IF the test designers assigned equal weight to A., B., C., and
> D. [read: the test designers evenly distributed the correct answers across
> A., B., C., and D].
The correct answers are irrelevant. We're talking about guesses here.
Also, the 25% thing is quite an approximation.
> Even if you had only 170 students taking the test, the probability of them
> all selecting A. is 0.000000000000000000000576, which isn't even worth
> mentioning (it's even smaller than the probability that ONE American
female
> teacher scores higher than the median score of her male students majoring
in
> engineering, physics, chemistry, or even business). The probability when
> you get to thousands or tens of thousands of students are off the map.
But we're not only talking about that probability. We're talking about the
probability that each question does not have 25% of people guessing it.
That includes a question having 0 - 24% of people guessing it, as well as
26 -100% of people guessing it. These things add up to a noticeable number.
> So it is only with a very small sample size that it's possible to have a
> scenario different than that above: 25% for each answer.
>> TIMSS was not a small enough sample size to be able to make that argument,
> though.
TIMSS is still small enough that you cannot say that the probability a
quarter will guess each question is 1.
