Distribution of abilities

From the Abstract of the paper The best and the rest: revisiting the norm of normality of individual performance, Ernest O’Boyle Jr. and Herman Aguinis:

We revisit a long-held assumption in human resource management, organizational behavior, and industrial and organizational psychology that individual performance follows a Gaussian (normal) distribution. We conducted 5 studies involving 198 samples including 633,263 researchers,
entertainers, politicians, and amateur and professional athletes. Results are remarkably consistent across industries, types of jobs, types of performance measures, and time frames and indicate that individual performance is not normally distributed—instead, it follows a Paretian (power law) distribution. Assuming normality of individual
performance can lead to misspecified theories and misleading practices. Thus, our results have implications for all theories and applications that directly or indirectly address the performance of individual workers including performance measurement and management, utility
analysis in preemployment testing and training and development, personnel selection, leadership, and the prediction of performance, among others.

 

I am all in marking exams just now and am not able to look into the paper carefully.

Some thoughts though.

The general thesis of non-normality is fine but it is like claiming that the Earth orbits the Sun.

Calibrated measures, such as IQ and many psychological tests, almost by definition are normal in populations (in fact, in the population for which they are developed). There are underlying assumptions, of course, but the point is that they are taylored to be such. Also, exam performance may be calibrated to look like normal by appropriate choice of the set of questions and allocation of marks.

Looking at Study 1, for example, it is ridiculous to talk about normal distribution. Firstly, before looking at the data we know that they are non-negative and generally with small values. The methodology of selecting “leading” journals” makes the numbers even smaller. Before critisising I should think more but a selective procedure invalidates basic assumptions about validity of normal approximation.

The histograms shown towards the end of Study 1 clearly show this. These histograms should have been put at the beginning of the study. The mean and the standard deviation are almost meaningless for this kind of data (counts, maximum at low values). If anything, the histograms suggest starting with exponential or Gamma, and discard the normal outright. Pareto is fine, as well.

Also, using \(\chi^2\) to evaluate the quality of the fit is primitive.

By the way, the fact that Pareto is better than normal does not show that it is any good. Any of the distributions mentioned above will be better than normal.

As it happens, I recently a second year Assignment for Practical Statistics, where students do this kind of thing (fitting exponential distributions, evaluating the fit). They would not have got good marks for using \(\chi^2\) test. QQ-plots and Kolmogorov Smirnov type tests are much better.

Students certainly do not get good marks if they simply fit a distribution and do not evaluate the quality of the fit.

Draft Mathematics Curriculum

On 23 May 2012 Department for Education published Draft Programme of Study for Primary Mathematics. From the official announcement:

The Secretary of State has written to Tim Oates, the Chair of the Expert Panel, with his response to the panel’s recommendations for the primary curriculum. The Secretary of State has also confirmed that he will write again to the panel about the secondary curriculum in due course. You can view a copy of the letter from Michael Gove to Tim Oates regarding the National Curriculum update. Draft Programme of Study for […] mathematics has also been published. These drafts are a starting point for discussion with key stakeholders at this stage, but there will be a full public consultation on revised drafts which will start towards the end of this year.

This blog could be a natural place to start an in-depth discussion of the new curriculum. The following (independently developed) draft curriculum could be useful for such a discussion:

A. D. Gardiner, A draft school mathematics curriculum for all written from a humane mathematical perspective: Key Stages 1–4, The De Morgan Journal, 2 no. 3 (2012),  pp. 1–138.

Abstract: This draft was hammered out by a small group, which included experienced school teachers, textbook authors, curriculum administrators, and mathematicians. In particular, many helpful suggestions from Tony Barnard, Richard Browne, Rosemary Emanuel, and David Rayner have contributed to the current version. It offers a mathematician’s-eye-view of school mathematics to age 16, which we hope will serve as a useful focus for wider discussion and debate.

Comments are most welcome and should be sent to

Anthony.D.Gardiner >>>at<<< gmail.com

Alternatively, leave a comment at this post.

Preparedness of new undergraduates for degree level study

At Cambridge Assessment, which is a department of the University of Cambridge and the parent organisation of three awarding bodies, researchers have been exploring the perceived preparedness of new undergraduates for degree level study.

Three strands of research have been completed:

  • A questionnaire survey of 633 university lecturers (including 179 mathematics lecturers) on the impacts of qualifications for 16 to 19 year olds on higher education
  • Focus groups on lecturers’ views about the effectiveness of curricula for 16 to 19 year olds as preparation for university
  • A literature review on the pedagogical differences between A Level and university.

The research focuses primarily on mathematics, biology, and English. Both qualitative and quantitative methods were used as part of a ‘mixed methods’ approach.

Summaries of the research are available at:

http://www.cambridgeassessment.org.uk/ca/Viewpoints/Viewpoint?id=139723

Full reports can be requested from the same website.

There are several key findings relating to mathematics:

  • There is a healthy appetite among mathematics lecturers for engagement in research exploring the transition from A level to higher education.
  • Over half of mathematics lecturers think that mathematics undergraduates are underprepared for degree level study.
  • ICT, teamwork, intellectual curiosity are the skills and attributes likely to be considered strengths of typical mathematics undergraduates when they begin degree level study.
  • Most lecturers think that academic writing, self-directed study, independent inquiry and research, and critical thinking skills are weaknesses of typical undergraduates when they begin degree level study. Depth of subject knowledge is also a concern for most mathematics lecturers.
  • Mathematics, further mathematics, and physics are the A level subjects considered by mathematics lecturers to provide the best preparation for a mathematics degree.
  • Biology, chemistry and mathematics are the A level subjects considered by biology lecturers to provide the best preparation for a biology degree.
  • More generally, history, English and mathematics are the A level subjects considered to provide the best preparation for degree level study by lecturers across a wide range of subjects.
  • According to almost 60% of mathematics lecturers, their institutions provide additional support classes for underprepared 1st year undergraduates.
  • Over 60% of mathematics lecturers have had to adapt their teaching approaches to teach underprepared 1st year undergraduates.

The research received considerable media attention at the start of April, when emerging findings were presented at a UCAS conference in Birmingham. The presentation coincided with an exchange of letters between the Secretary of State for Education and the national regulator, Ofqual, setting out a new policy in which Higher Education is to have more influence on the development of future A levels.

The three strands of research form part of a wider research programme which extends over several years. This work is an important means of restoring and strengthening links between qualifications developers and HE.

Ofqual: Comparison of A Levels with International Qualifications

Report published by Ofqual purports to address, among others, the following issues:

Issue 3: Different levels of demand within mathematics – The number of different mathematics assessments at a variety of levels available to students in many education systems was also in contrast to A level Mathematics. Is there a need for A level Mathematics to have further lower-level options in addition to AS?

Issue 4: Breadth versus depth within mathematics – Within the more challenging mathematics courses considered, A level Mathematics is unusual in covering both pure mathematics and the application of mathematics in the same course. While this means that more fields within mathematics are available to study, other education systems include more demanding mathematics which an A level student can only access through additional A level courses. Would a more focused A level mathematics course better serve the needs of more capable mathematicians?

Issue 5: Specialism within mathematics – A level Mathematics includes optional routes. This means students with the same grade in the qualification may not be equally well prepared for a specific further course of study. Would distinct qualifications, building on a mathematical core but emphasising the different specialisms, better serve students and those seeking to match them to appropriate further opportunities?

Pdf file of the report: Comparison of A Levels with International Qualifications.

A report on the pilot of the linked pair of GCSEs in mathematics

AlphaPlus Consultancy prepared for DfE The independent evaluation of the pilot of the linked pair of GCSEs in mathematics (MLP): Second Interim Report. This is the third of seven formative evaluation reports on the pilot of the linked pair of GCSEs in mathematics (MLP). A final summative evaluation report will be presented in December 2013.

A quote from p.6:

Problem solving and functionality are central to mathematics at Key Stage 4 (KS4). The previous reports on the MLP [Mathematics Linked Pairs] have identified the lack of a shared understanding by centres of what problem solving and functionality mean in relation to mathematics teaching and learning generally and in particular in relation to the revised assessment objectives (AOs) for GCSE mathematics. The fact that stakeholders have no common definition for these terms across the range of instances and contexts in which they use them, such as the two MLP qualifications, is problematic. An absence of clear definitions might lead stakeholders to fail to recognise and understand the different types of problem solving which the structure of the MLP promotes. The two previous MLP reports indicated that both effective teaching and assessment of problem solving and functionality are still in relatively early stages of their development. This is not an issue specific to the MLP: centres offering the MLP together with the single GCSE, awarding organisations and wider stakeholders all suggest that the issues regarding the teaching of problem solving are also evident for the single GCSE in mathematics.

Ofqual – Standards Review

Ofqual published Review of Standards in GCSE Mathematics 2004 and 2008. Principal findings:

  • The major change that affected all GCSE mathematics examinations between 2004 and 2008 was a move from a three-tier examination system of foundation, intermediate and higher tiers to a two-tier system, comprising foundation and higher only. These changes had a significant effect on the demand of the examination by changing the balance of questions focused on each grade.
  • The spread of grades to be covered in each tier increased and in some awarding organisations this resulted in a rise of structuring within questions. In addition question design showed an increasing trend towards structuring of questions. Both factors made examinations less demanding over time.
  • The increasing numbers of centres entering students for specifications with modular examinations highlighted a mixed effect on demand. OCR’s modular assessment design minimised the effect of the changes and allowed standards to be maintained over time, whereas AQA’s modular design (also available in 2004) fragmented the assessment and increased structuring in questions, making the examinations less demanding.
  • The layout of question papers, the language used and the clarity of graphs and diagrams had all improved over the time period reviewed, providing a better quality assessment in mathematics.