Validity in Research Design

Conclusions drawn from analyzing survey data are only acceptable to the degree to which they are determined valid.  Validity is used to determine whether research measures what it intended to measure and to approximate the truthfulness of the results. Researchers often use their own definition when it comes to what is considered valid.  In quantitative research testing for validity and reliability is a given.  However some qualitative researchers have gone so far as to suggest that validity does not apply to their research even as they acknowledge the need for some qualifying checks or measures in their work.  This is wrong.   To disregard validity is to put the trustworthiness of your work in question and to call into question others confidence in its results.   Even when qualitative measures are used in research they need to be looked at using measures of reliability and validity in order to sustain the trustworthiness of the results.  Validity and reliability make the difference between “good” and “bad” research reports. Quality research depends on a commitment to testing and increasing the validity as well as the reliability of your research results.

Any research worth its weight is concerned with whether what is being measured is what is intended to be measured and considers the ways in which observations are influenced by the circumstances in which they are made.   The basis of how our conclusions are made play an important role in addressing the broader substantive issues of any given study. For this reason we are going to look at various validity types that have been formulated as a part of legitimate research methodology.

Face Validity

This is the least scientific method of validity as it is not quantified using statistical methods.  This is not validity in a technical sense of the term.  It is concerned with whether it seems like we measure what we claim.  Here we look at how valid a measure appears on the surface and make subjective judgments based off of that.  For example,  if you give a survey that appears to be valid to the respondent and the questions are selected because they look valid to the administer.   The administer asks a group of random people, untrained observers,  if the questions appear valid to them.  In research its never sufficient to rely on face judgments alone and more quantifiable methods of validity are necessary in order to draw acceptable conclusions.  There are many instruments of measurement to consider so face validity is useful in cases where you need to distinguish one approach over another.  Face validity should never be trusted on its own merits.

Content Validity

This is also a subjective measure but unlike face validity we ask whether the content of a measure covers the full domain of the content. If a researcher wanted to measure introversion they would have to first decide what constitutes a relevant domain of content for that trait.  This is considered a subjective form of measurement because it still relies on people’s perception for measuring constructs that would otherwise be difficult to measure.   Where it distinguishes itself is through its use of experts in the field or individuals belonging to a target population.  This study can be made more objective through the use of rigorous statistical tests.  For example you could have a content validity study that informs researchers how items used in a survey represent their content domain, how clear they are, and the extent to which they maintain the theoretical factor structure assessed by the factor analysis.

Construct Validity

A construct represents a collection of behaviors that are associated in a meaningful way to create an image or an idea invented for a research purpose.  Depression is a construct that represents a personality trait which manifests itself in behaviors such as over sleeping, loss of appetite, difficulty concentrating, etc.  The existence of a construct is manifest by observing the collection of related indicators.  Any one sign may be associated with several constructs.  A person with difficulty concentrating may have A.D.D. but not depression.  Construct validity is the degree to which inferences can be made from operationalizations(connecting concepts to observations) in your study to the constructs on which those operationalizations are based.  To establish construct validity you must first provide evidence that your data supports the theoretical structure.  You must also show that you control the operationalization of the construct, in other words, show that your theory has some correspondence with reality.

  • Convergent Validity - the degree to which an operation is similar to other operations it should theoretically be similar to.
  • Discriminative Validity - if a scale adequately differentiates itself or does not differentiate between groups that should differ or not differ based on theoretical reasons or previous research.
  • Nomological Network - representation of the constructs of interest in a study, their observable manifestations, and the interrelationships among and between these.  According to Cronbach and Meehl,  a nomological network has to be developed for a measure in order for it to have construct validity
  • Multitrait-Multimethod Matrix - six major considerations when examining Construct Validity according to Campbell and Fiske.  This includes evaluations of the convergent validity and discriminative validity.  The others are trait method unit, multi-method/trait, truley different methodology, and trait characteristics.

Internal Validity

This refers to the extent to which the independent variable can accurately be stated to produce the observed effect.  If the effect of the dependent variable is only due to the independent variable(s) then internal validity is achieved. This is the degree to which a result can be manipulated.

Statistical Conclusion Validity

A determination of whether a relationship or co-variation exists between cause and effect variables.   Requires ensuring adequate sampling procedures,  appropriate statistical tests, and reliable measurement procedures. This is the degree to which a conclusion is credible or believable.

External Validity

This refers to the extent to which the results of a study can be generalized beyond the sample. Which is to say that you can apply your findings to other people and settings.   Think of this as the degree to which a result can be generalized.

Criterion-Related Validity

Can alternately be referred to as Instrumental Validity. The accuracy of a measure is demonstrated by comparing it with a measure that has been demonstrated to be valid.  In other words, correlations with other measures that have known validity. For this to work you must know that the criterion has been measured well.  And be aware that appropriate criteria do not always exist.  What you are doing is checking the performance of your operationalization against a criteria.  The criteria you use as a standard of judgment accounts for the different approaches you would use:

  • Predictive Validity - operationalization’s ability to predict what it is theoretically able to predict.  The extent to which a measure predicts expected outcomes.
  • Concurrent Validity - operationalization’s ability to distinguish between groups it theoretically should be able to.  This is where a test correlates well with a measure that has been previously validated.

When we look at validity in survey data we are asking whether the data represents what we think it should represent.  We depend on the respondent’s mind set and attitude in order to give us valid data.  In other words we depend on them to answer all questions honestly and conscientiously.  We also depend on whether they are able to answer the questions that we ask.  When questions are asked that the respondent can not comprehend or understand then the data does not tell us what we think it does.

Enjoy this post?

Subscribe for future posts.

  • Pingback: Interpreting Questions | ActiveCampaign, Inc.

  • Pingback: What Is a Web Survey? | ActiveCampaign, Inc.

  • ashu

    I am not clear on the use of construct validity in your context. How can one proove that the data for research conforms to reality?
    How can do in order to establish construct validity and to provide evidence that your data supports the theoretical structure?

  • Habib

    Construct validity refers to validity of the ideas and concepts used in the research. Constructs are “concepts” and “ideas”. This particular validity is often problematic and not clear cut. There is a subjective (interpretive) element in it that makes it somewhat debatable.

  • http://google Obert

    Does the concept of internal validity (history, maturation, instrumentaility etc) apply in qualitative business research? If they dont what are the threats to internal validity in qualitative business research?

  • Mus

    Please I’m writing a thesis and want to develop a questionnaire form for a survey I want to conduct. In constructing my the questionnaire what should I include or avoid to ensure the questionnaire stands the test of validity?. The topic of the thesis borders on Challenges of internet use in Africa.

  • Irfan Malik

    this is the total theory about the validity i am interesting to know that what statistical tools are applicable and how they can be applied? and one thing that is missing in this article is lack of examples if you include the practical example with the theory it will make it very usefulnregards nIrfan MaliknPakistan

  • Drwho314159265358979

    Two things:

    1. I would suggest placing your definitions within some organizational structure to add clarity (check Wikipedia’s article on validity). For example, you could distinguish between validity of measurement (construct, content and criterion), and validity of experimental design (all the others, plus construct validity); also, you could subordinate them the way you did with criterion-related validity and its two types (so that face validity is shown to be a type of content validity, for example).

    2. Your definition of concurrent validity conflates the two subtypes of concurrent validity (discriminant and convergent); where discriminant validity compares scores for groups that do and do not have a characteristic, and convergent validity refers to the extent of correlation of related variables.

  • Drwho314159265358979

    “…some qualitative researchers have gone so far as to suggest that
    validity does not apply to their research even as they acknowledge the
    need for some qualifying checks or measures in their work.  This is
    wrong.   To disregard validity is to put the trustworthiness of your
    work in question and to call into question others confidence in its
    results.   Even when qualitative measures are used in research they need
    to be looked at using measures of reliability and validity in order to
    sustain the trustworthiness of the results.”

    By supplying this (reasonably accurate, if incomplete) list of different kinds of validity, you have not supported your argument that qualitative research must be shown to be both valid and reliable.

    “Any research worth its weight is concerned with whether what is being measured…””

    It is precisely because there are no measurements made in qualitative
    research that validity and reliability are not (in fact, cannot be) calculated or tested.

  • http://pulse.yahoo.com/_WP2PG4HO4HHDUKLNLSJ5SBTQ3Y atiye j

    i am writing a dissertation using stickers, how can i pre-test and validate it. 

  • Tonya Porter

    Qualitative disregards RELIABILITY not validity. It does not have to replicable because behaviour is often not replicable but it is valid to what the researcher has set out to measure and observe.

    • Drwho314159265358979

      Tanya Porter,

      You’re partially right…qualitative research is not concerned with reliability. However, many qualitative researchers reject the idea of validity being a requirement of qualitative inquiry, because they reject the assumption that there is an objective reality outside of our perception, and therefore, it makes no sense to be concerned with the “truth”
      or “falsehood” of an observation.

      Guba and Lincoln gave a set of four alternatives to validity, which can be used by qualitative researchers.  They are summarized on this page:

      http://www.socialresearchmethods.net/kb/qualval.php

  • Joe Maxwell

    Tariq’s post is another attempt to force qualitative research into a quantitative framework. However, rejecting his argument doesn’t require abandoning the concept of validity in qualitative research; it just means that validity needs to be defined in ways that fit what qualitative researchers actually do. See the entry on Validity in the Sage Encyclopedia of Qualitative Research (Sage Publications, 2008). I think most qualitative researchers accept that some conclusions are more credible, trustworthy, and useful than others, and there are a number of different approaches to assessing this that are currently in use.
    A currently prominent view is that validity is a property of conclusions, not methods or designs, because any method or design can result in valid conclusions in some situations and invalid ones in others. Methods are obviously important, but they don’t assure the validity of the results; the latter requires testing your conclusions against plausible alternative conclusions. I can’t give the full argument for this here; see the chapter on Understanding, Validity, and Evidence, in my book A Realist Approach for Qualitative Research (Sage, 2011).

  • arshdeep

    maxwell… i am a fan of your work in the feild of qualitative research. it is always enriching to read your work… :) my dissertation and research methodology classes have always been guided by your readings!!!

  • M_mrhunk

    An FMCG company’s marketing manager engaged the services of a research supplier to find out the consumers’ preferences of company’s products.  A research report was presented to marketing manager from research supplier after identifying the consumers’ preferences. The marketing manager rejected the report by saying that research conducted by your firm is not valid. In the reply, the research supplier’s representative said the instrument used in this research is valid. Marketing managers argued that using valid instrument could not assure the validity of research while representative of research supplier was persistent in defending the viewpoint that both the validity of an instrument and the validity of a research study are same.  Both individuals tried to convince each other but no consensus was reached.
     
    Please discuss the standpoint of which individual is correct in your opinion. If both types of claims are different or same then which types of validity validate the whole research and research instrument separately or collectively?

  • Pingback: Medicare Fee Schedule » looking for example of claim form for measure