A survey is a method of gathering information from a number of individuals, known as a sample, in order to learn something about the larger population from which the sample is drawn.  Although surveys come in many forms, and serve a variety of purposes, they do share certain characteristics.  In order for the objectives of a survey to be met, the results must reliably project on the larger public, from which the sample is drawn.  A sample can be scientifically chosen so that each individual in a population has a known chance of selection.  This ensures that a sample is not selected haphazardly or uses only those eager to participate.  The sample size for a survey will depend on the degree of reliability necessary and how the results are to be used.  A properly selected sample should be able to reflect the various characteristics of a total population within a very small margin for error.  There are many surveys that study the total adult population but many others that focus on selected populations:  employees, academics, industry experts, computers users, or customers that use a particular product or service.

Some surveys focus on opinion and attitudes while others are more concerned with collecting factual information. Many surveys combine questions of both types.   A respondent could be asked what they heard or read about an issue, what they know about it, their opinion, how strongly they feel and why, interest in the issue, past experiences with it, and also certain demographic information which will help the survey analyst classify the responses (such as age, sex, marital status, occupation, and place of residence).  Questions can be open ended (”What does that make you think of?”) or closed (”Do you agree or disagree?”); they may ask the respondent to rate a product or a service on some kind of scale; they may ask for a ranking of various alternatives.  The questionnaire could be very brief – a few questions taking no longer than five minutes, or it could take a demanding hour or more of a respondent’s time.  A survey is usually rooted in situations where an individual or institution is confronted with an information need and no existing data will suffice. Once the information need has been identified and a determination made that existing data is inadequate, objectives are laid out for the investigation. These objectives should remain as specific, clear cut and unambiguous as possible.

Designing the questionnaire is a critical stage of the survey development process.  The questionnaire links the information need to the realized measurement.  Scaling techniques used for measurement can be comparative or non comparative.  If proper care and attention is not placed on clearly defining concepts and unambiguously phrasing questions, the resulting data is apt to contain serious biases.  Questions used in surveys must be refined to minimize interpretation problems and thus reduce measurement error.  If the respondent is unable to understand a question or fails to comprehend the question the way it was intended then the data is neither valid nor reliable.  A question must use language in a way that makes the intended observer’s meaning behind that question obvious.   In order to do this a survey must reflect an understanding of the population being sampled. For example, many people do not distinguish between robbery which requires the theft to be in the immediate presence of the victim and burglary which can involve breaking and entering without any confrontation.  The National Crime Survey, done by the Bureau of the Census, does not even mention the word “robbery” when it asks questions about robbery victimization.  Rather they ask several questions that use universally understood phrases, consistent with the operational definition of robbery, that when used together are able to capture the desired responses.

Keeping response errors and biases to a minimum factor heavily in designing a survey. How questions are interpreted by respondents must be carefully considered.  Also, you need to consider the length of the survey. If a questionnaire is too long than it can be burdensome to the respondent, inducing respondent fatigue that leads to response errors, refusals, incomplete questionnaires, and can contribute to higher non-response rates in subsequent surveys involving the same respondent.  Memory plays an important role when surveys deal with past events.  For most people the greater the demand a question places on memory, such as being asked to recall trivial details occurring on any given random date, the less accurate the responses and therefore the less reliable the survey data that is collected.  An appropriate choice of reference period should be made so that a respondent is not forced to report on events that happened too long ago.  Grouping similar questions together can reduce the cognitive burden a survey places on a respondent.  The validity of a survey can be put in jeopardy if questions are too sensitive, if they may prejudice the respondent, if they unduly invade a respondent’s privacy, and if the information that is sought is too difficult for a willing respondent to provide.

A web survey is one of five general ways research is typically conducted using surveys.  Surveys are also given using mail (the kind that requires postage), telephone, in-person interviews, and intercept surveys (respondents are “intercepted” as they pass through a heavily trafficked area).  The web survey most closely resembles the type of surveys which were traditionally mailed out.  Only this time you’re using email to send a link that takes you to an HTML form that is designed for completion and submission through the computer.   Information is collected quickly because as soon as your respondent has finished taking the survey, their responses are immediately submitted.  Web surveys can also be embedded on your web site using what is known as an Iframe.  An IFrame is an HTML element that makes it possible to embed an HTML document inside of another HTML document.   You can also create a Pop-Up survey on your web site so that when a site visitor visits a page the HTML form pops up from their web site.   If they have pop-up blockers enabled then this method will prevent them from taking your survey so you can also create a link to the survey which you can display on your web site.

The web survey is a cost effective way of administering a survey that allows you to collect large amounts of information without having to pay for interviewers, paper supplies or postage, and does not require separate data entry for responses to be processed.  The rise in web based surveys is due in no small measure to the increasingly widespread availability of computers.   Particularly in organizational or professional settings, the ability to receive a questionnaire and complete it at home or in the office on a computer is very convenient for most people.  You can expect to wait at least a few weeks for a questionnaire that is mailed out to a respondent to be returned.  A web survey allows you to rapidly collect data in a timely manner.   Information can be collected and processed in just a few days.    It also allows respondents ample time to carefully consider response selection and to enter in text for open ended questions.  If factual information is required then the respondent has enough time to consult their records.

You can increase response rate to web surveys by sending respondents a pre-notification of the intent of the survey.   This lets your respondents know what the survey is about before they invest any of their time into taking it.  You can also send follow-up reminders to respondents that have not completed the survey. This can be used in a situation where a respondent must complete the survey within a given date range.  Web surveys assume a minimal level of computer literacy so keeping your surveys simple can also help your response rate.  The technical nature of online surveys make them ideal for specialized or well defined populations that have access to an email account and/or a computer.  Coverage bias may result when targeting populations where computers or internet access are not widespread.  Determining an appropriate type of survey to administer to your respondents must take the population that is being measured into consideration.

Survey questions often ask respondents for autobiographical information.  The accuracy of responses to these questions is dependent on the respondent’s ability to recall memories.  Unless you have a condition known as hyperthymestic syndrome you do not have a superior autobiographical memory.  Neither do your survey respondents.  In fact there have only been three confirmed cases of individuals with hyperthymestic syndrome,  a name given by researchers at University of California-Irvine who used the greek for excessive (hyper) and remembering (thymesis).  One of the subjects researched went by the initials AJ until she used her real name, Jill Price, in a book published last May titled “The Woman Who Can’t Forget: The Extraordinary Story of Living with the Most Remarkable Memory Known to Science – A Memoir.”  She was originally the subject of a study published in the Journal Neurocase in 2006 that led to the discovery of two other cases: a Wisconsin man named Brad Williams and Rick Barron from Ohio.  What makes them extraordinary is their ability to recall specific events from their personal experience and the abnormal amount of time they spend thinking about it.  When you ask them about a random date they can describe events that occurred on that day,  tell you what the weather was like, and other trivial details that most people would not be able to recall.   For most people the greater the demand a question places on memory, such as being asked to recall trivial details occurring on any given random date, the less accurate the responses and therefore the less reliable the survey data that is collected.  Because recall is not reliable, respondents rely on inferences to fill in the blanks for any details that they are not able to recall.  Understanding the cognitive processes involved with memory in survey response can help you improve the reliability of your surveys.

In order to appreciate how remarkable an accurate memory is then we must first understand what goes into forming and retrieving a memory.  Memory is usually understood in general terms as an individual’s mental ability to store, retain, and recall information.  Memories are stored in nerve network patterns.  Each nerve cell in the brain connects with a thousand other nerve cells.  Memories are retained on the basis of whether these connections are strong or weak. When we learn to do something better such as play a musical instrument then these connections become stronger.  Information that comes from a respondent becomes more reliable when a respondent is exposed to the same information repeatedly.  When we learn to ignore something such as a the constant humming coming from a train near an apartment then these connections become weaker. Likewise the information that a survey respondent is likely to remember is not going to come from experiences they have consciously sought to ignore.  A nerve cell is also referred to as a neuron.  Memories are recalled when we activate a network of interconnected neurons.  Information comes to us through our sense of sight, smell, taste, hearing, and touch.  When we recall a memory we re-fire the same neural paths that we used to sense the original experience and in a way we recreate the event.  A question may ask about an event that was experienced by a respondent but it could also ask about a concept or an idea.  When memories come from concepts or ideas we extract the essence of sensed experience to form generalized concepts.  This means that actual events and the how they are perceived by the respondent’s senses have a physical basis even when questions are only relevant to a particular concept or idea.

In the movie “50 First Dates” Adam Sandler must win Drew Barrymore’s heart each day as if it were the first time they met because she suffers from memory loss as the result of a brain injury.  She is no longer able to encode, store, and retrieve new memories.  Every morning when she wakes up with no recollection of the previous day.  She is able to recall previously stored memories from before she suffered an injury but she is no longer able to create new memories based on recent events.  Short term memory allows you to remember recent events, experiences, and information while long term memory stores information so that it can be used again and again.  In order for Drew Barrymore to create new memories she must store information in a usable form in a process known as encoding.  Memory must then be stored for later use.  Much of these stored memories lie outside of our awareness until we need it. She must then be able bring the stored memories into conscious awareness.  No matter how hard Adam Sandler tried he could not get Drew Barrymore to remember him the next day because the brain injury she suffered did not allow memories to ever develop to the point where they were able to be encoded, stored, so they could be later retrieved.  When a question is asked in a survey it is assumed that any reference to past events are in a place where they can be brought into conscious awareness.  The stage model of memory holds that memory occurs in three stages: sensory memory, short term memory, and long term memory.

During the sensory memory stage we hold onto sensory information for fractions of a second before some of the data is moved over to short term memory. Only the data that catches the respondent’s attention is moved to short term memory.  Data that does not catch their attention is lost for good.  Questions that are not based on issues that are likely to attract a respondent’s attention initially may not have any basis for reference.  Short term memory is also known as working memory.   When a survey question is too long, contains too many embedded clauses or complex syntax, it can impose a cognitive burden on the respondent’s working memory.  Every respondent has a different level of working memory.   Working memory is generally assumed to be around 20-30 seconds.  Many short term memories are forgotten.  This is because short term memory is small. It can hold about 7 items at one time.  However when information is attended to then it can often be brought to the attention of working memory.

Remember when we discussed hyperthymestic syndrome.  One of the criteria established for this was an excessive amount of time thinking about past experiences.  Memories are therefore present in working memory even when they concern seemingly trivial events.  Unless your respondent is unusually obsessed with the question you are asking then this is not always going to be the case.  Particularly when you ask them a question that they may have not thought about for a while.  The accuracy of survey responses decreases over time.  If a question is relevant to a respondent then it will be easier for them to retrieve this information into working memory.  Therefore the questions that you ask your respondent are likely to increase in accuracy in relation to relevancy.  When memory passes into long term memory it exists on a preconsious or unconscious level.   While this information exists largely outside of our level of awareness it can still be brought into working memory fairly easily.  Long term memories can be used to make decisions, interact with others, and solve problems.  The processes involved with long term memory are the same processes that Drew Barrymore could no longer accomplish in “50 First Dates.”   The three processes involved are encoding, storing, and retrieval.

Finally, memories are arranged in clusters in which related information is grouped together so that it can then be parsed more easily.  When information is organized into categories it becomes easier to remember and therefore recall into working memory.  You might associate a coffee shop with hours spent studying, reading, and the intelligent or funny conversations you had while you were there. Over time similar events you experienced while at that coffee shop begin to get grouped together.   As time passes you are no longer able to separate what you think you experienced at the coffee shop from what you infer happened.  If someone asked you what you did at a coffee shop you have not been at for awhile then all your memories will blend in with other memories.  It might seem like you were either reading something or talking with someone you usually went there with.  However, certain events while you were at the coffee shop may trigger memories that distinguish one experience from another.  The time you spilled coffee on your laptop for example.  Including cues in survey questions can allow respondents to distinguish one event from another and improve the accuracy of their survey responses.  Another way to improve memory in surveys is by asking questions that surround a target event in order to bring your respondent back to the event’s original memory encoding. For example, you may be able to recall what you were doing at the coffee shop better if you are reminded that you slept through an exam the next day.  Understanding how memory is used to recall events can improve the accuracy of your survey by separating events to the extent that you are able to distinguish them.

People ask questions in order to learn what they wouldn’t otherwise know.  When a respondent answers a question on a survey they must rely on their comprehension to interpret what that question means.  Then during a process of memory retrieval they access relevant information for content to be included in their answer.  Judgment is used for comparing and evaluating ideas in a way that allows information to be synthesized.  During response selection an answer is selected and placed in the required response format. All stages of the cognitive process just described rely on the first stage which is question interpretation. For this reason questions used in surveys must be refined to minimize interpretation problems and thus reduce measurement error.  If the respondent is unable to understand a question or fails to comprehend the question the way it was intended then the data is neither valid nor reliable.  A question must use language in a way that makes the intended observer’s meaning behind that question obvious.   In order to do this a survey must reflect an understanding of the population being sampled. Before administering a survey you need to ask yourself whether your questions measure what you intend them to measure.  Is it valid?   Do your respondents interpret your questions the same way.  Is it reliable?  In order to ensure that the respondent is answering the question we think they are answering it’s important to make questions clear and precise.

Level of Wording

Question wording can affect the answers that are received.  The wording used in questions should be appropriate to the level of education and characteristics of the respondents.  A researcher must be aware of the respondent’s understanding of difficult words, colloquialisms, jargon, technical terms, and language proficiency when settling on an appropriate level of wording.  They also must be aware of words or phrases that are gender or ethnic sensitive.  If the language used is not the first language of all your respondents then the level of language proficiency of all your respondents must be carefully considered.   Level of wording is important because a respondent that is embarrassed to admit that they do not know something is likely to give any answer rather than admit they do not know.  You may also see a lot of neutral response choices such as  “do not know” or “no opinion.” Also, it can lead to a high rate of refusal to complete survey rate.   In general you should use wording that is simple, direct, and to the point.  A question that would not be appropriate to the level of wording of your average survey respondent could be:

“Did the attorney deliberately obfuscate the truth by inundating the jury with statements that were verbose and declamatory?”

An understanding of the population you are administering the survey to must be worked out in advance.   If you are administering a survey to attorneys you would use a different level of wording than you would if you were asking the same questions to a group of layman.

Ambiguous Questions

If there is more than one way to interpret a question than that question is ambiguous.  When a question is vague it has an imprecise range of application.  If a question uses vague wording  or is ambiguous it can lead respondents to interpret a question in a variety of ways.  An example of an ambiguous question might be:

“Who do you buy groceries for?”

It is not clear whether either you or any household pets are included in this question.   Is this asking who you usually buy groceries for?  Does this include anything you might pick up for your parents that you no longer live with?  A better way to ask this question would be:

“What is the total number of household occupants that you purchased groceries for in the past year?”

Ambiguity can also arise if a respondent is unclear about what type of question is being asked, the motive for the question being asked, the purpose for a question being asked, and when inferring motives behind a question that have no truth in reality.

Confusing Questions

A question is confusing if it leads to a feeling of uncertainty in the respondent when the intent or meaning is not clear.  Here is an example:

“Does it seem possible or impossible that the Cubs will win the World Series, or do you feel they will always find a way to lose?”

1_ It seems possible

2_ It seems impossible

Here the respondent may feel trapped into either agreeing that the Cubs will find a way to lose or saying that it is impossible for the Cubs to win the World Series.   A clearer way to ask this question would be:

“Does it seem possible that the Cubs will ever have a chance to win the World Series?”

1_It is possible that it could happen.

2_It is not possible that could happen.

Working Memory Overload

In order for a respondent to process a question it can not impose a cognitive burden on their working memory.   Working memory is short term memory used to hold information while the respondent is comprehending a question.  This can often happen when a question attempts to be too precise, by injecting complex syntax and embedded clauses,  in an attempt to clearly define a question.   Earlier we talked about vagueness and ambiguity.   Couldn’t you make our earlier question more precise by rewording it as:

“Including yourself, any pets you own, but not including any relatives or friends that do not live in your household for at least 320 days a year, how many household occupants did you buy groceries, by groceries we mean any item of food bought at the grocery store, for in the last year?”

The problem with asking questions like this is that you are going to tax your respondents memory and some of them will not even get what you are trying to say.  Individuals differ in how much their working memory can handle.  Questions that pose a burden to some may not pose a burden to others.  Avoid complex syntax in your survey questions.  This means that its grammatical composition can not be dense, structurally ambiguous, or not well formed syntactically.   Memory overload can occur with extremely long sentences, using too many logical operators (such as or, and, if-else, then), and quantifiers.    In order to make responses more precise it is better to ask a series of simpler questions.  The two major consequences of overworking memory are that items drop out of working memory or cognitive processing can slow down.  The respondent may simply drop some of the intended meaning of the question and they could spend a lot of time coming up with an incomplete interpretation of the question.

Double Barreled Questions

When a question introduces two or more issues with the expectation of a single response it is a double barreled question.   This inadvertently confuses the respondent and limits the useful data that you can collect.  The respondent is forced to answer two issues with a single response which they may agree with and disagree with respectively as in the following question:

“Is the big city a great place to experience culture and raise a family?”

A person may very well feel that a big city is a great place to experience culture because of its diversity but that doesn’t mean that they want to raise their family there for a variety of reasons.

Leading Questions

A presupposition is anything that is assumed to exist or be true and every question has a number of nouns or propositions that are presumed to be true.   Questions that contain a false presupposition are known as leading questions.   Under certain circumstances false presuppositions can cause respondents to make incorrect inferences about what occurred and to misremember events.  Imagine a cop asking the following question:

“At what point did you quit speeding when you saw the truck swerving onto oncoming traffic?”

Pretend you weren’t speeding.  There is evidence to suggest that asking leading questions like this can cause respondents to misremember what really happened.

Loaded Questions

In order to obtain accurate results a survey question should be neutral in order to welcome as many points of views as possible.  A loaded question contains an incriminating assumption if the respondent accepts it to be true.   An example of this can be found here:

“Do you believe that we should redistribute wealth by allowing tax breaks to expire?”

The loaded word here is “redistribute” as it is often associated with socialism.   The question is asked in such a way that a respondent may not feel comfortable giving an honest answer.   Injecting emotional language in surveys can bias survey results.

Pretesting

If you have a clear understanding of each survey question’s intent you can correct flaws in the survey instrument at the outset by pretesting your questions.  Common methods used in pretesting include expert reviews, forms appraisal, cognitive interviews, and the use of focus groups.

Keep the focus of each question on the specific attribute or phenomenon that you are measuring.  Collecting quality responses is highly dependent on the clarity of questions.   Keep your syntax simple so that your respondent is not easily confused.  Although you want your question to be precise you need to avoid using long, dependent clauses or compound, complex sentences as they can overwork your respondent’s memory and cause them to answer questions incorrectly.  Just as you should not ask a question that is relevant to the research you are conducting you should not use a word that is not relevant to the question you are asking.  Keep it brief. The longer a question is the more likely the respondent is to misinterpret it.  Focus on asking one question at a time.  Brevity can not come at the expense of the intended meaning of the question.  You are simply being succinct or in other words choosing the shortest way to pose a question.  Pretesting can allow you to further refine your questions so that the meaning you intend is not lost on your respondent.

When I think of reliability I imagine always knowing what to expect.  If a person is able to produce the same quality work consistently then they are considered reliable.  You see it in sports all the time. Certain players have a knack for coming through in key situations no matter how late in the season or how worn down they are.   However, I can imagine few jobs that require more reliability than a surgeon.  Having an off day for them could prove disastrous. For a measure to be reliable it must demonstrate consistency as well as repeatability.  When carrying out research our results should be accurate across a range of measurements.  In surveys you would like to think that you would get the same response no matter what mood your respondent is in but that is not always the case.   A surgeon must deal with difficult situations while showing the same precision and reliability.  That is a quality to be admired but you can not always expect everyone to act like a surgeon at all times.  It’s also possible that your respondent won’t know what you mean when you ask them a certain question resulting in an answer that is entirely different from what you are attempting to measure.

Test-retest Reliability

If your survey respondent had to take your survey again would they answer the same questions the same way?  Test-retest reliability measures reliability over time.  A number of factors can affect reliability over time such as a person’s mood, time of the day, where the questions are placed in the survey (context effect), circumstantial events, vagueness, etc.  A good test will take into account factors that may influence survey results over time and minimize them so that results show little variation. If a test is unreliable then any one of a number of factors can lead to varying results depending on when the question is asked.  In general the more time a person takes between retesting the more variation you can expect in the responses.

If you ask Joe Q what he thinks about Candidate X on Tuesday he may view him favorably because X gave a really good speech on Monday.   Say Candidate X is indited later in the week in a corruption scandal.  Joe previously indicated that a candidate’s integrity is very important.  Last week he said that he was leaning toward Candidate X.  Now that Candidate X has been exposed you may think he is likely to give you a different response if you asked Joe the same question next Tuesday.   The reliability of opinion polls can be doubtful depending on the questions we ask because opinions tend to fluctuate over time.  What does Joe Q mean when he rates integrity as very important?  Perhaps Joe Q considers anyone that shares his ideology to have integrity.  Its possible that Joe Q would vote for Candidate X no matter what he thought of him personally because they share the same ideology.  Probing Joe’s past voting record would be more indicative of voter preference than asking a subjective question about integrity.  Asking him more objective questions that would not fluctuate from week to week would have higher test-retest reliability.

Parallel Forms Reliability

Another challenge reliability faces is in knowing what the best questions to ask are.  What does Joe Q mean when he rates integrity as very important?  Could we come up with better questions to predict how voters like Joe Q would vote? Another way to improve the reliability of a survey is to ensure that it is representative of the data you are trying to collect.   To do this increase the sample size.  If you are gathering research to find out whether voters like Joe Q are likely to vote for Candidate X then you need to find more people like Joe and ask them different questions or question sets based on the same construct.

You come up with a large set of questions to ask in your survey.  The construct that you are measuring is voter preference. The large question set is split in half and you administer each set to half of the targeted population.  You can then take a look at which questions are better indicators of voter preference. This combines what is known as a split test method with parallel form evaluation.

You can use parallel forms to measure a construct for people that are not like Joe Q.  Here you would divide a population that is representative of all likely voters in two.  Develop a large question set that measures a particular construct and then administer to each half of your representative population.  Now you can learn which questions are better indicators for voter preference for a representative population.

Inter-rater Reliability

This is necessary if you are conducting your survey using an interview process.   If  multiple people are interviewing Joe Q to ask what his opinion on politics is then inter-rater reliability measures the degree to which the observers agree.  This is the best way to measure reliability if you are using observation for your research.

Internal Consistency Reliability

The purpose of asking questions in surveys is to assess a particular construct or idea.  Therefore different questions that measure the same construct should yield similar results.  Reliability is determined on the basis of whether results are consistent for different items that measure the same construct.  For example, you could check for reliability on your survey by asking a respondent two similar questions meant to measure the same thing.

• Average Inter-Item Correlation – when we ask a respondent two similar questions to measure the same construct.  This compares correlations between this and any other paired questions to measure the same construct by calculating the mean of all paired comparisons.
• Average Itemtotal Correlation – where you take the average inter-item correlation and calculate a total score for each item.
• Split-half Correlation – you divide items that measure the same construct into two tests,  apply them to the same group of people, and calculate the correlation between the two scores.
• Cronbach’s Alpha – when we calculate the average split half estimates from a sample population.

In order to draw conclusions, formulate theories, or make generalizations about your research you need to ensure the reliability of the data you collect.  In general reliability is threatened when assessments are taken over time, rely on different standards of judgment, or assessments are highly subjective.  You can improve reliability by ensuring that your surveys are written clearly and without ambiguity.  You should construct your response options so that they are appropriate and meaningful.

Conclusions drawn from analyzing survey data are only acceptable to the degree to which they are determined valid.  Validity is used to determine whether research measures what it intended to measure and to approximate the truthfulness of the results. Researchers often use their own definition when it comes to what is considered valid.  In quantitative research testing for validity and reliability is a given.  However some qualitative researchers have gone so far as to suggest that validity does not apply to their research even as they acknowledge the need for some qualifying checks or measures in their work.  This is wrong.   To disregard validity is to put the trustworthiness of your work in question and to call into question others confidence in its results.   Even when qualitative measures are used in research they need to be looked at using measures of reliability and validity in order to sustain the trustworthiness of the results.  Validity and reliability make the difference between “good” and “bad” research reports. Quality research depends on a commitment to testing and increasing the validity as well as the reliability of your research results.

Any research worth its weight is concerned with whether what is being measured is what is intended to be measured and considers the ways in which observations are influenced by the circumstances in which they are made.   The basis of how our conclusions are made play an important role in addressing the broader substantive issues of any given study. For this reason we are going to look at various validity types that have been formulated as a part of legitimate research methodology.

Face Validity

This is the least scientific method of validity as it is not quantified using statistical methods.  This is not validity in a technical sense of the term.  It is concerned with whether it seems like we measure what we claim.  Here we look at how valid a measure appears on the surface and make subjective judgments based off of that.  For example,  if you give a survey that appears to be valid to the respondent and the questions are selected because they look valid to the administer.   The administer asks a group of random people, untrained observers,  if the questions appear valid to them.  In research its never sufficient to rely on face judgments alone and more quantifiable methods of validity are necessary in order to draw acceptable conclusions.  There are many instruments of measurement to consider so face validity is useful in cases where you need to distinguish one approach over another.  Face validity should never be trusted on its own merits.

Content Validity

This is also a subjective measure but unlike face validity we ask whether the content of a measure covers the full domain of the content. If a researcher wanted to measure introversion they would have to first decide what constitutes a relevant domain of content for that trait.  This is considered a subjective form of measurement because it still relies on people’s perception for measuring constructs that would otherwise be difficult to measure.   Where it distinguishes itself is through its use of experts in the field or individuals belonging to a target population.  This study can be made more objective through the use of rigorous statistical tests.  For example you could have a content validity study that informs researchers how items used in a survey represent their content domain, how clear they are, and the extent to which they maintain the theoretical factor structure assessed by the factor analysis.

Construct Validity

A construct represents a collection of behaviors that are associated in a meaningful way to create an image or an idea invented for a research purpose.  Depression is a construct that represents a personality trait which manifests itself in behaviors such as over sleeping, loss of appetite, difficulty concentrating, etc.  The existence of a construct is manifest by observing the collection of related indicators.  Any one sign may be associated with several constructs.  A person with difficulty concentrating may have A.D.D. but not depression.  Construct validity is the degree to which inferences can be made from operationalizations(connecting concepts to observations) in your study to the constructs on which those operationalizations are based.  To establish construct validity you must first provide evidence that your data supports the theoretical structure.  You must also show that you control the operationalization of the construct, in other words, show that your theory has some correspondence with reality.

• Convergent Validity - the degree to which an operation is similar to other operations it should theoretically be similar to.
• Discriminative Validity - if a scale adequately differentiates itself or does not differentiate between groups that should differ or not differ based on theoretical reasons or previous research.
• Nomological Network - representation of the constructs of interest in a study, their observable manifestations, and the interrelationships among and between these.  According to Cronbach and Meehl,  a nomological network has to be developed for a measure in order for it to have construct validity
• Multitrait-Multimethod Matrix - six major considerations when examining Construct Validity according to Campbell and Fiske.  This includes evaluations of the convergent validity and discriminative validity.  The others are trait method unit, multi-method/trait, truley different methodology, and trait characteristics.

Internal Validity

This refers to the extent to which the independent variable can accurately be stated to produce the observed effect.  If the effect of the dependent variable is only due to the independent variable(s) then internal validity is achieved. This is the degree to which a result can be manipulated.

Statistical Conclusion Validity

A determination of whether a relationship or co-variation exists between cause and effect variables.   Requires ensuring adequate sampling procedures,  appropriate statistical tests, and reliable measurement procedures. This is the degree to which a conclusion is credible or believable.

External Validity

This refers to the extent to which the results of a study can be generalized beyond the sample. Which is to say that you can apply your findings to other people and settings.   Think of this as the degree to which a result can be generalized.

Criterion-Related Validity

Can alternately be referred to as Instrumental Validity. The accuracy of a measure is demonstrated by comparing it with a measure that has been demonstrated to be valid.  In other words, correlations with other measures that have known validity. For this to work you must know that the criterion has been measured well.  And be aware that appropriate criteria do not always exist.  What you are doing is checking the performance of your operationalization against a criteria.  The criteria you use as a standard of judgment accounts for the different approaches you would use:

• Predictive Validity - operationalization’s ability to predict what it is theoretically able to predict.  The extent to which a measure predicts expected outcomes.
• Concurrent Validity - operationalization’s ability to distinguish between groups it theoretically should be able to.  This is where a test correlates well with a measure that has been previously validated.

When we look at validity in survey data we are asking whether the data represents what we think it should represent.  We depend on the respondent’s mind set and attitude in order to give us valid data.  In other words we depend on them to answer all questions honestly and conscientiously.  We also depend on whether they are able to answer the questions that we ask.  When questions are asked that the respondent can not comprehend or understand then the data does not tell us what we think it does.

A non comparative scale can also be variously referred to as a monadic or metric scale.  Respondents evaluate only one object at a time and the resulting data set is either interval or ratio scale. An example of this would be if you would take an object such as an iPod and ask each respondent the degree to which they viewed the iPod favorably.  This technique includes continuous rate scales and itemized rate scales such as Likert, Semantic Differential, and Stapel Scale.  This type of scale is favored in market research due to the fact that more can be done with it.  These scales can be used when characteristics of physical objects are being compared but they are most often used when attitudes, perceptions, or other psychological characteristics are being measured.

Thurstone Scaling

The three methods described for this are equal appearing interval method, successive intervals method, and paired comparisons method.  This was the first formal technique for measuring attitudes when it was introduced and it does this by asking respondents to indicate whether they agree or disagree with each of a large set of statements about attitudes.   This scale calculates a weight or attitudinal value for each statement.  This weight is calculated on the basis of a rating assigned by a group of judges.  Because the importance of each statement is determined by judges it reflects the absolute rather than relative attitudes of judges.  The scale indicates the intensity of attitudes and changes in this intensity can be measured should this study be reconstructed.  This is not often used today because it is expensive and time consuming.  A major criticism is that judges and respondents may assess the importance of statements differently and thus the respondent’s attitudes may not be properly reflected.

Likert Scaling

This ordered one dimensional scale gives equal weight or attitudinal value to items or statements when reflecting an attitude toward an issue in a question.  Generally done using a five point or seven point scale. Respondents are typically asked to select from choices ranging from “strongly agree” to “strongly disagree.”  This is an extremely popular means of measuring attitudes.  Because it is the most commonly used scale for responses to survey questions it can be argued that it is the most commonly misused.  Choices are assumed to have equal distance between them and can even be numbered.  This is often referred to as a summative scale as this is often scored by summing up numerical assignments to the responses given.  In scoring, numbers are usually assigned to each response in a question (for example 1 to 5).

Continuous Rate Scale

Alternately referred to as a Graphic Rating Scale,  respondents rate items by placing a mark on a line that is usually labelled with numbers (for example 1 to 100).  The form this scale takes may vary.

Phrase Completion Scale

This was developed in response to problems associated with Likert Scale.   These are concise unidimensional measures that taps ordinal data in a way that approximates interval leval data.   This consists of a phrase followed by an 11 point response key in which 0 represents the absence of a theoretical construct and 10 represents the theorized maximum amount of the construct being measured.  Just as is the case with Likert Scale this is considered a summative scale where after the questionnaire is completed the score on each item is summed together.

Semantic Differential Scale

This is a five or seven point itemized ordinal scale with dichotomous pairs of descriptive words or phrases representing two extreme point of views and a neutral response.  The extreme points are meant to be interepreted strongly or extremely.  An example would be a statement a statement that said “Reading about scaling techniques is:” and on the left side you would see “boring” and on the right side you would see “interesting.”  The respondent would then mark one of the five to seven blanks to indicate their attitude about the attribute.  As a control for tendencies of those with either a very positive or a very negative attitude sometimes the negative adjective or phrase appears on the left and sometimes it appears on the right.  Items on this scale may be scored on either a -3 to +3 or a 1 to 7 scale.  This scaling tool is frequently used in linguistics and social psychology.

Stapel Scale

Similar to the Sematic Differential Scale with some exceptions.   Points are assigned as numbers which is not the case in a Sematic Differential scale.  Instead of two dichotomous descriptive words or phrases only one word or phrase is used.  If the respondent disagrees then a negative number is marked. Word choice has the potential of biasing respondents. This is not widely used as it is thought to be confusing to respondents.

Mathematically Derived Scale

A scale in which researchers infer respondents answers mathematically as in the case of multi dimensional scaling and conjoint analysis.

Scaling emerged from the social sciences in an attempt to measure or order attributes with respect to quantitative attributes or traits.  Scaling provides a mechanism for measuring abstract concepts.  A comparative scale is an ordinal or rank order scale that can also be referred to as a nonmetric scale.  Respondents evaluate two or more objects at one time and objects are directly compared with one another as part of the measuring process.  For example you could ask someone if they prefer listening to MP3s through a Zune or an iPod. You could take it a step further and add some other MP3 player brands to the comparison.  MP3 players would be scaled relative to each other and the scale position of any one player would depend on the the scale position of the remaining players.  Because they are being compared differences such as who has the click wheel are effectively forced.  Where this is limiting is evident when you find no standard of comparison outside the objects being compared. No generalizations are made outside of these objects.  Often used when physical characteristics of objects are being compared.

Guttman Scaling

This can also be referred to as a cumulative scoring or scalogram analysis.  The intent of this survey is that the respondent will agree to a point and their score is measured to the point where they stop agreeing.  For this reason questions are often formatted in dichotomous yes or no responses.  The survey may start out with a question that is easy to agree with and then get increasingly sensitive to the point where the respondent starts to disagree.  You may start out with a question that asks if you like music at which point you mark yes.   Four questions later it may ask if you like music without a soul and which is produced by shady record labels only out to make money at which point you may say no. If you agreed with the first 5 questions and then started disagreeing you would be rated a 5.  The total of questions you agreed to would be added up and your final score would say something about your attitude toward music.

Rasch Scaling

This probabilistic model provides a theoretical basis for obtaining interval level measurements based on counts from observations such as total scores on assessments. This analyzes individual differences in response tendencies as well as an item’s discrimination and difficulty.  It measures how respondents interact with items and then infers differences between items from responses to obtain scale values.  This model is typically used analyze data from assessments and to measure abilities, attitudes, and personality traits.

Rank-Order Scaling

This gives the respondent a set of items and then asks the respondent to put those items in some kind of order. The “order” could be something like preference, liking, importance, effectiveness, etc.  This can be a simple ordinal structure such as A is higher than B or be done by relative position (give each letter a numerical value as in A is 10 and B is 7).   You could present five items and ask the respondent to order each one A-E in order of preference.  In Rank-Order scaling only (n-1) decisions need to be made.

Constant Sum Scaling

With this ordinal level technique respondents are given a constant sum of units such as points, money, or credits and then asked to allocate them to various items.   For example,  you could ask a respondent to reflect on the importance of features of a product and then give them 100 points to allocate to each feature of the product based on that.   If a feature is not important then the respondent can assign it zero.   If one feature is twice as important as another then they can assign it twice as much.   When they are done all the points should add up to 100.

Paired Comparison Scale

This is an ordinal level technique where a respondent is presented with two items at a time and asked to choose one.   This is the most widely used comparison scale technique.   If you take n brands then [n (n-1)/2] paired comparisons are required.  A classic example of when paired comparison is used is during taste tests.  For example you could have a taste test in which you have someone try both Coke and Pepsi and then ask them which one they prefer.

Bogardus Social Distance Scale

This is a cumulative score that is a variant of the Guttman scale, agreement with any item implies agreement with the preceding items. This scale is used to measure how close or distant people feel toward other people.  Social distance is a concern when it comes to issues related to racial integration or other forms of equality.  This is applicable to team formation in the work place for example.  Some people accept other people easily and use trustworthiness as the basis of their relationship with other people.  Other people do not accept people who are not like them and tend to keep those that are not like them at arms length.

Q-Sort Scaling

This is a rank order procedure where respondents are asked to sort a given number of items or statements and classify them into a predetermined number of sets (usually 11) according to some criterion such as preference, attitude, or behavioral intent.  Using cards that note an item to be ranked is the most popular and simplest method to use in the sorting process.  In order to increase statistical reliability at least 60 cards should be used and no more than 140.  This is good for discriminating among a large group of items in a relatively short amount of time.

We gather knowledge through observation and when we collect enough observations we form concepts.  The process of connecting observations to concepts is known as operationalization.  Specifying what we mean when we use certain terms is referred to as conceptualization.  Web survey research involves a systematic measurement of observations in order to come up with conclusions that lead to concepts. When we observe objects we understand them in terms of attributes.  These attributes may vary from object to object.  For example if hair is the object then the color would be the attribute.   A variable is understood as a value that changes and is therefore a  logical grouping of attributes.   You can classify every observation in terms of one and only one attribute.  You can not call hair both blond and red but you could call blondish red hair sandy.  The response options we allow survey respondents make a difference in the survey data we collect.  Given the option of classifying blondish red hair either blond or red there will be some that classify it as red and others that classify it as blond.  A neutral response option or allowing your survey respondent to decide the degree to which hair is either blond or red will give you better data to base your conclusions on.

Furthermore,  variables represent separate dimensions of concepts. A dimension can be understood as different aspects or facets of a concept.  The variable of gender consists of two attributes,  male and female.  Once the variable has been defined by one of those attributes then the only the variable involved in determining the concept of gender would be gender.  However, a concept such as socioeconomic status is defined by multiple variables.  When determining a person’s socioeconomic status we look at variables such as income, education, and occupational prestige.  Any one of which can change.  When coming to conclusions in survey research we must account for all the variables that go into determining a particular concept.

These variables can be numerical or classificatory.  When we understand a variable’s level of measurement we have a better understanding of how variables vary and can therefore understand what we have just measured more fully.  The relationship of the values assigned to attributes is referred to in terms of levels of measurement.  Knowing the level of measurement helps you to decide how to interpret the data collected for a variable.  They determine the level of mathematical precision with which the values of a variable can be expressed.  The nominal level of measurement is qualitative and has no mathematical interpretation.  The quantitative levels of measurement – ordinal, interval, and ratio – are progressively more mathematically precise as you move along the levels.

Nominal Scale

When variables have values that have no mathematical interpretation they differ in kind or quality but not in amount.  This measure offers names or labels for characteristics.   At this level data can be placed into categories and counted only in regard to frequency of occurrence.  There is no ordering or valuation implied.   When we talk about hair color we are referring to measurement on a nominal scale but no valuation is implied with any of the possible responses.

Ordinal Scale

When variables can logically be ranked ordered from greatest to leased.  For example,  in a customer satisfaction survey you may ask a client if they are “very satisfied”, “satisfied”, “dissatisfied”, or “very dissatisfied.”  A customer that responds “very satisfied” is more satisfied then one that marks “dissatisfied” but you can not quantify this as being 2 units more satisfied.  The interval between values can not be interpreted.   On this level measurement provides information about the order of categories but does not indicate the magnitude of differences between them.

Interval Scale

At the interval level numbers represent fixed measurement units but have no true zero point.   However,  the distance between numbers does have meaning.  A temperature zero degrees does not signify an absence of temperature any more than zero AD does not signify an absence of time.  This provides still more meaningful information about a variable.   It labels, orders, and uses consistent units of measurement to indicate the exact value of each category of response.

Ratio Scale

This is based on a true zero point and you can measure how much more one attribute is to another.  On the ratio level we can say that 10 is two times as much as 5 and 10 is 5 more than 5.   Because numbers have a zero point they can be added, subtracted, multiplied, and divided in a meaningful way.

As you progress up each level of measurement the measurement includes all the qualities below it and adds something new.  When applicable it is preferable to measure survey data that uses a higher level of measurement (interval, ratio) than a lower one (nominal, ordinal).

The order in which questions are asked in your web surveys can influence the results of your survey.  For this reason you should group your questions into logically coherent sections.   Grouping questions that are similar together will make your survey easier to complete for your respondent.  As will grouping questions with a similar format. Your transition from one question to another should be smooth. A poorly constructed survey will confuse your respondents,  bias their answers, and jeopardize the quality of your work.

The following are some guidelines for  sequencing questions for a research survey:

• Introductory Questions – These questions should stimulate interest without being difficult to answer.  Make the first question related to the stated subject matter.  This should be simple and straightforward.  Pretend you’re having a conversation with someone for the first time and you do not want to offend or bore them.
• Sensitive Questions – Place these late in the survey.  If rapport is established then there is a greater likelihood that the respondent will be willing to answer these questions.
• Related Questions – Grouping related questions together gives the respondent a chance to concentrate on specific issues without being distracted.   At the same time remain cognizant of arranging consecutive questions that evoke reflexive (given without adequate thought) responses together.
• Logical Sequence – There should be a clear logical order to the particular series of questions that are being asked.  Questions should be arranged in a sequential or temporal order.

Context effect describes the influence of environmental factors on one’s perception of a stimulus.  Context effect in surveys involve how responses can be shaped by the order of questions, format of responses, and any visual images. Respondents rely on their own unique evaluations, impressions, values, and beliefs when formulating answers.   The context of a question influences what a respondent considers as they answer that question. A question’s context can affect a respondent’s answer by suggesting a standard of comparison.

The use of images in web based surveys may affect respondent’s answers.   One study measured the effect of showing respondents an image of a healthy woman exercising versus a sick woman in bed and then asked them to rate their own health.  When shown the healthy woman respondents consistently rated their own health lower than when exposed to the image of the unhealthy woman.  Three different surveys were used with varying factors such as size and placement of the image and also the location of the question in the survey.

In my previous post, I talked about some of the reasons to start doing web surveys. Of course, for someone who has no experience with this way of gathering information, starting to run surveys can be a little daunting. As always, you’ll want to make the user’s experience your first priority. This means taking a little extra care in crafting the surveys themselves and the ways that you promote them.

Survey early, Survey often

Don’t go back through the last 5 years of business data and figure out everything you’ve ever wished you knew about your clients and their habits and dump it all into one survey. It would be great if all of your customers were willing to spend 30 or even 60 minutes answering your questions, but they are not. In fact, if you ask them to do this, they’ll probably lose a lot of respect for you and be less likely to do business with you again. You’ll probably have better results doing a short 5-6 question survey every few months.

Mix it up a little

Since you’re adding frequent, short surveys, why not hook them into different actions that your site’s visitors are usually taking anyway? If you have a membership area, you could add in a quick survey between the login page and the member’s area. If you offer something for download, you could stick one in before the download is served. Make it feel like a quick, relatively unobtrusive extra step in the middle of an action that your clients are used to performing anyway. If you play this right, you can actually gain an extra layer of data that might tell you about the different types of answers you get from users engaged in different types of activity on your site.

Explain yourself

You have to be classy about these types of things. Remember that you’re asking for your clients to take time out of their busy days, and explain to them why you think it’s important for them that they help you out. Will the results of this survey be used to make your site nicer for them to use? Will it help to improve your products or services in some specific way? Let your clients know that you care about their experience with you and are offering this survey in order to make sure that you keep improving. Make them love you.

Offer some incentives

They don’t have to be major incentives, just a little something commensurate with the effort you’re asking for. If you offer an information product and are using your surveys to improve it, offer clients a free or discounted copy of the new version for helping you improve it. It’s only fair. If it’s a membership site you’re asking questions about, consider giving a free week of membership for completing your survey. The other day I saw a bank offering free blankets for opening a checking account. I can’t possibly imagine whose banking decisions they’re trying to sway with this offer, but I’m sure they must have put some kind of market research into it. People like to feel rewarded.

With a little bit of brainstorming about the types of information you could use from your clients, a little planning about how you intend to extract the information and put it to use, and some basic common sense about the way that you ask people for their help, you might just open up your business to some surprising potentials that have been right under your nose for years. Or you might strike gold with one client’s dead-on analysis or fond wish. A good web-based survey software or survey service package could change your life.

If you’re just getting started with survey design, you’ll find that it can be a little tricky to write questions that get the information you’re looking for without biasing the response. I highly recommend looking over these basic guides to evaluation pamphlets provided by applied sociologist Gene Shackman.