When I think of reliability I imagine always knowing what to expect.  If a person is able to produce the same quality work consistently then they are considered reliable.  You see it in sports all the time. Certain players have a knack for coming through in key situations no matter how late in the season or how worn down they are.   However, I can imagine few jobs that require more reliability than a surgeon.  Having an off day for them could prove disastrous. For a measure to be reliable it must demonstrate consistency as well as repeatability.  When carrying out research our results should be accurate across a range of measurements.  In surveys you would like to think that you would get the same response no matter what mood your respondent is in but that is not always the case.   A surgeon must deal with difficult situations while showing the same precision and reliability.  That is a quality to be admired but you can not always expect everyone to act like a surgeon at all times.  It’s also possible that your respondent won’t know what you mean when you ask them a certain question resulting in an answer that is entirely different from what you are attempting to measure.

Test-retest Reliability

If your survey respondent had to take your survey again would they answer the same questions the same way?  Test-retest reliability measures reliability over time.  A number of factors can affect reliability over time such as a person’s mood, time of the day, where the questions are placed in the survey (context effect), circumstantial events, vagueness, etc.  A good test will take into account factors that may influence survey results over time and minimize them so that results show little variation. If a test is unreliable then any one of a number of factors can lead to varying results depending on when the question is asked.  In general the more time a person takes between retesting the more variation you can expect in the responses.

If you ask Joe Q what he thinks about Candidate X on Tuesday he may view him favorably because X gave a really good speech on Monday.   Say Candidate X is indited later in the week in a corruption scandal.  Joe previously indicated that a candidate’s integrity is very important.  Last week he said that he was leaning toward Candidate X.  Now that Candidate X has been exposed you may think he is likely to give you a different response if you asked Joe the same question next Tuesday.   The reliability of opinion polls can be doubtful depending on the questions we ask because opinions tend to fluctuate over time.  What does Joe Q mean when he rates integrity as very important?  Perhaps Joe Q considers anyone that shares his ideology to have integrity.  Its possible that Joe Q would vote for Candidate X no matter what he thought of him personally because they share the same ideology.  Probing Joe’s past voting record would be more indicative of voter preference than asking a subjective question about integrity.  Asking him more objective questions that would not fluctuate from week to week would have higher test-retest reliability.

Parallel Forms Reliability

Another challenge reliability faces is in knowing what the best questions to ask are.  What does Joe Q mean when he rates integrity as very important?  Could we come up with better questions to predict how voters like Joe Q would vote? Another way to improve the reliability of a survey is to ensure that it is representative of the data you are trying to collect.   To do this increase the sample size.  If you are gathering research to find out whether voters like Joe Q are likely to vote for Candidate X then you need to find more people like Joe and ask them different questions or question sets based on the same construct.

You come up with a large set of questions to ask in your survey.  The construct that you are measuring is voter preference. The large question set is split in half and you administer each set to half of the targeted population.  You can then take a look at which questions are better indicators of voter preference. This combines what is known as a split test method with parallel form evaluation.

You can use parallel forms to measure a construct for people that are not like Joe Q.  Here you would divide a population that is representative of all likely voters in two.  Develop a large question set that measures a particular construct and then administer to each half of your representative population.  Now you can learn which questions are better indicators for voter preference for a representative population.

Inter-rater Reliability

This is necessary if you are conducting your survey using an interview process.   If  multiple people are interviewing Joe Q to ask what his opinion on politics is then inter-rater reliability measures the degree to which the observers agree.  This is the best way to measure reliability if you are using observation for your research.

Internal Consistency Reliability

The purpose of asking questions in surveys is to assess a particular construct or idea.  Therefore different questions that measure the same construct should yield similar results.  Reliability is determined on the basis of whether results are consistent for different items that measure the same construct.  For example, you could check for reliability on your survey by asking a respondent two similar questions meant to measure the same thing.

• Average Inter-Item Correlation – when we ask a respondent two similar questions to measure the same construct.  This compares correlations between this and any other paired questions to measure the same construct by calculating the mean of all paired comparisons.
• Average Itemtotal Correlation – where you take the average inter-item correlation and calculate a total score for each item.
• Split-half Correlation – you divide items that measure the same construct into two tests,  apply them to the same group of people, and calculate the correlation between the two scores.
• Cronbach’s Alpha – when we calculate the average split half estimates from a sample population.

In order to draw conclusions, formulate theories, or make generalizations about your research you need to ensure the reliability of the data you collect.  In general reliability is threatened when assessments are taken over time, rely on different standards of judgment, or assessments are highly subjective.  You can improve reliability by ensuring that your surveys are written clearly and without ambiguity.  You should construct your response options so that they are appropriate and meaningful.

Conclusions drawn from analyzing survey data are only acceptable to the degree to which they are determined valid.  Validity is used to determine whether research measures what it intended to measure and to approximate the truthfulness of the results. Researchers often use their own definition when it comes to what is considered valid.  In quantitative research testing for validity and reliability is a given.  However some qualitative researchers have gone so far as to suggest that validity does not apply to their research even as they acknowledge the need for some qualifying checks or measures in their work.  This is wrong.   To disregard validity is to put the trustworthiness of your work in question and to call into question others confidence in its results.   Even when qualitative measures are used in research they need to be looked at using measures of reliability and validity in order to sustain the trustworthiness of the results.  Validity and reliability make the difference between “good” and “bad” research reports. Quality research depends on a commitment to testing and increasing the validity as well as the reliability of your research results.

Any research worth its weight is concerned with whether what is being measured is what is intended to be measured and considers the ways in which observations are influenced by the circumstances in which they are made.   The basis of how our conclusions are made play an important role in addressing the broader substantive issues of any given study. For this reason we are going to look at various validity types that have been formulated as a part of legitimate research methodology.

Face Validity

This is the least scientific method of validity as it is not quantified using statistical methods.  This is not validity in a technical sense of the term.  It is concerned with whether it seems like we measure what we claim.  Here we look at how valid a measure appears on the surface and make subjective judgments based off of that.  For example,  if you give a survey that appears to be valid to the respondent and the questions are selected because they look valid to the administer.   The administer asks a group of random people, untrained observers,  if the questions appear valid to them.  In research its never sufficient to rely on face judgments alone and more quantifiable methods of validity are necessary in order to draw acceptable conclusions.  There are many instruments of measurement to consider so face validity is useful in cases where you need to distinguish one approach over another.  Face validity should never be trusted on its own merits.

Content Validity

This is also a subjective measure but unlike face validity we ask whether the content of a measure covers the full domain of the content. If a researcher wanted to measure introversion they would have to first decide what constitutes a relevant domain of content for that trait.  This is considered a subjective form of measurement because it still relies on people’s perception for measuring constructs that would otherwise be difficult to measure.   Where it distinguishes itself is through its use of experts in the field or individuals belonging to a target population.  This study can be made more objective through the use of rigorous statistical tests.  For example you could have a content validity study that informs researchers how items used in a survey represent their content domain, how clear they are, and the extent to which they maintain the theoretical factor structure assessed by the factor analysis.

Construct Validity

A construct represents a collection of behaviors that are associated in a meaningful way to create an image or an idea invented for a research purpose.  Depression is a construct that represents a personality trait which manifests itself in behaviors such as over sleeping, loss of appetite, difficulty concentrating, etc.  The existence of a construct is manifest by observing the collection of related indicators.  Any one sign may be associated with several constructs.  A person with difficulty concentrating may have A.D.D. but not depression.  Construct validity is the degree to which inferences can be made from operationalizations(connecting concepts to observations) in your study to the constructs on which those operationalizations are based.  To establish construct validity you must first provide evidence that your data supports the theoretical structure.  You must also show that you control the operationalization of the construct, in other words, show that your theory has some correspondence with reality.

• Convergent Validity - the degree to which an operation is similar to other operations it should theoretically be similar to.
• Discriminative Validity - if a scale adequately differentiates itself or does not differentiate between groups that should differ or not differ based on theoretical reasons or previous research.
• Nomological Network - representation of the constructs of interest in a study, their observable manifestations, and the interrelationships among and between these.  According to Cronbach and Meehl,  a nomological network has to be developed for a measure in order for it to have construct validity
• Multitrait-Multimethod Matrix - six major considerations when examining Construct Validity according to Campbell and Fiske.  This includes evaluations of the convergent validity and discriminative validity.  The others are trait method unit, multi-method/trait, truley different methodology, and trait characteristics.

Internal Validity

This refers to the extent to which the independent variable can accurately be stated to produce the observed effect.  If the effect of the dependent variable is only due to the independent variable(s) then internal validity is achieved. This is the degree to which a result can be manipulated.

Statistical Conclusion Validity

A determination of whether a relationship or co-variation exists between cause and effect variables.   Requires ensuring adequate sampling procedures,  appropriate statistical tests, and reliable measurement procedures. This is the degree to which a conclusion is credible or believable.

External Validity

This refers to the extent to which the results of a study can be generalized beyond the sample. Which is to say that you can apply your findings to other people and settings.   Think of this as the degree to which a result can be generalized.

Criterion-Related Validity

Can alternately be referred to as Instrumental Validity. The accuracy of a measure is demonstrated by comparing it with a measure that has been demonstrated to be valid.  In other words, correlations with other measures that have known validity. For this to work you must know that the criterion has been measured well.  And be aware that appropriate criteria do not always exist.  What you are doing is checking the performance of your operationalization against a criteria.  The criteria you use as a standard of judgment accounts for the different approaches you would use:

• Predictive Validity - operationalization’s ability to predict what it is theoretically able to predict.  The extent to which a measure predicts expected outcomes.
• Concurrent Validity - operationalization’s ability to distinguish between groups it theoretically should be able to.  This is where a test correlates well with a measure that has been previously validated.

When we look at validity in survey data we are asking whether the data represents what we think it should represent.  We depend on the respondent’s mind set and attitude in order to give us valid data.  In other words we depend on them to answer all questions honestly and conscientiously.  We also depend on whether they are able to answer the questions that we ask.  When questions are asked that the respondent can not comprehend or understand then the data does not tell us what we think it does.

As we are developing the new versions of both Email Marketing and Help Desk we have put a great amount of detail into usability.  The term usability is often used to simply describe the user interface.  But, the term encompasses a much larger area of software development.

A specific case in our recent development is cron management.  A number of our applications require you to setup a cron job.  In some cases you may need to setup 3 or 4 cronjobs depending on the features you are using.This is something that can be a pain for our users.  Once you setup the application you should not have to do more work when you start using a feature or purchase an add-on.  Thus we are designing our new cron management feature.  Our new cron management feature serves 2 primary purposes:

1. Ability to have a single cron – yet have different timed events.  This means that while you will only have one cron that you have to setup, our software can choose to execute different actions at different time intervals.
2. Improve the process of expanding your application and upgrading.  With this new feature in place you will not need to manually add any crons or events if you wish to use extra features, new add-ons, etc..

Those 2 purposes are the primary objectives of this feature and a great example of improving usability.   This feature will debut in Email Marketing 5 and Help Desk 3.

Effective email marketing writing is persuasive to your subscribers, enhances your image and credibility, and adds clarity to the message in your newsletter. Even the most eye popping email design graphics and layouts will lose their appeal if subscribers are presented with email copywriting that is shoddy or substandard. In the movie Glengarry Glen Ross, Alec Baldwin bellows the following advice to his hapless sales team; “A-B-C. A-Always, B – Be, C-Closing! Always be closing!! A-I-D-A. Attention, interest, decision, action.” AIDA is an acronym used in marketing that is short for Attention, Interest, Desire, and Action.  This can be applied to email marketing as well.

Attention – Grab your subscriber’s attention with an engaging subject line so they open your mailing. This is your invitation to your subscriber to hear what you have to say. Once your newsletter is opened you can also get your subscriber’s attention with the appearance of the newsletter as well as the headlines and lead sentences. You may want to think of this as your pick up line or your “hook”.

Interest - If you are meeting someone for the first time and all you do is talk about yourself then you shouldn’t be surprised if that person gets up and walks away. This could be the point where your subscriber quits reading your newsletter or they opt out of your list altogether. However, if you include a person in your conversation then they are going to want to stick around. Focus on advantages and benefits as they relate to the substance of your communication.

Desire – Strong interest yields to desire. Use the power of persuasion in your conversation now that you have your subscriber’s interest. The better you can relate to someone in your conversation, the more they will want to hear. Use the body of your newsletter to convince your subscriber that you have what they want and they will feel a need for what you are offering. You are writing to build up motivation for your subscriber for what you are offering with your newsletter.

Action – Your newsletter should have a purpose and your writing should lead your subscriber to a clear call to action. You’ve now convinced your listener that you have what they want. Make it easy for them to do something about it! This action could be clicking on a link, purchasing a product, or even replying to your newsletter. You should structure your mailing in a way that is likely to generate a direct response from your reader.

Now, a word about subject lines or what we have noted as your all important hook. While you want to make this interesting enough for your subscriber to have a reason for opening your mailing you are going to want to exhibit caution. Think of the subject lines that you see in your spam box. You are going to want to be weary of using dollar signs, too much punctuation, or appearing needy as these are often techniques used by spammers. The subject line is important because you do not want to ruin the integrity of your list by being seen as spam. If you google top 10 spam subject lines you will get a good idea of what you want to stay away from.

A good rule of thumb is to keep your subject lines consistent and familiar to your subscribers. You can start with the same words so that your recipients recognize you. For example, “ActiveCampaign Newsletter: Name of the Newsletter” as a lot of email clients will cut off the end of the subject line. Try to keep the length of your subject line below 50 characters. Some internal testing may come in handy to see the maximum number of characters you can use before the subject line is cropped. You need not worry about this too much as you want to use as many characters as is necessary to get your subject across. The trick to writing a good subject line is to appeal to your subscriber’s interest which should be some combination of your target audience and relating the purpose of the newsletter to their needs.

There are many stylistic ways of holding your subscribers interest. Be succinct in your writing. This means giving your subscriber relevant information in as few words as possible. The more succinct your writing is the more weight your words will carry.

Include the most important parts of your newsletter in the beginning. Subscribers will read the introduction first before they decide whether the rest is worth their time. Many subscribers are going to scan through rather than read everything you write in your newsletter so arrange your text accordingly. This may include using dashes and bullet points as well as blocks of text rather than chunky paragraphs. You would write a newsletter differently than you would write a term paper or a thesis.

You can also engage your subscriber’s interest by using the active voice in your writing. This conveys a powerful message because when the verb of a sentence uses the active voice, the subject is doing the acting. When you use the passive voice the subject is being acted upon which can make your writing sound weak.

Keep your newsletter personal. Make your subscriber feel as though you are talking directly to them. One obvious way to do this is through using personalization tags such as the subscriber’s name.

Write to your audience. You can find out more about your subscribers demographic by using a tool such as A/B split to find out what your subscribers respond to. You do not have to be overly formal as in traditional business or marketing writing. It is perfectly alright to be irreverent at times so long as you are not overly casual and risk setting the wrong tone for your audience. It is important to know who your subscribers are before setting a tone for your audience. Remember you can set up separate mailings that target specific market and customize each campaign accordingly.

You’ll need to take in a few more technical suggestions into considerations with your newsletter. It’s best to avoid URLs altogether in your links. If you do use URLs in your mailing make sure that is reduced in length and that it fits on a single line. If you must use two lines for your URL then tell the subscriber how to piece the composite URL in their window. Also along those same lines, be sure to wrap all text at 68 characters per line. Although all email clients vary, all email applications will correctly display text at 68 characters or less per line. Also many email readers will only read ASCII characters. Be sure to avoid the use of “smart quotes.” Use a text editor and not a word processor when composing your messages.

In general you can expect about 10% of your mailings are going to be reported as bounces. If your bounce rates are considerably higher than this then there are a few things you may want to look at. It could be that your mail server is rejecting mailings from being sent due to the content in your messages or you have been blacklisted. Another explanation for why your email delivery failure may be high is that the mail server you are using is either temporarily down or it is unreachable. High bounce rates impact your campaign not only because the sender’s reputation suffers but the delivery rate to valid email addresses is slowed. For this reason it is very important to keep your subscriber lists clean.

Bounces generally fall into one of two categories, hard bounces and soft bounces. What is typically referred to as a hard bounce is a mailing that was returned by a recipient mail server. This could be because the address is either invalid or the domain name doesn’t exist or is not recognized.

Examples of hard bounces which are defined by bounce codes in Email Marketing include:

5.0.0 – Hard – Address does not exist
5.1.1 – Hard – Bad destination mailbox address
5.1.6 – Hard – Mailbox has moved
5.1.7 – Hard – Bad sender’s mailbox address syntax
5.1.8 – Hard – Bad sender’s system address
5.3.2 Hard System not accepting network messages
5.2.3 Hard Message length exceeds administrative limit

In general a soft bounce is returned because of a temporary problem having to do with the email address or the recipient’s email server. For example, even if the email address you are sending to is valid the message could still bounce if the recipient’s mailbox is full.

Examples of soft bounces which are defined by bounce codes in Email Marketing include:

5.2.0 Soft Other or undefined mailbox status
5.3.1 Soft Mail system full
5.5.3 Soft Too many recipients

You can reduce the number of bounces in your lists by including unsubscription links in your mailings, you can have Email Marketing add an unsubscribe link to your mailings automatically or through a personalization tag. Another thing you can do is add an update profile link to your mailing, Email Marketing includes an update your profile personalization tag which you can also easily insert into your mailing. Another way to decrease bounces is to have addresses removed through setting up bounce management. In Email Marketing you can set up a cron job, cron_bounce.php that will cycle through your list and remove hard and soft bounces after a set number of times. Unless you set up this cron job you will need to manually run the bounce check until Email Marketing cycles through all your mailings and removes addresses after they bounce a set number of times. Bounces will not show up in statistics unless bounce management is set up using a POP account for bounces and if no bounce management is selected all bounces will be sent to the From sender’s address.

Another good idea is to include custom fields for phone or address so that you can use the phone or mail to update the address of those subscribers that bounce. Follow this link for more email marketing tips.

Here are some of the recent articles we have added to our knowledgebase. If you have not used our knowldgebase in the past I would highly suggest to check it out. We have many useful articles (often added daily) and they cover all the topics you need. From setting up, customizing, and specific features.

We just released version 4.53.3 of our Email Marketing bulk email software.

Couple of new features include:

1. Subscription Rules. Automatically subscribe/unsubscribe from multiple lists when person subscribes/unsubscribes from a given email list. These rules are setup per list.
2. CSV import file can be separated by comma or tab
3. Added ability to choose delimiter when importing subscribers
4. Added support for emails with long TLDs (such as .travel)

Other Changes made:

* Changed Batch Removals page to strip out double quotes prior to parsing the list
* Fixed a bug with switching lists to Dutch
* Fixed a bug in pagination on shared mailing reports page
* Fixed a bug in embedder in case that server cannot fetch the image
* Fixed a bug with including read tracker into image embedder
* Fixed error caused by rows having different numbers of fields in the import file
* Fixed bug in message quota counting
* Fixed bug: fetch: will now work for autoresponders with headers
* fixed bug in cron_responder that sent out unsubscribe autoresponders when person is subscribed

ActiveCampaign Email Marketing users have long been able to easily import a list of subscribers using a .csv file or a simple copy and paste method. But what happens if your subscriber lists are on another MySQL database? This is a quandary that many of our past clients have run into. Additionally wouldn’t it be nice if you could run automated daily syncs using a cronjob or a scheduled task! The database sync tool allows you to do this as it connects your Email Marketing to an external MySQL database. Previously sold as an add-on, it is now freely available as a standard feature of Email Marketing.

The sync add-on allows you to navigate and browse through your database information in a simply and easily. With the sync add-on you can synchronize email addresses, names, subscription dates, and any number of optional fields. Each sync is specific to a single mailing list and you can create as many syncs as you wish. Set any number of sync rules for each sync. Such as “I only want to sync rows that have a field with a certain value” and many other options. The rules allow you to specify conditions in the formats of equals, does not equal, contains, does not contain, and greater than, less than. For example you could say I want to select rows from my external database/table where the row has field age greater than 18 and field subscribed equal to one!

The sync add-on allows you to set options to keep your list up to date. Every time you run the sync it will add any new subscribers from your external source and the sync process will update all of your subscribers information in Email Marketing with the latest information from your external source (including subscriber fields). And because you have the ability to sync any number of fields from your external database, you can incorporate user data from other systems into your email templates for a much more personalized email design.

We have released the first beta of our new Email Marketing Monitor App. This desktop application allows you to monitor your messages in queue along with your scheduled mailings. It also acts as a cron emulator which will help eliminate any stalling issues you may have on your server.

Here are some screenshots:

You can download Email Marketing Monitor Beta (Desktop App.) through our support center.

Should you have any comments or suggestions – please let us know.

In the last couple days we have released minor updates to 4 of our products. Here is a summary of what we changed just recently.

• Email Marketing 4.51.6
  • Fixed redirection problem for users with only one list
  • Fixed empty Sender header problem
  • Fixed fetch@send bug in autoresponders
  • Fixed full path issue in auto_remind cron file
  • Fixed template JS bug for test email if not mime
  • Now parses links only if option is turned on
  • Now fetches reply-to field properly
  • Added fetch personalized to INSTANT responders too
• Survey Software 1.4.12
  • specific required fields update
  • add item page updates
  • invitation updates
  • css updates
  • php short tags removed
  • library/survey add/update item pages updated
• Knowledge Management 2.60.24
  • Moved trial check to avoid header() problems
  • Public Search page completely rewriten
  • Implemented public users, separately from admins
  • Fixed Settings/User page bugs
  • Public users are no longer members of Global Authentication System
  • pdf.php is from now on unencoded
  • Fixed installer bug for removing existing tables
  • Added support for Demo Mode
  • Fixed a major bug in function.php that occures on public side when article has no links in it
  • fixed include problems
  • Added new language strings
• Help Desk 2.60.4
  • Integrated new Knowledge Management system
  • Fixed expiration date on announcements (to allow indefinite ones)
  • Chart fix for HTTPS via IE issue
  • Fixed subject escaping problem in public viewTicket
  • Added more vars to use in public account_confirm.tpl.txt (username, name, email)
  • New language strings

As alwyas – we are continually seeking feedback on our products and services. Should you have any questions, comments, or concerns let us know at info@activecampaign.com