Reliability and Validity of Human Services Manageme research

Solution Preview

RELIABILITY AND VALIDITY

1. Define and describe 2 types of reliability. Provide an example of reliability as it applies to human services research or to human services management research.
There are several general types of reliability estimates: (Wikipedia, the free Dictionary)
Inter-rater reliability is the variation in measurements when taken by different persons but with the same method or instruments. In statistics, inter-rater reliability, inter-rater agreement, or concordance is the degree of agreement among raters. It gives a score of how much homogeneity, or consensus, there is in the ratings given by judges. It is useful in refining the tools given to human judges, for example by determining if a particular scale is appropriate for measuring a particular variable. If various raters do not agree, either the scale is defective or the raters need to be re-trained.
For Example: In a study of organizational culture (ex. The school), the data gathering instrument may be tested for its reliability by asking two or more validators to determine the appropriateness of the questionnaire. The extent to which the validators or judges are in consensus determines reliability.
There are a number of statistics which can be used to determine inter-rater reliability. Different statistics are appropriate for different types of measurement. Some options are: joint-probability of agreement, Cohen's kappa and the related Fleiss' kappa, inter-rater correlation, concordance correlation coefficient and intra-class correlation.
Test-retest reliability is the variation in measurements taken by a single person or instrument on the same item and under the same conditions. Test-retest is a statistical method used to determine a test's reliability. The test is performed twice; in the case of a questionnaire, this would mean giving a group of participants the same questionnaire on two different occasions. If the correlation between separate administrations of the test is high (~.7 or higher) then it has good test-retest reliability.
Test-retest reliability is desirable in measures of constructs that are not expected to change over time. For example, if you use a certain method to measure an adult's height, and then do the same again two years later, you would expect a very high correlation; if the results differed by a great deal, you would suspect that the measure was inaccurate. The same is true for personality traits such as extraversion, which are believed to change only very slowly. In contrast, if you were trying to measure mood, you would expect only moderate test-retest reliability, since people's moods are expected to change from day to day. Very high test-retest reliability would be bad, since it would suggest that you were not picking up on these changes.
In some cases it is important to pick the right interval between tests. For example, the Center for Epidemiological Studies Depression Scale (CES-D) asks ...