The Academic Medical Center Linear Disability Score (ALDS) item bank: item response theory analysis in a mixed patient population

Abstract

Background

Currently, there is a lot of interest in the flexible framework offered by item banks for measuring patient relevant outcomes. However, there are few item banks, which have been developed to quantify functional status, as expressed by the ability to perform activities of daily life. This paper examines the measurement properties of the Academic Medical Center linear disability score item bank in a mixed population.

Methods

This paper uses item response theory to analyse data on 115 of 170 items from a total of 1002 respondents. These were: 551 (55%) residents of supported housing, residential care or nursing homes; 235 (23%) patients with chronic pain; 127 (13%) inpatients on a neurology ward following a stroke; and 89 (9%) patients suffering from Parkinson's disease.

Results

Of the 170 items, 115 were judged to be clinically relevant. Of these 115 items, 77 were retained in the item bank following the item response theory analysis. Of the 38 items that were excluded from the item bank, 24 had either been presented to fewer than 200 respondents or had fewer than 10% or more than 90% of responses in the category 'can carry out'. A further 11 items had different measurement properties for younger and older or for male and female respondents. Finally, 3 items were excluded because the item response theory model did not fit the data.

Conclusion

The Academic Medical Center linear disability score item bank has promising measurement characteristics for the mixed patient population described in this paper. Further studies will be needed to examine the measurement properties of the item bank in other populations.

Background

Functional status is now seen as an important determinant of patients' quality of life and a wide variety of instruments have been developed [1]. Instruments for quantifying functional status tend to have a fixed length and administer all items to the whole group of patients under scrutiny. Currently, interest is moving towards the more flexible framework offered by item banks in conjunction with item response theory. An item bank is a collection of items, for which the measurement properties of each item are known [2, 3]. Since item response theory centres on the measurement properties of individual items, rather than the instrument as a whole, it is not essential for all respondents to be examined using all items when using an item bank. It is even possible to select the 'best' items for individual patients using computerised adaptive testing algorithms [4]. This can reduce the burden of testing considerably for both patients and researchers. Furthermore, results from studies using different selections of items can be directly compared. Item banks measuring concepts such as quality of life [2, 5], the impact of headaches [6], fatigue [7, 8] and functional status [9–12] have been described.

Before an item bank can be implemented, it is essential to calibrate it. During the calibration process, the measurement properties of the individual items and the item bank as a whole are investigated. In contrast to the procedures used when developing fixed length instruments, it is not essential to present all items to all respondents in the calibration sample. It is often more efficient to offer targeted sets of items to particular groups within the sample. The items in common between any two sets of items are known as anchors. This kind of design is known as an incomplete, anchored calibration design and allows all items and patients to be calibrated on the same scale [13]. These designs have been used widely in preparing item banks for educational testing and has had some recognition in the development of medical instruments [14, 15]. Developments in psychometric theory mean that it is now possible to perform the same types of analysis on data resulting from incomplete designs, as on complete designs [16–18]. The consequences of the use of this kind of design in the development of the ALDS item bank have been discussed previously [14, 19]. If the primary aim of a study is to estimate the parameters of the two-parameter logistic item response theory model, as in this paper, little statistical information can be obtained from patients, whose functional status is much higher or lower than the difficulty of the items, with which they are presented [14, 19]. The respondents described in this paper were chosen to maximise the statistical information on, and hence minimise the standard errors of the estimates of, the parameters of the item response theory model. For this reason, they may not be representative of the populations described.

The Academic Medical Center Linear Disability Score (ALDS) item bank was developed to quantify functional status in terms of the ability to perform activities of daily life. The ALDS item bank covers a large number of activities, which are suitable for assessing respondents with a very wide range of functional status and many types of chronic conditions. The items were obtained from a systematic review of generic and disease specific functional health instruments [1]. The methodology used to develop the ALDS item bank, including the use of incomplete calibration designs, has been described in depth [14]. Other papers have examined technical [20] and practical [21] aspects of methods to deal with missing item responses and the use of a 'not applicable' response category for the items. It has been shown that some of the ALDS items may have different measurement characteristics for males and females and for younger and older respondents [22]. The first results showed that the ALDS item bank had acceptable psychometric properties in a residential care population [9].

This study expands the results described in previous papers by examining the measurement properties of a selection of ALDS items, judged to be clinically relevant for a range of medical specialities, in a mixed population. The sample of respondents consisted of: residents of supported housing, residential care or nursing homes; patients attending an outpatients clinic for the treatment of chronic pain; hospitalized stroke patients; or attending an outpatients clinic for Parkinson's disease. These groups of patients were chosen because they have a broad range of chronic conditions and levels of functional impairment.

Methods

Respondents

A total of 1002 respondents were included. The respondents were previously described [9] residents of supported housing, residential care or nursing homes (551 respondents – 55%) and patients included in a number of studies in the Academic Medical Center, Amsterdam, the Netherlands. The studies were to examine: the effectiveness of treating patients with chronic pain in a specialised outpatients' clinic (235 – 23%); the effectiveness of treatment of stroke in an academic setting (127 – 13%); the progression of Parkinson's disease when only standard medication is prescribed (89 – 9%). The median age of the whole sample was 78 years (range 19 to 103 years) and 691 (69%) were female. Since the respondents described in this paper were chosen to minimise the standard errors of the estimates of the parameters in the item response theory model, they may not be truly representative of the populations described. This is particularly true for the residents of supported housing, residential care or nursing homes and for the stroke patients.

Items

Each item in the ALDS item bank describes an activity of daily life. Examples include: 'Walking for more than 15 minutes'; 'showering'; and 'washing up'. The items were obtained from a systematic review of generic and disease specific instruments designed to measure functional health status [1]. Respondents were asked whether they could carry out each activity on their own at the present time. Each item has two response categories: 'I could carry out the activity' and 'I could not carry out the activity'. Two response categories were used because it has been shown that this maximises the reproducibility of scoring between time points and interviewers and increases clinical interpretability [23]. If a respondent had never had the opportunity to experience an activity, a 'not applicable' response was recorded. In the analysis, responses in the 'not applicable' category were treated as if the individual items has not been presented to the patients [21].

During the data collection, the interviewers signaled that some items were too 'hospital' based ('washing oneself in bed' for patients living at home or in residential care), had become 'old-fashioned' ('using a public telephone' and 'using a carpet beater') or were so alike that respondents could not differentiate between them ('showering and washing ones hair' and 'showering, but not washing ones hair'). For this reason, all of the 170 items included in the data collection were re-evaluated by two of the authors (NW and RJdH). A total of 115 items were judged to be actually suitable for inclusion in the ALDS item bank.

Data collection

The respondents attending a clinic for chronic pain filled in a questionnaire with a single set of 88 items (set A). These items were chosen by one of the authors (MGWD) because they were clinically relevant for this patient population and spanned the whole range of functional status represented by the ALDS item bank. All other respondents were interviewed by specially trained nurses or doctors. The respondents who had had a stroke were all presented with a single set of 21 items chosen by one of the authors (NW) to cover the lower end of the ALDS item bank (set B). The residents of supported housing, residential care or nursing homes and the respondents with Parkinson's disease were presented with one of four sets of 80 items (sets C, D, E and F), which were described previously [9]. In these sets, each of 160 items covering the whole range of levels of functioning represented by the item bank was randomly allocated to two sets. Items sets C and D have half their items in common, as do sets D and E, sets E and F and sets F and C. The data collection design is illustrated in Figure 1. The items that are in each set are indicated by the solid blocks. It can be seen that all sets except B contain items from the whole range of the item bank and that set B mainly contains items, which are from the lower end of the range of functional status represented by the ALDS item bank. Further details of the item sets are given in Table 1. The the items that are in each set and the number of respondents, to whom each item was presented and the number responding in each category are indicated in Table 2.

Figure 1

The items in each of the item sets A to F.

Table 1 characteristics of the 6 sets of items

Table 2 the 77 items and their measurement properties

Statistical analysis

The statistical analysis is has been developed from previous work [14] and very similar to that in a previous paper [9]. The analysis concentrates on the two-parameter logistic item response theory model [24]. This model has been chosen because it allows a more realistic model [25] for the data to be built than when the more restrictive one-parameter logistic model [26]. In addition, the one-parameter logistic model has been shown to be unsuitable as a final model for describing data resulting from functional status items [9, 14]. In the two-parameter logistic item response theory model, the probability, P _ik , that patient k responds to item i in the category 'can' is modeled using

where θ _k denotes the ability of patient k to perform activities of daily life. The discrimination parameter (α) and the difficulty parameter (β) describe the measurement characteristics of item i.

In step (a) items were excluded from further analysis if the item had been presented to fewer than 200 patients or if fewer than 10% or more than 90% of the responses were in the category 'can carry out'. In step (b), the items were examined using the one parameter logistic item response theory model [26] to investigate whether the item difficulty parameter (β _i ) was similar for male and female and for younger and older patients. This model was chosen because the parameters can be estimated using a smaller number of patients than are required to estimate the parameters in the two-parameter model satisfactorily [17]. The cutoff point between younger and older patients was 78 years, the median age. Items were excluded from further analysis if the difference in the value of the item difficulty parameters was more than half of the value of the standard deviance of the underlying distribution of ability parameters (θ). This is equivalent to a moderate effect size [27].

In step (c), estimates of the item parameters (α _i and β _i ) were obtained. The fit of the model to the data from each item was assessed using G² statistics [17]. Items, for which the fit statistic had a p-value of less than 0.01 were excluded from further analysis. In step (d), the dimensionality of the item bank was examined using item response theory based full information factor analysis [9, 16, 17]. An exploratory factor analysis was carried out on each of the six item sets. To examine the population as a whole, a confirmatory factor analysis was carried out using the data from all 1002 respondents. In addition, Cronbach's coefficient alpha was calculated for each of the six item sets and for all of the data [18, 28]. Steps (a), (b) and (c) were carried out in Bilog, version 3.0 [17] using marginal maximum likelihood estimation techniques with an empirically obtained distribution of the person parameters (θ). Step (d) was carried out using TESTFACT, version 4.0 [17].

Results

Of the 115 items that were regarded as suitable for inclusion in the ALDS item bank, 38 were removed from and 77 were retained in the item bank. In step (a), a total of 24 items were removed from further analysis. Two items had been presented to fewer than 200 respondents, 1 item had fewer than 10% of responses in the category 'can carry out' and 21 items had more than 90% of responses in the category 'can carry out'. In step (b), a total of 11 items were removed from further analysis. Four items had different measurement characteristics for younger and older patients. Seven items had different measurement characteristics for male and female patients. In step (c), 3 items were removed from further analysis because their item fit statistic had a p-value less than 0.01. The item parameters (α and β) are given, with their standard errors, in Table 2. The probability that respondents with a range of levels of functional status can perform the items is illustrated in Figure 2. A histogram of the values of the difficulty parameters (β _i ) is given in Figure 3. It can be seen that the items cover the whole range of functioning, although there are more 'easy' than 'difficult' items.

Figure 2

The probability that respondents with a range of levels of functional status can perform the items.

Figure 3

A histogram of the values of the difficulty parameters (β _i ).

In step (d), the values of Cronbach's coefficient alpha varied between 0.92 and 0.97 for the six sets of items and was equal to 0.98 for the whole data set. The values for each set of items are given in Table 1. The data sets had between 1 and 3 latent roots larger than 1 and the whole data set had 5 latent roots larger than 1. In general, there was one very large latent root and a number marginally greater than one. The percentage of variance explained by a single factor varied between 64% and 78% for the item sets and was equal to 77% for the whole data set.

Discussion

In this study, an item response theory analysis of the ALDS item bank has been examined using an incomplete calibration design and a sample of 1002 respondents from: supported housing schemes, residential care or nursing homes (551); an outpatients' clinic for patients with chronic pain (235); following a stroke (127); and an outpatients clinic for Parkinson's disease (89). Each item in the analysis was presented to between 297 and 705 respondents. This is well above the minimum, of 200 respondents, regarded as necessary to implement the two-parameter logistic item response theory model [17].

The resulting item bank contains 77 items representing a wide range of levels of functional status. Although there are a number of items, which have very similar item parameters or content, there is no need to reduce the number of items further. Since estimates of respondents' functionals status are comparable, even when different sets of items are used to score them, researchers can choose items, which are particularly relevant to their 'population'. In this way, accurate estimates can be obtained, whilst minimising the burden of testing on both researchers and participants in clinical studies.

Before the item response theory based analysis was carried out, 55 of 170 items included in the data collection design were removed from the item bank because they were judged to be unsuitable for inclusion in the ALDS item bank. The insight required to judge that some of the items were unsuitable for the ALDS item bank could only been obtained once the items had been presented to a wide range of respondents. In the future, when developing an entirely new item bank, it may wise to conduct a broad pilot study before embarking on the full calibration study.

Previous results have shown that a proportion of items in the ALDS item bank have different measurement properties for men and women and for younger and older patients [9, 22, 29]. These results have been confirmed in this paper. Ideally, potential differences between the measurement characteristics of the items for different patient populations, for different groups of raters and for the interview and self-report versions of the ALDS item bank should also be examined in the same way as the differences between age and gender based groups. However, this was not possible for two reasons. Firstly, the groups of respondents with Parkinson's disease or acute stroke were too small to perform this analysis satisfactorily. Secondly, the levels of functioning in the respondents with chronic pain were much higher than those of the respondents living in residential care. This means that it was not possible to compare the groups at similar levels of functional status. Thirdly, all of patients in any given group were rated in the same way. Hence, it is not possible to separate differences caused by groups of raters and those caused by characteristics of the patient groups.

The respondents described in this paper were chosen to maximise the statistical information on, and hence minimise the standard errors of the estimates of, the parameters of the item response theory model. For this reason, they may not be representative groups from the populations described. This is particularly true for the residents of supported housing, residential care or nursing homes and for the stroke patients.

This does not have any consequences for the interpretation and implementation of the estimates of the parameters of the item response theory model [14] or the item response theory based factor analysis, but means that the values of Cronbach's alpha should be confirmed in future studies. In addition, the results for patients after a stroke and with Parkinson's disease need to be confirmed due to the small sample sizes used. Furthermore, in future studies it will be essential to examine whether the 77 items presented in this paper have the same measurement characteristics if they are presented to patients in an interview by nurses or by doctors or if patients respond to the items in a self-report situation.

The results presented in this article are different to those presented in a previous article examining the data from the residents of supported housing schemes, residential care or nursing homes [9]. There are two main reasons for this. Firstly, the selections of items included in the analysis were different. Secondly, the data described in this paper were collected from a mixed population of respondents. Previous research has commented on the differences between the one-parameter and two-parameter logistic item response theory models. In this paper, the two-parameter logistic item response theory model has been used. This model was chosen because previous results have shown that the one-parameter logistic model is unsuitable for this type of data [9].

Conclusion

The results in this paper have shown that the ALDS item bank has promising measurement characteristics for a mixed patient population. The authors feel that the item bank can be used as a reliable indicator of functional health status in residents of supported housing, residential care or nursing homes, patients with chronic pain, acute stroke or Parkinson's disease. This paper has examined a mixed patient population, so the authors expect that the item bank will have good measurement characteristics for a wide range of other populations. However, care should be taken when using the ALDS item bank in other populations until these results have been confirmed.

Although this examination of the ALDS item bank has concentrated on six sets of items, future applications of the item bank are not bound to these sets of items. If these results are confirmed in future studies, the ALDS item bank will form a good foundation for a computerised adaptive testing procedure [4]. It would also be possible to select fixed length sets of items, specifically tailored to the level of functional status or clinical characteristics of a certain group of patients.

Funding

RH, NW and RL were supported by a grant from the Anton Meelmeijer fonds, a charity supporting innovative research in the Academic Medical Center, Amsterdam, The Netherlands.