Composite reliability of multilevel data: It's about observed scores and construct meanings
This article shows how the concept of reliability of composite scores, as defined in classical test theory, can be extended to the context of multilevel modeling. In particular, it discusses the contributions and limitations of the various level-specific reliability indices proposed by Geldhof, Prea...
Saved in:
Published in | Psychological methods Vol. 26; no. 1; p. 90 |
---|---|
Main Author | |
Format | Journal Article |
Language | English |
Published |
United States
01.02.2021
|
Online Access | Get more information |
Cover
Loading…
Abstract | This article shows how the concept of reliability of composite scores, as defined in classical test theory, can be extended to the context of multilevel modeling. In particular, it discusses the contributions and limitations of the various level-specific reliability indices proposed by Geldhof, Preacher, and Zyphur (2014), denoted as ω̃
and ω̃
(and also α̃
and α̃
). One major limitation of those indices is that they are quantities for latent, unobserved level-specific composite scores, and are not suitable for observed composites at different levels. As illustrated using simulated data in this article, ω̃
can drastically overestimate the true reliability of between-level composite scores (i.e., observed cluster means). Another limitation is that the development of those indices did not consider the recent conceptual development on construct meanings in multilevel modeling (Stapleton & Johnson, 2019; Stapleton, Yang, & Hancock, 2016). To address the second limitation, this article defines reliability indices (ω
, ω
, ω
, α
, α
, α
) for three types of multilevel observed composite scores measuring various multilevel constructs: individual, configural, shared, and within-cluster. The article also shows how researchers can obtain sample point and interval estimates using the derived formulas and the provided R and Mplus code. In addition, a large-scale national data set was used to illustrate the proposed methods for estimating reliability for the three types of multilevel composite scores, and practical recommendations on when different indices should be reported are provided. (PsycInfo Database Record (c) 2021 APA, all rights reserved). |
---|---|
AbstractList | This article shows how the concept of reliability of composite scores, as defined in classical test theory, can be extended to the context of multilevel modeling. In particular, it discusses the contributions and limitations of the various level-specific reliability indices proposed by Geldhof, Preacher, and Zyphur (2014), denoted as ω̃
and ω̃
(and also α̃
and α̃
). One major limitation of those indices is that they are quantities for latent, unobserved level-specific composite scores, and are not suitable for observed composites at different levels. As illustrated using simulated data in this article, ω̃
can drastically overestimate the true reliability of between-level composite scores (i.e., observed cluster means). Another limitation is that the development of those indices did not consider the recent conceptual development on construct meanings in multilevel modeling (Stapleton & Johnson, 2019; Stapleton, Yang, & Hancock, 2016). To address the second limitation, this article defines reliability indices (ω
, ω
, ω
, α
, α
, α
) for three types of multilevel observed composite scores measuring various multilevel constructs: individual, configural, shared, and within-cluster. The article also shows how researchers can obtain sample point and interval estimates using the derived formulas and the provided R and Mplus code. In addition, a large-scale national data set was used to illustrate the proposed methods for estimating reliability for the three types of multilevel composite scores, and practical recommendations on when different indices should be reported are provided. (PsycInfo Database Record (c) 2021 APA, all rights reserved). |
Author | Lai, Mark H C |
Author_xml | – sequence: 1 givenname: Mark H C orcidid: 0000-0002-9196-7406 surname: Lai fullname: Lai, Mark H C organization: Department of Psychology |
BackLink | https://www.ncbi.nlm.nih.gov/pubmed/32673041$$D View this record in MEDLINE/PubMed |
BookMark | eNo1j81KxDAYRYMozo9ufADJzlU1f2067qSMOjDgRlcuhi_JV4m0TUnSgXl7C-rZXDiLy70rcj6EAQm54eyeM6kfesxsRtT6jCz5Rm4Kriq5IKuUvhnjStbqkiykqLRkii_JZxP6MSSfkUbsPBjf-XyioaX91GXf4RE76iDDI93lu0TBhCnTYBLGIzqabIg428FRG4aU42Qz7REGP3ylK3LRQpfw-i_X5ON5-968Fvu3l13ztC9AliwXFcx7HRqDmpdghRS1UqXWjtcztgZheFtXrVW2UqUw2s1acKulYC1IEGty-9s7TqZHdxij7yGeDv8vxQ9LXVQg |
CitedBy_id | crossref_primary_10_1108_JABES_10_2021_0185 crossref_primary_10_1108_JABS_06_2022_0212 crossref_primary_10_5993_AJHB_47_3_12 crossref_primary_10_1007_s10902_023_00625_7 crossref_primary_10_1093_geronb_gbab203 crossref_primary_10_1186_s40359_023_01209_z crossref_primary_10_1177_03616843231176222 crossref_primary_10_1002_jcpy_1350 crossref_primary_10_26710_sbsee_v5i3_2672 crossref_primary_10_1111_joop_12424 crossref_primary_10_1016_j_lindif_2024_102435 crossref_primary_10_1007_s11136_024_03678_0 crossref_primary_10_1016_j_jvb_2024_104014 crossref_primary_10_1111_bjep_12645 crossref_primary_10_1080_1359432X_2023_2198708 crossref_primary_10_1177_02654075211067254 crossref_primary_10_1111_bjdp_12504 crossref_primary_10_1080_08957347_2022_2034822 crossref_primary_10_1177_21677026231196121 crossref_primary_10_1177_00315125241248709 crossref_primary_10_1016_j_ufug_2023_128016 crossref_primary_10_3390_su142315942 crossref_primary_10_1007_s12208_022_00353_3 crossref_primary_10_7759_cureus_58379 crossref_primary_10_1016_j_stueduc_2023_101274 crossref_primary_10_1027_1015_5759_a000829 crossref_primary_10_1111_jcal_12924 crossref_primary_10_1177_00332941241241632 crossref_primary_10_1111_bjhp_12659 crossref_primary_10_1186_s40479_023_00229_w crossref_primary_10_1108_IJOTB_04_2022_0065 crossref_primary_10_1007_s12144_022_03955_6 crossref_primary_10_1002_jocb_584 crossref_primary_10_1111_ijn_13074 crossref_primary_10_1108_EBHRM_10_2023_0307 crossref_primary_10_1177_01461672221115558 crossref_primary_10_1177_00187267241236579 crossref_primary_10_1016_j_jbusres_2022_113551 crossref_primary_10_1016_j_bodyim_2024_101681 crossref_primary_10_1111_bjep_12703 crossref_primary_10_1080_10400419_2022_2124356 crossref_primary_10_1080_1359432X_2022_2104716 crossref_primary_10_1080_00273171_2024_2307034 crossref_primary_10_1108_LODJ_03_2023_0110 crossref_primary_10_1111_josi_12534 crossref_primary_10_1007_s12671_024_02318_5 crossref_primary_10_1016_j_tsc_2024_101585 crossref_primary_10_1186_s40536_024_00207_w crossref_primary_10_1146_annurev_clinpsy_080921_083128 crossref_primary_10_1080_02699931_2024_2314066 crossref_primary_10_1108_JMP_03_2022_0121 |
ContentType | Journal Article |
DBID | NPM |
DOI | 10.1037/met0000287 |
DatabaseName | PubMed |
DatabaseTitle | PubMed |
DatabaseTitleList | PubMed |
Database_xml | – sequence: 1 dbid: NPM name: PubMed url: https://proxy.k.utb.cz/login?url=http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed sourceTypes: Index Database |
DeliveryMethod | no_fulltext_linktorsrc |
Discipline | Psychology |
EISSN | 1939-1463 |
ExternalDocumentID | 32673041 |
Genre | Journal Article |
GroupedDBID | --- --Z -~X .-4 07C 0R~ 123 29P 354 53G 5VS 7RZ ABGFU ABIVO ABNCP ACHQT ACPQG AEHFB AETEA AFMIJ ALMA_UNASSIGNED_HOLDINGS AWKKM AZXWR CGNQK CS3 EPA F5P FTD HVGLF HZ~ ISO LW5 NPM O9- OHT OPA OVD P2P ROL SES SPA TEORI TN5 UHS XJT YNT ZPI |
ID | FETCH-LOGICAL-a350t-6a002debbe715ac232844577d18888c8a2b1f86fc4c6452b7d88821c7320fa3a2 |
IngestDate | Thu May 23 23:47:58 EDT 2024 |
IsPeerReviewed | true |
IsScholarly | true |
Issue | 1 |
Language | English |
LinkModel | OpenURL |
MergedId | FETCHMERGED-LOGICAL-a350t-6a002debbe715ac232844577d18888c8a2b1f86fc4c6452b7d88821c7320fa3a2 |
ORCID | 0000-0002-9196-7406 |
PMID | 32673041 |
ParticipantIDs | pubmed_primary_32673041 |
PublicationCentury | 2000 |
PublicationDate | 2021-Feb |
PublicationDateYYYYMMDD | 2021-02-01 |
PublicationDate_xml | – month: 02 year: 2021 text: 2021-Feb |
PublicationDecade | 2020 |
PublicationPlace | United States |
PublicationPlace_xml | – name: United States |
PublicationTitle | Psychological methods |
PublicationTitleAlternate | Psychol Methods |
PublicationYear | 2021 |
SSID | ssj0014384 |
Score | 2.585777 |
Snippet | This article shows how the concept of reliability of composite scores, as defined in classical test theory, can be extended to the context of multilevel... |
SourceID | pubmed |
SourceType | Index Database |
StartPage | 90 |
Title | Composite reliability of multilevel data: It's about observed scores and construct meanings |
URI | https://www.ncbi.nlm.nih.gov/pubmed/32673041 |
Volume | 26 |
hasFullText | |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnZ1bS8MwFICDU5C9iPe75EHwYVR7T-qbiLKJ7mmDgQ8jaRMQXDdYX_TXe06TdhcvqHsoo2lHlq85PTk5F0LOWcZZ4ipYpgquHZB-viOZ5g5uAWnJpRsoDBR-6sbtfvgwiAazwoVldEkhL9P3L-NK_kMVzgFXjJL9A9n6R-EEfAe-cATCcPwVY5zM6HRVlj55MRm3zY45ugm-oj9QC11AcdnfKUrTvHFEHks0xoKuOcUsllMb3GaTybZGSuS1Ad0qrouC0tSdrtXxR1PTGuN-Wm1rd7WWBN-rnI_xRWCkXxIkDojOYF48moD2hcfAyDpT5vOTCDZB_NAPtzTdsfmL4O9ORiUM0BpBtpicVz-3LqXDrpoapME4irQummfstlEY8LDKQRuwq1knmmS9unFp_VDqEb1NsmEXAPTG0NwiKyrfJs16eN92yHONlc5hpWNNZ1gpYr2mneJiSkuktEJKDVIKSGmNlFZId0n__q5323ZsDQxHBJFbOLGA_mdKSsW8SKSg__IwjBjLPA6flAtfeprHOg1T3KKWMPdgzeSlLPBdLQLh75HVfJyrA0LTLJOJn4U6TeIw5iLxXcljJnUmlIhddUj2zcAMJybRybAasqNvW45Jc_YknZA1DTNLnYKaVsizEs0Hlfc_ZQ |
link.rule.ids | 786 |
linkProvider | National Library of Medicine |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Composite+reliability+of+multilevel+data%3A+It%27s+about+observed+scores+and+construct+meanings&rft.jtitle=Psychological+methods&rft.au=Lai%2C+Mark+H+C&rft.date=2021-02-01&rft.eissn=1939-1463&rft.volume=26&rft.issue=1&rft.spage=90&rft_id=info:doi/10.1037%2Fmet0000287&rft_id=info%3Apmid%2F32673041&rft_id=info%3Apmid%2F32673041&rft.externalDocID=32673041 |