Development and content validation of STIL.sg instrument for measuring system thinking and generic science skills in socio-technical contexts

Authors

  • Meiry Akmara Dhina Universitas Pendidikan Indonesia, Indonesia
    ID Indonesia
  • Ida Kaniawati Universitas Pendidikan Indonesia, Indonesia
    ID Indonesia
  • Ade Gafar Abdullah Universitas Pendidikan Indonesia, Indonesia
    ID Indonesia
  • Lilik Hasanah Universitas Pendidikan Indonesia, Indonesia
    ID Indonesia
  • Sugeng Rifqi Mubaroq Akademi Digital Bandung, Indonesia
    ID Indonesia

DOI:

https://doi.org/10.62672/joease.v4i2.145

Keywords:

Content validation, Educational assessment, Generic science skills, Socio-technical , System thinking

Abstract

This research presents the systematic development of the STIL.sg instrument for measuring System Thinking and Generic Science Skills in Socio-Technical Inquiry Learning contexts, followed by rigorous content validation through structured expert judgment. Using Evidence-Centered Design approach, a 27-item multiple-choice instrument was first developed based on established theoretical frameworks; the instrument was subsequently subjected to content validation by a panel of domain experts. Content validation involved five experts from science education, learning evaluation, and pharmacy fields. The Content Validity Ratio (CVR) values ranged from 0.68 to 0.94, with an overall Content Validity Index (CVI) of 0.784, indicating excellent content validity. Modified Kappa coefficients (κ) ranged from 0.65 to 0.89, confirming inter-rater reliability. Physical pharmacy was chosen as the application context due to its interdisciplinary nature requiring both systematic thinking and fundamental scientific skills. The developed instrument demonstrates strong theoretical foundation and expert consensus, providing a tool with strong evidence of content validity for measuring complex cognitive skills in applied science education.

References

Arieli-Attali, M., Ward, S., Thomas, J., Deonovic, B., & von Davier, A. A. (2019). The expanded evidence-centered design (e-ECD) for learning and assessment systems: A framework for incorporating learning goals and processes within assessment design. Frontiers in Psychology, 10, 853. https://doi.org/10.3389/fpsyg.2019.00853

Arnold, R. D., & Wade, J. P. (2015). A definition of systems thinking: A systems approach. Procedia Computer Science, 44, 669-678.

Assaraf, O. B.-Z., & Orion, N. (2005). Development of system thinking skills in the context of earth system education. Journal of Research in Science Teaching, 42(5), 518-560.

Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe's content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79-86.

Brotosiswoyo, B. (2001). Hakikat pembelajaran MIPA dan kiat pembelajaran fisika di perguruan tinggi. Jakarta: PEKERTI-MIPA Universitas Terbuka.

Camelia, F., & Ferris, T. L. J. (2018). Validation studies of a questionnaire developed to measure students' engagement with systems thinking. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 48(4), 574-585. https://doi.org/10.1109/TSMC.2016.2607224

Dalawi, I., Isa, M. R., Chen, X. W., Azhar, Z. I., Ismail, I. S., Bahar, R., & Awaludin, A. (2023). Development of the Malay language of understanding, attitude, practice and health literacy questionnaire on COVID-19 (MUAPHQ C-19): Content validity & face validity analysis. BMC Public Health, 23(1), 1131. https://doi.org/10.1186/s12889-023-16044-5

Dhina, M. A., Kaniawati, I., Abdullah, A. G., & Hasanah, L. (2024). Physics in pharmacy education: A socio-technical inquiry into advancing systems thinking and generic science. Educational Innovation Research, 4, 1-15.

Dugan, K., Mosyjowski, E. A., Daly, S. R., & Lattuca, L. R. (2022). Systems thinking assessments in engineering: A systematic literature review. Systems Research and Behavioral Science, 39(3), 534-551. https://doi.org/10.1002/sres.2808

Echevarría-Guanilo, M. E., Gonçalves, N., & Romanoski, P. J. (2017). Psychometric properties of measurement instruments: Conceptual bases and evaluation methods. Texto e Contexto Enfermagem, 26(4), e1600017.

Gomez, A., Guzman-Jimenez, R., & Cardenas-Garro, J. (2025). Designing a high school computational thinking test using evidence-centered design. International Conference on Technological Innovation and AI Research (ICTIAIR 2025), 17-24. https://doi.org/10.1049/icp.2025.1239

Hyytinen, H., Tuononen, T., & Braun, E. (2023). Editorial: Generic skills in higher education. Frontiers in Education, 8, 1162156.

Koehler, M. J., Shin, T. S., & Mishra, P. (2012). How do we measure TPACK? Let me count the ways. In Educational technology, teacher knowledge, and classroom impact (pp. 16-31). IGI Global.

Newton, S., Alemdar, M., Rutstein, D., Edwards, D., Helms, M., Hernandez, D., & Usselman, M. (2021). Utilizing evidence-centered design to develop assessments: A high school introductory computer science course. Frontiers in Education, 6, 695376. https://doi.org/10.3389/feduc.2021.695376

Nitko, A. J., & Brookhart, S. M. (2011). Educational assessment of students (6th ed.). Pearson.

Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459-467.

Popham, W. J. (2010). Classroom assessment: What teachers need to know (6th ed.). Pearson.

Porcino, A. J., Sayah, F., Ohinmaa, A., Parker, S., & Johnson, J. A. (2019). Physical examination skills training in pharmacy curricula across Canada: The current state. Canadian Pharmacists Journal, 152(6), 394-400. https://doi.org/10.1177/1715163519874677

Roebianto, A., Savitri, S. I., Aulia, I., Suciyana, A., & Mubarokah, L. (2023). Content validity: Definition and procedure of content validation. Testing, Psychometrics, Methodology in Applied Psychology, 30(1), 5-18. https://doi.org/10.4473/TPM30.1.1

Ruiz Moral, R., Caballero Martínez, F., García de Leonardo, C., Monge, D., Cañas, F., Castaño, P. M., Montón, C., & Parras, M. (2019). Design and implementation content validity study: Development of an instrument for measuring patient-centered communication. BMC Health Services Research, 19(1), 411. https://doi.org/10.1186/s12913-019-4247-z

Sireci, S. G. (2016). On the validity of useless tests. Assessment in Education: Principles, Policy & Practice, 23(2), 226-235.

Shute, V. J., & Becker, B. J. (2010). Innovative assessment for the 21st century. Educational Psychology Review, 22(4), 387-411. https://doi.org/10.1007/s10648-010-9148-y

Songer, N. B. (2015). Science assessment: The next generation. Smithsonian Science Education Center. Available at: https://ssec.si.edu/stemvisions-blog/science-assessment-next-generation

Swanson, D. B., & Case, S. M. (2018). Assessment in basic science instruction: Directions for practice and research. Advances in Health Sciences Education, 23(2), 437-453. https://doi.org/10.1007/s10459-017-9797-0

Terras, K., Boyle, P., Ahmed, A. H., & Wooller, J. M. (2024). Setting priorities for physical examination in pharmacy education: A Delphi study. Canadian Pharmacists Journal, 157(2), 70-76. https://doi.org/10.1177/17151635241228259

Wilson, A. B., & Notebaert, A. J. (2018). Assessing systems thinking: A tool to measure complex reasoning through ill-structured problems. Thinking Skills and Creativity, 28, 110-130. https://doi.org/10.1016/j.tsc.2018.03.003

Downloads

Published

21-06-2026

How to Cite

Dhina, M. A., Kaniawati, I., Abdullah, A. G., Hasanah, L., & Mubaroq, S. R. (2026). Development and content validation of STIL.sg instrument for measuring system thinking and generic science skills in socio-technical contexts. Journal of Environment and Sustainability Education, 4(2), 395–403. https://doi.org/10.62672/joease.v4i2.145

Issue

Vol. 4 No. 2 (2026)

Section

Articles

License

Copyright (c) 2026 Journal of Environment and Sustainability Education

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.