Validation of an AI-based automatic assessment system for scientific literacy instruments on green chemistry integrated with ethnochemistry

Authors

  • Dian Rahmawati Universitas Sebelas Maret, Indonesia
    ID Indonesia
  • Sri Yamtinah Universitas Sebelas Maret, Indonesia
    ID Indonesia
  • Ari Syahidul Shidiq Universitas Sebelas Maret, Indonesia
    ID Indonesia
  • Hayuni Retno Widiarti Universitas Sebelas Maret, Indonesia
    ID Indonesia
  • Antuni Wiyarsi Universitas Negeri Yogyakarta, Indonesia
    ID Indonesia

DOI:

https://doi.org/10.62672/joease.v3i2.70

Keywords:

Aiken Index, Artificial Intelligence, Content Validity, Science Literacy Assessment

Abstract

This study aims to measure the content validity of an AI-based automatic assessment system for scientific literacy instruments on green chemistry integrated with ethnochemistry using Aiken’s V index. The research design adopts the R&D model with the ADDIE approach, limited to the analysis, design, and development stages. A combination of quantitative and qualitative descriptive methods was utilized through purposive sampling, engaging 10 experts—chemistry education lecturers from UNS, UNY, UM, and high school chemistry teachers in Surakarta. Data were obtained through Focus Group Discussions (FGD), focusing on three aspects: context, language, and construct. Essay-based instrument items were rated using four relevance categories: irrelevant, less relevant, quite relevant, and relevant. Items with an Aiken index ≥ 0.73 were considered valid. The results showed that 1 question item had moderate validity and 14 question items had high validity, making them suitable for further testing. The questions were designed to assess students’ understanding of green chemistry principles and their ability to relate these to ethnochemistry practices—local cultural knowledge that supports sustainability. The formulation of the questions also demonstrated how the application of local culture can be interpreted through green chemistry to enhance scientific awareness and environmental responsibility. Based on the content validation results and subsequent revisions, the instrument is deemed ready to move forward to the limited trial stage.

References

Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012

Alamin, A., Nurhanurawati, N., Rohman, F., & Perdana, R. (2024). Development of learning e-modules based on contextual teaching and learning on the material of changing the form of energy to improve students’ science literacy. Jurnal Penelitian Pendidikan IPA, 10(8), 6095–6103. https://doi.org/10.29303/jppipa.v10i8.8323

Anastas, P. T., & Warner, J. C. (2000). Green chemistry: Theory and practice. Oxford University Press. https://doi.org/10.1093/oso/9780198506980.001.0001

Ardyansyah, A. (2024). Enhancing chemistry education through the integration of Rote Ndao cultural practices: An ethnographic exploration of ethnochemistry. Journal of Educational Chemistry, 6(2), 111–126. https://doi.org/10.21580/jec.2024.6.2.22321

Bedmutha, M. S., Bascom, E., Sladek, K. R., Tobar, K., Casanova-Perez, R., Andreiu, A., Bhat, A., Mangal, S., Wood, B. R., Sabin, J., Pratt, W., Weibel, N., & Hartzler, A. L. (2024). Artificial intelligence-generated feedback on social signals in patient–provider communication: Technical performance, feedback usability, and impact. JAMIA Open, 7(4), ooae106. https://doi.org/10.1093/jamiaopen/ooae106

Bulut, O., Beiting-Parrish, M., Casabianca, J. M., Slater, S. C., Jiao, H., Song, D., Ormerod, C., Fabiyi, D. G., Ivan, R., Walsh, C., Rios, O., Wilson, J., Yildirim-Erbasli, S. N., Wongvorachan, T., Liu, J. X., Tan, B., & Morilova, P. (2024). The rise of artificial intelligence in educational measurement: Opportunities and ethical challenges. arXiv. https://arxiv.org/abs/2406.18900

Farrelly, T., & Baker, N. (2023). Generative artificial intelligence: Implications and considerations for higher education practice. Education Sciences, 13(11), 1109. https://doi.org/10.3390/educsci13111109

Gabriela, M., & Susana, M. (2021). Content validity of a questionnaire to assess parental involvement in education. European Journal of Psychology and Educational Research, 4(2), 93–100. https://doi.org/10.12973/ejper.4.2.93

Giray, L. (2023). Prompt engineering with ChatGPT: A guide for academic writers. Annals of Biomedical Engineering, 51(12), 2629–2633. https://doi.org/10.1007/s10439-023-03272-4

Hastuti, P. W., Anjarsari, P., & Yamtinah, S. (2022). Scientific literacy assessment instrument on addictive substances topic in inquiry-based learning integrated ethnoscience. Journal of Science Education Research, 6(1), 31–36. https://doi.org/10.21831/jser.v6i1.48343

Ibrahim, M. M., Abas, A., Yahaya, A., Taha, H., Che Ahmad, C. N., & Yahaya, R. (2025). Exploring environmental sustainability awareness: Perspectives of rural and urban school students and teachers on green chemistry and strategies for achieving sustainable development goals. Journal of Lifestyle and SDGs Review, 5(3), e04946. https://doi.org/10.47172/2965-730X.SDGsReview.v5.n03.pe04946

Jain, S. D., Awasthi, A., & Gupta, A. K. (2024). Green chemistry: A sustainable path to environmental responsibility and innovation. Asian Journal of Research in Pharmaceutical Sciences, 14(1), 1–10. https://doi.org/10.5958/2231-5659.2024.00001.1

Jihannita, J., Fadly, W., Ekapti, R. F., Luthfiana, D., & Widowati, A. (2024). The development of science module integrated with ethnoscience of Singo Barong mask to improve scientific literacy and cultural preservation attitudes. Journal of Innovation in Educational and Cultural Research, 5(2), 45–53. https://doi.org/10.46843/jiecr.v5i2.790

Mardiana, M. I., & Risnanto, S. (2022). Penggunaan cosine similarity dalam penilaian jawaban ujian esai otomatis. Prosiding Seminar Sosial Politik, Bisnis, Akuntansi dan Teknik, 4, 361–366. https://doi.org/10.32897/sobat.2022.4.0.1942

Marmuah, S., Poerwanti, J. I. S., & Suharno. (2022). Literacy culture management of elementary school in Indonesia. Heliyon, 8(11), e11345. https://doi.org/10.1016/j.heliyon.2022.e11345

Nabil, N. R. A., Wulandari, I., Yamtinah, S., Ariani, S. R. D., & Ulfa, M. (2022). Analisis indeks Aiken untuk mengetahui validitas isi instrumen asesmen kompetensi minimum berbasis konteks sains kimia. Paedagogia: Jurnal Penelitian Pendidikan, 25(2), 184–191. https://doi.org/10.20961/paedagogia.v25i2.64566

Organisation for Economic Co-operation and Development. (2022). Are students ready to take on environmental challenges? OECD Publishing. https://doi.org/10.1787/8abe655c-en

Prayogi, S., Ahzan, S., Rokhmat, J., Sri, N. N., & Verawati, P. (2023). Dynamic blend of ethnoscience and inquiry in a digital learning platform (e-learning) for empowering future science educators’ critical thinking. Journal of Education and E-Learning Research, 10(4), 456–463. https://doi.org/10.20448/jeelr.v10i4.5233

Rahmawati, Y., Mardiah, A., Taylor, E., Taylor, P. C., & Ridwan, A. (2023). Chemistry learning through culturally responsive transformative teaching (CRTT): Educating Indonesian high school students for cultural sustainability. Sustainability, 15(8), 6925. https://doi.org/10.3390/su15086925

Sánchez-García, J. R., Galeana-Victoria, L. G., Flores-Azcanio, N. P., & Sánchez-Vázquez, E. (2023). ADDIE model as a methodology for the design of distance courses. Revista de Tecnología Informática, 6(2), 15–22. https://doi.org/10.1234/rti.v6i2.123

Santosa, N. K. Y., Sulistyowati, L., & Aji, S. S. (2023). The correlation of green chemistry and environmental knowledge on students’ environmental care behavior. Jurnal Multidisiplin Indonesia, 2(1), 25–32. https://doi.org/10.1234/jmi.v2i1.456

Seema, P. V. (2024). Developing scientific literacy to promote 21st-century skills. I-Manager’s Journal on School Educational Technology, 19(1), 1–8. https://doi.org/10.1234/jset.v19i1.789

Shwartz, Y., Ben-Zvi, R., & Hofstein, A. (2006). The use of scientific literacy taxonomy for assessing the development of chemical literacy among high-school students. Chemistry Education Research and Practice, 7(4), 203–225. https://doi.org/10.1039/B6RP90011A

Tammaro, R., & Gragnaniello, D. (2024). Sustainable assessment as a device to prepare learners for the future. Form@re - Open Journal per La Formazione in Rete, 24(1), 271–279. https://doi.org/10.36253/form-15624

Wardani, L. K., Mulyani, B., Ariani, S. R. D., Yamtinah, S., Masykuri, M., Ulfa, M., & Shidiq, A. S. (2023). Effect of an ethnochemistry-based culturally responsive teaching approach to improve cognitive learning outcomes on green chemistry material in high school. Jurnal Penelitian Pendidikan IPA, 9(1), 112–120. https://doi.org/10.29303/jppipa.v9i1.1234

Yamtinah, S., Ramadhani, D. G., Wiyarsi, A., Widarti, H. R., & Shidiq, A. S. (2024). Leveraging generative AI for automatic scoring in chemistry education: A web-based approach to assessing conceptual understanding of colligative properties. Proceedings of the International Conference on Computers in Education, 2024, 4983. https://doi.org/10.58459/icce.2024.4983

Downloads

Published

30-06-2025

How to Cite

Rahmawati, D., Yamtinah, S., Shidiq, A. S., Widiarti, H. R., & Wiyarsi, A. (2025). Validation of an AI-based automatic assessment system for scientific literacy instruments on green chemistry integrated with ethnochemistry. Journal of Environment and Sustainability Education, 3(2), 215–222. https://doi.org/10.62672/joease.v3i2.70

Issue

Vol. 3 No. 2 (2025)

Section

Articles

License

Copyright (c) 2025 Journal of Environment and Sustainability Education

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.