A meta-analysis of the effects of problem-based learning on students' creative thinking in mathematics Yunita yunita1996nita@upi.edu Juandi Dadang Tamur Maximus Adem Angela Merici G Pereira Jerito Universitas Pendidikan Indoesia Department of Mathematics Education Jl. Dr. Setiabudi No. 229 Bandung 40154 Indonesia Universitas Katolik Santu Paulus Ruteng Jl. Jend A Yani 10 Ruteng Indonesia Teachers College Columbia University USA Guangxi Normal University China

This study examined the effect of Problem-based Learning (PBL) on students' creative mathematical thinking in Indonesia during the last eight years using meta-analysis. Data were obtained from primary studies published in national journals, conference proceedings, and master thesis from 2012 to 2020. This study analyzed 19 effect sizes from 19 primary studies that fulfill the inclusion criteria. The effect size index used the Hedges' g equation, and statistical analysis was assisted by comprehensive meta-analysis (CMA) software. The effect size variation test was conducted by analyzing the study's five characteristics, namely the year of study, level of education, sample size, length of treatment, and publication sources. Based on the analysis results, the overall effect size was 0.821, with a standard error of 0.130 at the 95% confidence interval (CI) referring to the random-effects model. The effect size is classified as moderate, meaning that PBL has a moderately positive effect on students' creative mathematical thinking. The results of the analysis of the characteristics of the study show that the application of PBL to improve students' creative mathematical thinking relates to the research year (Q = 55.683; p <0.05), the treatment duration (Q = 5.644; p <0.05), and publication sources (Q = 13.812; p <0.05). It was found that the PBL was more effective in conditions of treatment duration of fewer than four meetings. There is a tendency that PBL has a significant impact when it is first implemented. The findings' implications are discussed to provide important information as a basic idea for further PBL research and implementation.

Keywords: Problem-based learning, Creative thinking, Meta-analysis

 <bold id="bold-fb7f36a1a559f3cfc7bc06335606e02f">Introduction</bold>

Mathematical creative thinking is an essential prerequisite for academic achievement and success in the workplace (Bochniak, 2014). The development of creative thinking is significant for preparing students for future challenges and helping them solve problems (Ismail et al., 2018; Nuha, Waluya, &amp; Junaedi, 2018; Fauzi, Dirgeyase, &amp; Priyatno, 2019; Schoevers et al., 2019). In this case, the mastery of creative thinking in mathematics has far-reaching implications for students.

Mathematics instructions can be improved by implementing more appropriate models (Freeman-Green et al., 2015). Indeed, the learning process involving contextual problems can develop students' creative mathematical thinking (Maskur et al., 2020; Mustofa &amp; Hidayah, 2020; Sugiharto et al., 2019; Thorndahl &amp; Stentoft, 2020). Problem-based learning (PBL) is a learning model that helps students acquire knowledge and improve their skills (Servant-Miklos, 2018). PBL is considered one of the models which support the development of students’ creative thinking in mathematics.

Over the years, there appears to be an increase in the implementation of PBL in education (Ceker &amp; Ozdamli, 2016) since it has considerable effects on students' creative mathematical thinking. However, previous research examining this theoretical assumption has shown inconsistent results. On the one hand, several studies (e.g., Nugroho, Chotim &amp; Dwijanto, 2013; Khoiri, 2013; Katminingsih &amp; Widodo, 2015) have shown that PBL is effective in improving students' creative thinking in mathematics. On the other hand, other research found that the creative mathematical thinking of groups of students taught with PBL is no better than those who learn with conventional approaches (e.g., Ahmad, &amp; Gunawan, 2019; Alifiani, Dwijanto, &amp; Cahyono, 2019; Indriani, Widyasari, &amp; Amri, 2019). Despite the different results, teachers need convincing information to decide concerning the usefulness of the PBL with particular references to improving students' creative mathematical thinking.

To meet this need, it is necessary to integrate convincing quantitative findings to provide useful information for education policy (Higgins &amp; Katsipataki, 2015). A meta-analysis study is required to integrate and interpret the findings of both the overall study and the variables of the primary study (Tamur, Juandi, &amp; Adem, 2020). The meta-analysis provides profound and convincing conclusions (Siddaway, Wood, &amp; Hedges, 2019). When there is a need to draw accurate and convincing conclusions, it is necessary to analyze the results of various primary studies (Kulik, Kulik, &amp; Shwalb, 1986) with meta-analysis (Tamur, Juandi, &amp; Kusumah, 2020; Yaakub, 1998).

Preceding meta-analyses have evaluated the effectiveness of PBL in general (e.g., Mustaffa et al., 2014; Asror, 2016; Yanto et al., 2019). Mustaffa et al. (2014) synthesized eighteen individual studies and only highlighted the role of PBL in the development of students' various cognitive and affective dimensions. Also, Asror (2016) analyzed ten individual studies on the effects of PBL on students' mathematical abilities. Meanwhile, Yanto et al. (2019) focus on study design and research methods of 15 individual studies. Moreover, Anugraheni (2018) has conducted a unique meta-analysis testing the effectiveness of PBL on students' critical thinking. However, the study analyzed individual studies examining the effects of PBL in primary schools. Today, there has no meta-analysis that explicitly questions the effectiveness of PBL on students' overall creative mathematical thinking.

This study examined the overall effect of PBL and attempted to analyze the causes of variation in outcomes by examining the relationship between study characteristics. These characteristics contribute to revealing important information about how PBL will be implemented in the future. The questions addressed in the study were: (1) does the use of the PBL produce a more significant effect size on students' creative mathematical thinking than conventional approaches? And (2) does the effect size of students' creative mathematical thinking on the implementation of PBL between study groups vary in terms of the study year, education level, research class, sample size, treatment duration, and publication source?

 <bold id="bold-4">Methods</bold>

This study applied a meta-analysis. It is a set of quantitative techniques for combining evidence from several related studies (Cumming, 2012). The results of the meta-analysis are to tell us whether or not all of the simultaneous effects of a host of authentic research does work or does not do so (Cleophas &amp; Zwinderman, 2017). The procedures are as follows:

Firstly, empirical data is obtained from online databases, including ERIC (Education Resources Information Center), SAGE, and Springer. The three databases capture journal articles related to the research objectives. Furthermore, Semantic Scholar and Google Scholar databases are used to identifying journal articles, productions, and theses related to the research problems. The keywords used were "problem-based learning" and "creative thinking skills" to reach English articles. The keywords "pembelajaran berbasis masalah" and "kemampuan berpikir kreatif" were searched to achieve articles written in Bahasa. At this stage, we found thirty studies that examined the implementation of PBL on students' creative mathematical thinking.

Secondly, determining inclusion criteria, namely the feasibility of the standards used in selecting primary studies. At this stage, we have considered suggestions by Rothstein and Hopewell (2009) about the need to include “gray literature” such as unpaired theses and articles that complement other sources and reduce research bias. Thus, articles are selected regardless of the indexing. Following the research objectives, the inclusion criteria in this study were:

1. The articles were published from 2012 to 2020

2. The research design used experimental and quasi-experimental with the control class as a comparison

3. The duration of the treatment is at least three weeks

4. The treatment group used PBL

5. The independent variable measured is creative thinking

6. It contains statistical information for calculating effect sizes, i.e., mean, standard deviation, and sample size.

Based on the criteria, nineteen articles were obtained. Information regarding the studies is presented in Table 1.

The meta-analysis instrument was in the form of coding sheets (Tamur et al., 2020). The coding process can help identify lost or improperly coded data. The coding in this study was carried out by extracting information from nineteen primary studies into numerical data. This work involved two coders, namely the principal investigator and the co-researcher, then discussed with the supervisor. To ensure that the data entered were not errors, the collaborating research team was involved in filling out the coding form and then carrying out verification.

Primary studies on the effect of PBL on students’ creative mathematical thinking

No Study Characteristics Group Frequency
1 Study year 2012-2013 2
2013-2014 1
2014-2015 4
2015-2016 2
2016-2017 2
2017-2018 4
2019-2020 4
2 Education level Primary Schools 3
Secondary Schools 9
High Schools 3
Higher Education 4
3 Sample size ≤30 5
≥31 14
4 Treatment duration 3-4 meetings 3
>4 meetings 16
5 Publication source Theses 3
Proceeding 2
Journal 14

Bias risk was evaluated based on random sequence generation, disclosure, blinding, blinding the interveners, the blinding of results evaluators, incomplete data results, selective data reporting, and other factors (Schuch et al., 2016). In this study, the factor that can be identified is selective reporting, namely the journals' tendency only to publish articles that are considered significant. Because significant studies were more likely to be included in the meta-analysis than their unpublished counterparts, there is concern that the meta-analyses might overestimate accurate effect sizes (Borenstein et al., 2009). For this reason, the impact of publication bias must be controlled by examining the research funnel plot as a first step (Pigott &amp; Polanin, 2020). In a meta-analysis, in order to measure the presence or absence of publication bias, a funnel plot was used. In the absence of bias and heterogeneity, the plot should, therefore, appear funnel-shaped with the greatest variability at the bottom and least variability at the top (Langan et al., 2012). The funnel plot has two elements: (i) the distribution of the effect size (in our case, the PBL effect size) on students' creative thinking; (ii) confidence bands around the overall mean to assess whether the observed results are statistically significant at the 5% level (Abramo et al., 2016). If the distribution of effect sizes is symmetrical around the vertical line on the funnel plot, the study is free from bias. However, if not, then the fail-safe N statistic (FSN) was used to check whether there was an effect of publication bias or not. Figure 1 presents the research funnel plot.

Research funnel plot

Figure 1 shows that the distribution of the effect sizes is not all symmetrical around the perpendicular line as the overall effect size. Therefore, Rosental fail-safe N (FSN) statistics were used to check for publication bias. The formula used is N / (5k + 10) (Fragkos, Tsagris, &amp; Frangos, 2016), where N is the FSN value, and k is the number of studies included in the meta-analysis. If the numeric value obtained is greater than 1, it means that the result is far from bias. The results of the failure-safe N (FSN) statistical calculations using the CMA software are shown in Table 2.

Rosental's FSN statistics

No Bias Conditions Score
1 Number of observed studies 19.00000
2 Number of missing studies that would bring p-value to > alpha 971.00000

Based on Table 2, the FSN value and formula (Fragkos et al., 2016) state that 971 / (5 * 19 + 10) the result is 9.25 greater than 1. The result indicates the study included in the analysis is resistant to bias. Thus, there is no tendency for selective reporting to lead to overrepresentation. Thirdly, using statistical analysis to evaluate the effectiveness of PBL in creative mathematical thinking. In conducting the meta-analysis, statistical information is needed for effect size transformations, namely the mean, standard deviation, and sample size for each published study (Paulay et al., 2012). The unit of analysis in meta-analysis is the effect size (Glass, 2015). The effect size is an index which states the magnitude of the influence of the independent variable on the dependent variable (Ellis, 2010). In this study, the effect size index uses the Hedges' g equation. The equation uses the standard mean difference is given as:

the degrees of freedom used in estimating Swithin and d is the Cohen’s d calculated using the equation and the variance of the Hedge’s g effect size Vg=J2xVd where Vd is the variance of Cohen’s d (Marfo &amp; Okyere, 2019). In addition to the calculations of the weighted mean effect size of Hedges' g with its standard errors and 95% confidence intervals around each mean, CMA was also used to test homogeneity by calculating Q, p, and I2 (Hillmayr et al., 2020). The interpretation of the calculated effect size using criteria (Cohen, Manion, &amp; Morrison, 2002) as follows:

· it is weak if between 0 and 0.20;

· it is small if between 0.21 and 0.50;

· it is moderate if between 0.51 and 1.00; and

· it is large if higher than 1

If a statistical test at a predetermined level of probability (p <0.05) is reached, the null hypothesis is rejected, and the alternative hypothesis is accepted (Dunst &amp; Hamby, 2012). Within meta-analysis, the most important decision concerns the use of a fixed-effects model or a random-effects model (Haidich, 2010). The fixed-effect model suggests that study effects are equivalent between different studies. Therefore, there is no need to analyze the study. The random-effect model was used if the effect size was statistically heterogeneous (Qb> χ2.95; p <0.05). This shows that the results of our analysis are statistically significant for different types of study characteristics (Bayraktar, 2001).

 <bold id="bold-c28a3f953b0e0a30b8fc1bc221238508">Findings and Discussion</bold>

The calculation results regarding the overall effect size in each study with the help of CMA software are shown in Table 3.

The effect size of each study

No Author Effect Size Standard Error Lower Limit Upper Limit
1 Khoiri (2013) 1.75 0.28 1.20 2.31
2 Nugroho, et al. (2013) 0.94 0.25 0.46 1.43
3 Azmi, et al. (2014) 0.95 0.26 0.45 1.45
4 Anwar (2015) 0.75 0.26 0.25 1.26
5 Katminingsih and Widodo (2015) 0.95 0.22 0.53 1.37
6 Kurniawati (2016) 0.00 0.22 -0.43 0.43
7 Nurqolbiah (2016) 0.24 0.23 -0.21 0.68
8 Rochani (2016) 0.71 0.25 0.22 1.20
9 Arhasy and Mulyani (2017) 1.01 0.24 0.55 1.48
10 Fahrudin (2017) 0.71 0.29 0.15 1.27
11 Septian and Rizkiandi (2017) 1.75 0.27 1.21 2.28
12 Astuti, et al. (2018) 3.22 0.49 2.27 4.18
13 Risnawati, et al. (2018) 1.11 0.14 0.82 1.39
14 Ahmad and Gunawan (2019) 0.08 0.28 -0.47 0.62
15 Alifiani, at al. (2019) 0.49 0.25 0.02 0.99
16 Indriani, et al. (2019) 0.24 0.26 -0.27 0.75
17 Masitoh and Prasetyawan (2019) 0.19 0.23 -0.25 0.63
18 Masitoh (2019) 0.92 0.29 0.36 1.49
19 Sefrinal (2019) 0.50 0.25 0.02 0.99

Table 3 shows that the mean effect size of each study with 95% confidence limit is in the range of 0.00 to 3.22. Based on the interpretation of effect sizes (Cohen et al., 2002), there were three studies that had weak effect sizes, four studies had small effect sizes, seven studies had moderate effect sizes, and five studies had large effect sizes.

The information presented in Table 4 shows that the fixed effects model with a 95% confidence level has a lower limit value of 0.669 and an upper limit of 0.886 with the combined effect size of 0.777. The size of this effect is moderate. Referring to the results of the homogeneity test using the Q statistic, the Q value is 81.175. Furthermore, the value of the Q table with a significance level of 5% and degrees of freedom of 17 is 28,869. In this case, Q> Q table, so it is concluded that the effect size of the implementation of PBL on students' creative mathematical thinking is heterogeneous. Thus, the estimation uses a random-effects model. In the random-effects model, the combined effect size was 0.821 (moderate) with a 95% confidence level. In the statistical significance test to answer the first question, the z test was used. Under the random-effects model, the z-value is 6,295, and p is 0,000. Because the p-value <0.05, it was concluded that the use of the PBL resulted in a more significant effect size on students' creative mathematical thinking than conventional approaches.

The results of the meta-analysis comparing the effects model

No Model n Z P Qb I squared (p=0.05) Effect size Standard error 95% Confidence Interval
Lower limit Upper limit
1 Fixed effect 19 14.001 0.000 95.617 81.175 0.777 0.056 0.669 0.886
2 Random effects 19 6.295 0.000 0.821 0.130 0.566 1.077

This finding is in line with the results of Susanti, Juandi, and Tamur (2020), who found an effect size of 0.79 when they synthesized the results from 12 articles comparing the effectiveness of PBL on mathematical communication skills. Similar results were found by Paloloang et al. (2020), who found an effect size of 0.83 when they analyzed the results of 14 articles comparing the effectiveness of PBL on students' mathematical literacy abilities. However, this study's overall size is different from the results of Asror's (2016) study, which reported an effect size of 0.51 when he analyzed the impact of PBL on students' creative mathematical thinking. In general, this research is supported by another finding (Mustaffa et al., 2014) that PBL positively impacts mathematics learning. This shows that PBL and other cooperative learning models have a moderate effect on mathematics learning, specifically on students' creative mathematical thinking.

Of the 19 studies, each had an average effect size. The characteristics of the studies were carried out including; year of study, education level, sample size, duration of the experiment and research publication. A summary of the analysis results based on the research characteristics is presented in Table 5. The interpretations of Table 5 are presented as follows.

Firstly, the analysis of research characteristics in Table 5 shows that PBL is associated with differences in the study years (Q = 55.68; P <0.05). PBL implementation was most effective (combined effect size = 1,294) between 2012 and 2013 and ineffective (combined effect size = 0.261) between 2018 and 2020. It appears that the combined effect size of each study group was different. Surprisingly, the combined effect sizes of the most recent study groups were so small. According to Bayraktar (2001), this kind of phenomenon implies the Hawthorne effect that first-time PBL has a significant impact on students' creative mathematical thinking, perhaps because of its novelty. Similar results were also reported by Masitoh and Prasetyawan (2019) that PBL had a big impact when it was first implemented. The result of Qb is 55,683 > Qtabel is 12.59 with a significance level of 0.05, meaning that there are significant differences between groups in the aspects of the research year.

A summary of analysis results based on the study characteristics

Study Characteristics Group Number Studies Hedge’s g Test of null (2 Tail) Heterogeneity
Z-value P Between Classes Effect (Qb) Df(Q) P
Study Year 2012-2013 2 1.294 6.943 0.000 55.683 6 0.000
2013-2014 1 0.949 3.722 0.000
2014-2015 4 0.871 7.280 0.000
2015-2016 2 0.115 0.723 0.470
2016-2017 2 1.255 6.333 0.000
2017-2018 4 1.064 9.574 0.000
2019-2020 4 0.261 2.090 0.037
Education Level Primary Schools 3 0.929 5.919 0.000 6.806 3 0.078
Secondary Schools 9 0.851 11.367 0.000
High Schools 3 0.475 3.384 0.001
Higher Education 4 0.705 5.234 0.000
Sample Size ≤30 5 0.675 5.076 0.000 0.711 1 0.399
≥31 14 0.799 13.086 0.000
Treatment Duration 3-4 meetings 3 1.116 7.293 0.000 5.644 1 0.018
>4 meetings 16 0.726 12.186 0.000
Publication Sources Thesis 3 1.218 5.782 0.000 13.812 3 0.037
Proceedings 2 0.889 6.949 0.000
Journal 14 0.765 11.224 0.000

Secondly, based on the level of education, it consists of four groups: higher education, high schools, secondary schools, and primary schools, each of which has an effect size of 0.705; 0.475; 0.851; 0.929 (moderate), respectively. The result of Qb is 6.806 < Qtabel is 7.82 with a significance level of 0.05. In this case, there is no significant difference between groups at the level of education. On the other hand, it was found that PBL has a higher positive effect on secondary schools than high schools (Asror, 2016). Meanwhile, the results of other meta-analyses regarding the effects of cooperative learning on student academic achievement show that cooperative learning is more effectively applied at the high school level since it is able to act independently and according to the role in the group (Prasetiyo et al., 2014). Although the models used are different, it is necessary to investigate these differences in results further.

Thirdly, based on the sample size, it consisted of two groups, ≤ 30 and ≥31, with effect sizes of 0.675 and 0.799 (moderate), respectively. Based on the results, Qb is 0.711 < Qtabel is 3.84 with a significant degree of 0.05. This shows that there is no significant difference between sample size groups. However, these findings differ from Tamur et al. (2020) that PBL is more effective in small samples. However, they focus on realistic problems. PBL and realistic problem-based learning have some similarities, but they produce different results. This difference in results may be due to the large number of studies synthesized in this study. For this reason, it is necessary to carry out further research involving more individual studies to be analyzed. This will provide more comprehensive results for educators and interested parties.

Fourthly, based on the duration of the treatment consisted of two groups, the treatment duration of 3-4 meetings had an effect size of 1,116 (large), and more than four meetings had an effect size of 0.726 (moderate). From the result, Qb is 5,644 > Qtabel is 3.84, there were significant differences between groups in the duration of the trials. This is also in line with the finding of Kadir, Milama, and Khairunnisa (2013) that PBL will be more effective if the treatment duration is not more than three weeks. Thus, problem-based learning is more effective for creative mathematical thinking if 3-4 meetings are held. This result could be partial because PBL requires more time to solve problems (Sanjaya, 2007). On the other hand, in implementing PBL, the students' initial ability factors, level and speed of thinking, and other heterogeneous aspects make teachers compromise with time (Tyas, 2017).

Fifthly, based on the publication sources, this study concluded that the results found in the thesis document (effect size = 1,218) were greater than those in the proceeding articles (effect size = 0.889) and the journal articles (effect size = 0.765). This variable is analyzed to check whether selective reporting factors give rise to the effect of publication bias. The heterogeneity test results showed that the combined effect sizes of the three study groups were different. However, they showed no signs of bias. The reasons why the combined effect size of the thesis is much larger than that of proceedings or journals are still unclear.

The current research shows that PBL has a positive impact on students' creative thinking in mathematics. This study's effect size is consistent with previous studies' findings (e.g., Susanti et al., 2020; Paloloang et al., 2020), but the findings of this study contribute to the literature on study characteristics that need to be considered in implementing PBL. Although it has been suggested that teachers need convincing information regarding PBL implementation, this finding is based only on the primary studies that met the inclusion criteria. Further studies are still needed to be analyzed to gather relevant information. Overall, these findings do not describe the effectiveness of PBL on students' mathematical creativity.

 <bold id="bold-d938f0b72883f6dc61b6777bdc5f8e2a">Conclusion</bold>

This meta-analysis aimed to analyze the effectiveness of using PBL on students’ creative thinking in mathematics. The overall effect size was 0.821, with a standard error of 0.130 and 95% degrees of freedom. This shows that PBL has a moderate effect on students' creative mathematical thinking. The results of the studies’ analysis show that the application of PBL to the students' creative thinking relates to the study year, research class, treatment duration, and publication sources. It was found that the PBL is more effective in conditions of treatment duration of fewer than four meetings. There is a tendency that the implementation of PBL for the first time has a significant impact on students' creative thinking. Besides, the results of the analysis also show variations in the effect sizes according to the published sources. Although this does not indicate a publication bias, the hypothesis that the results of a significant study are likely to be published needs to be tested again. For this reason, more studies need to be carried out in this area to provide a complete picture to mathematics educators about the effectiveness of PBL in mathematics classrooms.

 <bold id="bold-f02d4f579dfeb4f8b8cb26a515b33baf">Acknowledgment</bold>

The authors thank M. Borenstein and the CMA team from Englewood, the USA, for all their technical assistance and contributions to the research project.

 Abramo, G., Ciriaco, A., Angelo, D., & Grilli, L. (2016). From rankings to funnel plots: The question of accounting for uncertainty when assessing university research performance. Journal of Informetric, 10(3), 854–862. Doi: 10.1016/j.joi.2016.07.005 Ahmad, A., & Gunawan, G. (2019). The impact of problem-based learning toward the students’ creative thinking in complex function subject. Paper presented at International Seminar on Applied Mathematics and Mathematics Education. Retrieved from https://iopscience.iop.org/article/10.1088/1742-6596/1315/1/012020/pdf Alifiani, I., Dwijanto, D., Cahyono, A. N. (2019). Mathematical creative thinking ability viewed by self-esteem in problem-based learning with open ended approach. Unnes Journal of Mathematics Education Research, 8(2), 195—202. Anugraheni, I. (2018). Meta-analisis model pembelajaran problem-based learning dalam meningkatkan keterampilan berpikir kritis disekolah dasar. Polyglot: Jurnal Ilmiah 14(1), 9-18. Doi: 10.19166/pji.v14i1.789 Anwar, M. S. (2015). Efektifitas pembelajaran PBL (problem-based learning) terhadap kemampuan berpikir kritis dan kreatif siswa materi tiga dimensi (Master Thesis). Retrieved from http://repository.ut.ac.id/6912/1/42243.pdf Arhasy, E. A., & Mulyani, E. (2017). Kontribusi model problem-based learning berbantuan media software maple terhadap kemampuan berpikir kreatif matematis dan self-regulated learning mahasiswa. Jurnal Siliwangi, 3(1), 197-203. Asror, A. H. (2016). Meta-analisis: PBL. PRISMA (Prosiding Seminar Nasional Matematika). Retrieved from https://journal.unnes.ac.id/sju/index.php/prisma/article/view/21682 Astuti, W. P., Wahyudi., & Indarini E. (2018). Efektifan model pembelajaran problem-based learning dan problem solving terhdap kemampuan berpikir kreatif matematika. Jurnal Ilmiah Pendidikan & Pembelajaran, 2(2), 159-166. Doi: 10.23887/jipp.v2i2.15349 Azmi, B. M., Irzani., & Khusnial, N. L. (2014). Efektifitas strategi problem based learning (PBL) terhadap kemampuan berpikir kreatif peserta didik. Beta: Jurnal Tadris Matematika, 7(2), 108-119. Bayraktar, S. (2001). A meta-analysis of the effectiveness of computer-assisted instruction in science education. Journal of Research on Technology in Education, 34(2), 173–188. Doi: 10.1080/15391523.2001.10782344 Bochniak, J. S. (2014). The effectiveness of computer-aided instruction on math fact fluency [Walden University]. Retrieved from http://scholarworks.waldenu.edu/cgi/viewcontent.cgi?article=1028&context=dissertations. Borenstein, M., Hedges, L. V, Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. John Wiley and Sons, Ltd. Doi: 10.1002/9780470743386 Ceker, E., & Ozdamli, F. (2016). Features and characteristics of problem-based learning, Cypriot Journal of Educational Sciences, 11(4), 195-202. Retrieved from https://files.eric.ed.gov/fulltext/EJ1140792.pdf. Cleophas, T.J., & Zwinderman, A. H. (2017). Modern meta-analysis review and update of methodologies. Swiss: Springer International Publishing. Cohen, L., Manion, L., & Morrison, K. (2002). Research methods in education. New York: Roultedge Taylor & Francis Group. Cumming, G. (2012). Understanding the new statistics effect sizes, confidence intervals, and meta-analysis. London, New York: Routledge. Dunst, C. J., & Hamby, D. W. (2012). Guide for calculating and interpreting effect sizes and confidence intervals in intellectual and development disability research studied. Journal of Intellectual & Developmental Disability, 37(2), 89-99. Doi: 10.3109/13668250.2012.673575 Ellis, P. D. (2010). The essential guide to effect sizes. Cambridge: Cambridge University Press. Doi: 10.1017/cbo9780511761676 Fahrudin, F. A. (2017). Efektivitas problem based learning terhadap kemampuan berpikir kreatif mahasiswa program studi tadris matematika UIN Mataram. Jurnal Teori dan Aplikasi Matematika, 1(1), 41-48. Doi: 10.31764/jtam.v1i1.185 Fauzi, K. M. A., Dirgeyase, I. W., & Priyatno, A. (2019). Building learning path of mathematical creative thinking of junior students on geometry topics by implementing metacognitive approach. International Education Studies, 12(2), 57. Doi: 10.5539/ies.v12n2p57 Fragkos, K. C., Tsagris, M., & Frangos, C. C. (2016). Exploring the distribution for the estimator of Rosenthal’s “fail-safe” number of unpublished studies in meta-analysis. Communications in Statistics- Theory and Methods, 46(11), 5672-5684. Doi: 10.1080/03610926.2015.1109664 Freeman-Green, S. M., O’Brien, C., Wood, C. L., & Hitt, S. B. (2015). Effects of the SOLVE strategy on the mathematical problem solving skills of secondary students with learning disabilities. Learning Disabilities Research and Practice, 30(2), 76–90. Doi: 10.1111/ldrp.12054 Glass, G. V. (2015). Meta-analysis at middle age: A personal history. Research Synthesis Methods, 6(3), 221–231. Doi: 10.1002/jrsm.1133 Haidich, A. B. (2010). Meta-analysis in medical research. Hippokratia, 14(Supp 1), 29–37. Retrieved from https://pubmed.ncbi.nlm.nih.gov/21487488/ Higgins, S., & Katsipataki, M. (2015). Evidence from meta-analysis about parental involvement in education which supports their children’s learning. Journal of Children’s Services, 10(3), 280–290. Doi: 10.1108/JCS-02-2015-0009 Hillmayr, D., Ziernwarld, L., Reinhold, F., Hofer, S. I., & Reiss, K. M. (2020). The potential of digital tools to enhance mathematics and science learning in secondary school: a context-specific meta-analysis. Computer & Education, 1-85. Doi: 10.1016/j.compedu.2020.103897 Indriani, D. S., Widyasari, W., Amril, L. O. (2019). Pengaruh model pembelajaran berbasis masalah terhadap kemampuan berpikir kreatif matematis siswa pada materi bilangan pecahan kelas IV SDIT At-Thoriq. E-Journal Skripsi: Fakultas Keguruan dan Ilmu Pendidikan, 2(3), 1-21. Ismail, S. N., Don, Y., Husin. F., & Khalid. R (2018). Instructional leadership and teachers’ functional competency across the 21st century learning. International Journal of Instruction, 11(3), 135-152. Doi: 10.12973/iji.2018.11310a Kadir., Milama. B., & Khairunnisa. (2013). Meta-analisis efektivitas pendekatan problem solving dalam pembelajaran sains dan matematika. Ciputat, Indonesia: Lembaga Penelitian UIN Syarif Hidayatullah Jakarta. Katminingsih, Y., & Widodo, S. (2015). Pengaruh model pembelajaran berdasarkan masalah terhadap kemampuan berpikir kreatif matematis siswa ditinjau menurut gender siswa SD Negeri Tarokan Kediri. Jurnal Math Educator Nusantara: Wahana Publikasi Karya Tulis Ilmiah di Bidang Pendidikan Matematika, 1(1), 77-89. Khoiri, W. (2013). Implementasi model problem-based learning berbantuan multimedia untuk meningkatkan kemampuan berpikir kreatif siswa kelas VII SMP Negeri 4 Kudus pada materi segitiga (Undergraduate thesis). Retrieved from https://lib.unnes.ac.id/18832/1/4101409045.pdf. Kulik, C. C., Kulik, J. A., & Shwalb, B. J. (1986). The effectiveness of computer-based adult education: A meta-analysis. Journal of Educational Computing Research, 2(2), 235–252. Doi: 10.2190/02hm-xcwg-q1vy-5emq Kurniawati. (2016). Pengaruh model project-based learning (PjBL) dan problem-based learning (PBL) terhadap kemampuan berpikir kreatif siswa (Master Thesis). Retrieved from http://repository.ut.ac.id/6724/1/42227.pdf. Langan, D., Higgins, J. P. T., Gregory, W., & Sutton, A. J. (2012). Graphical augmentations to the funnel plot assess the impact of additional evidence on a meta-analysis. Journal of Clinical Epidemiology, 65(5), 511–519. Doi: 10.1016/j.jclinepi.2011.10.009 Marfo, P., & Okyere, G. A. (2019). The accuracy of effect-size estimates under normals and contaminated normals in meta-analysis. Heliyon, 5(6), e01838. Doi: 10.1016/j.heliyon.2019.e01838. Masitoh, L. F. (2019). The effectiveness of problem-based learning (PBL) approach viewed from the students’ mathematical creative thinking ability. Jurnal Pendidikan Matematika Indonesia, 4(2), 47-52. Doi: 10.26737/jpmi.v4i2.699 Masitoh, L. F., & Prasetyawan, E. (2019). The effectiveness of scientific approach with open-ended problem based learning worksheet viewed from learning achievement, creative thinking ability, interest, and mathematics self-efficacy. Daya Matematis: Jurnal Inovasi Pendidikan Matematika, 7(3), 292-308. Maskur, R., Sumarno, Rahmawati, Y., Pradana, K., Syazali, M., Septian, A., & Palupi, E. K. (2020). The effectiveness of problem based learning and aptitude treatment interaction in improving mathematical creative thinking skills on curriculum 2013. European Journal of Educational Research, 9(1), 375–383. Doi: 10.12973/eu-jer.9.1.375 Mustaffa, N., Ismail, Z., Tasir, Z., Said, M. N. H. B. M. (2014). Problem-based learning (PBL) in mathematics: A meta-analysis. Paper presented at the 1st International Education Postgraduate Seminar, Johor Bahru-Malaysia. Retrieved from http://eprints.utm.my/id/eprint/60893/ Mustofa, R. F.,& Hidayah. Y. R. (2020). The effect of problem-based learning on lateral thinking skills. International Journal of Instruction, 13(1), 463-474. Doi: 10.29333/iji.2020.13130a Nugroho, I., Chotim, M., & Dwijanto, D. (2013). Keefektifan pendekatan problem-based learning terhadap kemampuan berpikir kreatif matematik. Unnes Journal of Mathematucs Education, 2(1). Doi: 10.15294/ujme.v2i1.3319 Nuha, M. A., Waluya, S. B., & Junaedi, I. (2018). Mathematical creative process Wallas model in students problem posing with lesson study approach. International Journal of Instruction, 11(2), 527–538. Doi: 10.12973/iji.2018.11236a Nurqolbiah, S. (2016). Peningkatan kemampuan pemecahan masalah, berpikir kreatif, dan self-confidence siswa melalui model pembelajaran berbasis masalah. Jurnal Penelitian Pendidikan dan Pengajaran Matematika, 2(2), 143-158. Paloloang, M. F. B., Juandi, D., Tamur, M., Paloloang, B., & Adem, A. M. G. (2020). Meta analisis: pengaruh problem-based learning terhadap kemampuan literasi matematis siswa di Indonesia tujuh tahun terakhir. AKSIOMA: Jurnal Program Studi Pendidikan Matematika, 9(4), 851–864. Doi: 10.24127/ajpm.v9i4.3049 Paulay, C., Luginnbuhl, W., Ampuero, S., & Bee, G. (2012). Expected effects on carcasss and pork quality when surgical castration is omitted-results of a meta-analysis study. Meat Sciece, 92(4), 858-862. Doi: 10.1016/j.meatsci.2012.06.007 Pigott, T. D., & Polanin, J. R. (2020). Methodological guidance paper: High-quality meta-analysis in a systematic review. Review of Educational Research, 90(1), 24–46. Doi: 10.3102/0034654319877153 Prasetiyo, A. Y., Yusmin, E., & Hartoyo, A. (2014). Meta-analisis pengaruh cooperative learning terhadap peningkata hasil belajar matematika siswa. Jurnal Pendidikan dan Pembelajaran Khatulistiwa, 3(2), 1-11. Risnawati, Amir, Z., Lubis, M. S., & Syafri, M. (2018). The effect of problem-based learning model (PBL) towards creative thinking ability and self-efficacy of junior high school students in Pekanbaru. Journal of Physics: Conference Series, 1116(2). Doi: 10.1088/1742-6596/1116/2/022039 Rochani, S. (2016). Keefektifan pembelajaran matematika berbasis masalah dan penemuan terbimbing ditinjau dari hasil belajar kognitif kemampuan berpikir kreatif. Jurnal Riset Pendidikan Matematika, 3(2), 273-283. Doi: 10.21831/jrpm.v3i2.5722 Rothstein, H. R., & Hopewell, S. (2009). Grey literature. In H. Cooper, L. V Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 103–125). New York: Russell Sage Foundation. Sanjaya, W. (2007). Strategi pembelajaran berorientasi standar. Jakarta: Kencana, Prenada Media Group. Schoevers, E. M., Leseman, P. P. M., Slot, E. M., Bakker, A., Keijzer, R., & Kroesbergen, E. H. (2019). Promoting pupil’s creative thinking in primary school mathematics: A case study. Thinking Skills and Creativity, 31, 323-334. Doi: 10.1016/j.tsc. 2019.02.003 Schuch, F. B., Vancampfort, D., Richards, J., Rosenbaum, S., Ward, P. B., & Stubbs, B. (2016). Exercise as a treatment for depression: a meta-analysis adjusting for publication bias. Journal of Psychiatric Research, 42–51. Doi: 10.1016/j.jpsychires.2016.02.023 Sefrinal. (2019). Pengaruh pendekatan problem-based learning terhadap kemampuan berpikir kreatif matematis siswa kelas X SMA Negeri 1 Sutera. Jurnal Kepemimpinan dan Pengurusan Sekolah, 4(2), 149-154. Doi: 10.34125/kp.v412.417 Septian, A., & Rizkiandi, R. (2017). Penerapan model problem-based learning (PBL) terhadap peningkatan kemampuan berpikir kreatif matematis siswa, Jurnal Prisma, 6(1), 1-8. Doi: 10.35194/jp.v6i1.22 Servant-Miklos, V.F.C. (2018). Fifty years on: A retrospective on the world’s first problem-based learning program at McMaster university medical school. Health Professions Education, 5(1), 3-12. Doi: 10.1016/j.hpe.2018.04.002 Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annual Review of Psychology, 70(1), 747–770. Doi: 10.1146/annurev-psych-010418-102803 Sugiharto, B., Corebima, A. D., Susilo, Herawati & Ibrohim. (2019). The pre-service biology teacher readiness in blended collaborative problem-based learning (BCPBL). International Journal of Instruction, 12(4), 113-130. Doi: 10.29333/iji.2019.1248a Susanti, N., Juandi, D., & Tamur, M. (2020). The effect of problem-based learning (PBL) model on mathematical communication skills of junior high school students – A meta-analysis study. JTAM (Jurnal Teori dan Aplikasi Matematika), 4(2), 145–154. Doi: 10.31764/jtam.v4i2.2481 Tamur, M., Juandi, D., & Kusumah, Y. S. (2020). The effectiveness of the application of mathematical software in Indonesia; A meta-analysis study. International Journal of Instruction, 13(4), 867–884. Doi: 10.29333/iji.2020.13453a Tamur, M., Juandi, D., & Adem, A. M. G. (2020). Realistic mathematics education in Indonesia and recommendations for future implementation: A meta-analysis study. Jurnal Teori dan Aplikasi Matematika, 4(1), 17–27. Doi: 10.31764/jtam.v4i1.1786 Thorndahl, K. L., & Stentoft, D. (2020). Thinking critically about critical thinking and problem-based learning in higher education: A scoping review. Interdisciplinary Journal of Problem-Based Learning, 14(1). Doi: 10.14434/ijpbl.v14i1.28773 Tyas, R. (2017). Kesulitan penerapan problem based learning dalam pembelajaran matematika. Tecnoscienza, 2(1), 43–52. Yaakub, M. N. (1998). Meta-analysis of the effectiveness of computer-assisted instruction in technical education and training [Virginia Polytechnic Institute and State University]. Retrieved from https://vtechworks.lib.vt.edu/handle/10919/30651 Yanto, F., Festiyed, F., Iswri, M., & Enjoni, E. (2019). Meta-analysis: Improving creativity through assessment in a problem-based learning environment. 178(ICoIE 2018), 23–26. Doi: 10.2991/icoie-18.2019.6