How many post hoc t-tests should I perform and how do i determine my sample size based on them?

Question

I have to complete my dissertation in a couple of days. I am proposing a RCT with two independent variables: treatment( group 1 : avatar cognitive therapy , group 2: CT) and time(pre-test, post-test and follow-up). I also have 2 dependent variables(severity and power beliefs) I have no actual data since this is a proposal, but I need help proposing the statistical analysis. So I will perform two mixed anova(2X3) tests to see the effects of my intervention on each outcome.

1. How many post hoc pairwise comparisons do I need to do ? my tutor said something about t test between subject at post-test and follow up(so these are 2 tests?) and t tests within subject to see how do the effects evolve in time( so 3 tests? 1 testing pre-post , one post-follow up and one pre-follow up?
2. If so, how do i calculate sample size a-priori with bonferroni correction? (estimated effect size, based on previous research would be 0.8) Thank you !!!

Answer 1

Your 2 (Treatment: Avatar CT vs. Standard CT) × 3 (Time: Pre, Post, Follow-up) which is a 2 × 3 mixed-design factorial ANOVA Is correct.

Post hoc tests are necessary only if your ANOVA yields a statistically significant interaction effect (Treatment × Time).

Mention in the report that you will apply a Bonferroni correction to protect against Type I error (false positives). Your adjusted alpha level per test will be 0.05 / 5 = 0.01.

An a-priori power analysis conducted in GPower for a large effect size (d = 0.8, f = 0.4), alpha = .01, and power = .80 indicated a total required sample size of N = [insert exact G*Power output] (N= [N/2] per arm).

If the Interaction Effect is significant, you will conduct 5 pairwise comparisons per outcome variable to locate the differences. Specifically, independent samples t-tests will compare groups at post-test and follow-up, and paired samples t-tests will examine within-subject changes across the three time points for each group.

For a Between-Subjects Comparisons (2 tests per ANOVA)

You want to see if the two treatment groups differ after the intervention starts. You skip the pre-test because random assignment assumes groups are equal at baseline.

Test 1: Avatar CT vs. Standard CT at Post-test

Test 2: Avatar CT vs. Standard CT at Follow-up

Within-Subjects Comparisons (3 tests per ANOVA)

You want to see how each group changes over time. You must run these 3 time-points for Group 1 and then repeat them for Group 2.

Test 3 & 4 (Pre to Post): Changes right after therapy for Avatar CT, and separately for Standard CT.

Test 5 & 6 (Post to Follow-up): To see if therapeutic gains are maintained for each group.

Test 7 & 8 (Pre to Follow-up): To see long-term overall improvement for each group.