She can choose as follows:
6 ways to choose a book on matrices * 1 way to choose 3 books on probability +
15 ways to choose 2 books on matrices * 3 ways to chose 2 books on probability +
20 ways to choose 3 books on matrices * 3 ways to choose a book on probability
you do the arithmetic to get the total.
BTW the numbers above are the products of binomial coefficients...the numerators in the hypergeometric distribution. If you divide your total by 9C4, you will get the probability of choosing 4 books such that there is at least one from each group.