Hi Fikri!
This is one of those problems where I think doing the second part first might lead to a better understanding of what a system is, which seems to be a goal of the problem.
First, let's define a system that includes only the upper block. Defining a system allows us to define internal (totally inside the system) and external (from outside the system) forces. Only external forces affect the system's net acceleration, a, and are what we use in Newton's second law (Fnet = ma)
If the top block of mass m is not to slip, it must accelerate with the lower block in the horizontal direction. For that, it needs a force acting on it in the horizontal direction. What force could this be for the top block? The only force that can act on the top block in the horizontal direction, if it is not slipping, is static friction (static, because the surfaces of the two blocks are not in motion with respect to each other). Remember that the string is only attached to the lower block, and thus does not apply a direct force to the upper block (our system). That force is transmitted to the upper block through friction.
The expression for the static friction force is: Fs ≤ μsN
where
μs = coefficient of static friction
N = normal force
We are interested in the maximum possible force we can exert on the blocks without the top block slipping. That means we are interested in the maximum static friction force, when Fs = μsN
So let's set up Newton's second law, in the horizontal direction (of motion), for our top block system in the case that we are exerting the maximum possible force without the top block slipping:
Fnet-x = Fs = ma --> μsN = ma --> (0.350)(N) = (5 kg)a --> a = (0.350)(N)/(5 kg)
This represents the maximum possible acceleration the top block can attain without slipping, and is the desired answer for Part (b). We can solve for that directly, once we have an expression for the normal force on the block (the normal force determines how tightly the surfaces in contact press together, which is part of what determines the magnitude of the friction force).
To solve for N, we have to look in the vertical direction, where N is. In this direction, we have a normal force (up) on the top block, and gravity (mg) down. So Newton's second law in that direction becomes (with up defined as positive):
Fnet-y = N - mg = ma = 0 (since we are assuming no net vertical acceleration on the upper block)
So N - mg = 0 --> N = mg = (5 kg)(9.8 m/s2) = 49 N
Now, a = (0.350)(49 N)/(5 kg) = 3.43 m/s2 [EDIT: This confirms Arturo's calculation above; not that it needed confirmation :) ]
This is the maximum possible (horizontal) acceleration of the top block without slipping, since it comes from the maximum value of the only force capable of accelerating the top block (system) horizontally.
********************************************
Now, let's answer the first question second.
Here is where redefining the system (which we are allowed to do at any point in a problem) can make things easier. Let's now define a system consisting of both the top and bottom blocks together (I sometimes refer to this as "wrapping our system bubble around," in this case, both blocks). We can do this as long as we know that acceleration of the two block is the same, which is the situation we are dealing with here where the blocks move together.
By doing this, we simplify the horizontal direction for this problem. With both blocks inside the system, the only horizontal external force acting on the system -- and thus the only force we have to put in Newton's second law in the horizontal direction -- is the pulling force of the string (Fp). The frictional force between the blocks is internal, and can be left out of Newton's second law. In this case, Newton's second law in the horizontal becomes:
Fnet-x = Fp = msysa
And we know that a has to be the maximum acceleration we already calculated (so that the upper block does not slip). The mass of the system (msys) is just the total mass inside the system [NOTE: I did not make a point of calling it msys when the upper block alone was the system, since msys = m in that case]. Since the system is both blocks, msys = (m+M) = (5 kg + 10 kg) = 15 kg.
So, the previous equation becomes:
Fp = (15 kg)(3.43 m/s2) = 51.45 N [EDIT: again, confirming Arturo's result]
This is thus the maximum force we could pull on the string and not have the blocks slip.
In problems like this, I have always found it helpful to define a system that makes Newton's second law as simple as possible, while still making the force(s) associated with the quantity I am trying to solve for external.
Hope this helps! It is just another way to look at the problem.