Dynamic Programming- Finding Optimal research and development policy.

Question

A software manufacturer can be in one of two states. In state 1 their software sells well, and in state 2, the product sells poorly. While in state 1, the company can invest in development of upgraded version of the software, in which case the one-stage reward is 4 units, and the probability of degrading to state 2 is 0.2. If no investment in new development occurs, then the reward is 6 units, but the probability of transition to state 2 is 0.5. While in state 2, if the company invests in software development, then the reward is -2 units, but the probability of transition to state 1 is 0.7. Without special efforts to improve, the reward is 1 and the probability of upgrading to state 1 is 0.

Formulate a dynamic programming problem to determine an optimal research and development policy. Solve the problem for a time horizon of 12 time intervals.

Answer 1

The long term expected returns are

State 1:

Invest: 4* 0.8 - 4 * 0.2= 2.4

Not invest: 6* 0.5 -6*0.5=0

State 2:

Invest : 0.7*2 - 0.3*2= 0.8

Not invest: 1*0 -1*1= -1