If, on the first observation the system is in state 1, what is the probability that it is in state 1 on the third observation?

Question

A Markov chain has the transition matrix shown below:
P=[0.2  0.4]

    [0.8  0.6]

 

1. If, on the first observation the system is in state 1, what is the probability that it is in state 1 on the third observation?

 

2. If, on the first observation, the system is in state 2, what state is the system most likely to occupy on the third observation? (If there is more than one such state, which is the first one.) 

 

3. If, on the first observation, the system is in state 2, what is the probability that it alternates between states 1 and 2 for the first four observations (i.e., it occupies state 2, then state 1, then state 2, and finally state 1 again)?

 

Answer 1

When I learned Markov Chains, the rows were the ones representing the current state and the columns represented the next one, which is the traditional representation. Your matrix P is the transpose of that format, but it doesn't affect analysis much. Here it goes.

 

1. You need P² to solve this part.

 

P² =

 

⌈0.2  0.4⌉ ⌈0.2  0.4⌉

⌊0.8  0.6⌋ ⌊0.8  0.6⌋ =

 

⌈0.36  0.32⌉
⌊0.64  0.68⌋

 

So is there is a 0.36 chance that you return to state 1.

 

2. From the P² matrix we computed, it is most likely to return to state 2 since it has a 0.68 chance of doing so.

 

3. We now only look at matrix P. The first transition must be to state 1, which has 0.4 chance, then to state 2, which has 0.8 chance, and back to state 1 which has 0.4 chance.

 

Thus the answer is (0.4)(0.8)(0.4) = 0.128 chance.