A verb +てみる means to try to do something new, so you can't use this construction if you have done "something" even once. On the other hand, verb +ようとするmeans "You are about to do something", "Not something new".
E.g. すしをたべてみる。→I will try sushi. (Eating sushi is his or her first experience.)
E.g. すしをたべようとする。→I am going to eat sushi. (He or she is about to eat sushi. (Eating sushi is "not" his or her first experience.))