For this type of problems involving composit functions of the form, f of f of f of ..of(x), trick is to look for the repeating pattern after evaluating a first few compositions.
f2(x) = f of f(x) =f((x√3 - 1) / (x + √3)) = [√3(x√3 - 1) -(x + √3)]/[(x√3 - 1) +√3(x + √3)]
=(3x-√3 -x -√3)/(x√3 -1 + x√3 + 3) =( 2x -2√3)/(2x√3 +2) = (x-√3)/(√3 x +1)
f3(x) = f(f2(x))=[√3(x-√3) -(√3x+1)]/[(x-√3) + √3(√3x+1)] = (-3 -1)/(x + 3x) = -1/x
f4(x) = f(f3(x)) = (-√3/x -1)/(-1/x + √3) =- (x+√3)/( √3 x -1)
f5(x) = f(f4(x)) = [-√3(x+√3) -(√3x -1)]/[-(x+√3)+√3(√3x -1)] = -(√3 x + 1)/(x - √3)
f6(x) = f(f5(x)) =[-√3(√3 x + 1) -(x-√3)]/[-(√3 x +1) + √3(x-√3)] = [-3x -√3 -x +√3]/[-√3x -1 +√3 x -3] =-4x/(-4) =x
f7(x) = f(f6(x)) = f(x)
i.e. after every 6 compositions the function repeats itself
ie. f(x) = f7(x) =f13(x) =f19(x) =..
x =f6(x) = f12(x) =f18(x) = ...
1998 = 333 * 6.
g(x) = f1998(x) = x.