Hi Atharva,
My first step would be to graph these. Hopefully you've already done that. Second step is to solve the two equations x2 - 5 = x and x2 - 2 = 5. These give us
x = 1/2 ± √(21/4) ≈ -1.79, 2.79
x = 1/2 ± 3/2 = -1, 2
Therefore, we're expecting solutions in the rough area of -1.79 to -1 and from 2 to 2.79. Now, let's look closely at how our greatest integer function behaves in those areas, starting with the positive. For x values between 2 and 2.999999..., [x] = 2. We see that that horizontal line intercepts y = x2-2 at 2. In order to figure out where the right end of that line intercepts y = x2-5, we solve x2-5 = 2 to get that x = √7. So, we have a solution set on the bound:
2 < x < √7
Now we need to consider what the function does on either side of this. At x = 3, [x] jumps up to 3, but if we evaluate x2-5 at 3, we note that curve is already at 4, so there are no further solutions to the right of this interval.
To the left of x = 2, [x] drops down to 1, so there will be another interval for x < 2. To find the left bound of this interval, we set x2-2 = 1 and get x = √3. So, we have a solution on the bound:
√3 < x < 2
Note that 2 isn't a solution, because then [x] = x2-2, and the initial problem had strict inequalities.
For the negative solutions, we note that between -2 ≤ x < -1, [x] = -2. So, we have a solution set with -1 as the right bound. To find the left bound, we set x2-5 = -2 to get x = ±√3. It's the negative solution that applies here, so we have a solution set on:
-√3 < x < -1
We can note that for x < -√3, [x] stays below the lower curve and that at x = -1, [x] jumps above the upper curve and stays there until the positive intervals we already found. So our complete solution set is:
-√3 < x < -1
√3 < x < 2
2 < x < √7
If you have additional questions on this, feel free to let me know.
