You are right.
Displacement is the net change in an object's position, as measured from the reference (starting) point. In this case the bicyclist rode eight meters east, then back to the west for three meters to get to a spot that is five meters east of the reference point. His net movement from the starting point is five meters.
Time is not relevant here because we are not measuring speed, velocity, or acceleration.