Rather is used when talking about an action.
Example: I would rather.....[insert verb]......than......[insert verb]....
Prefer is used when stating a persons preference of a person place or thing. Example: I prefer.......[insert noun]
I prefer blue to green. vs I would rather blue to green.
I would rather swim than hike. vs I prefer swim to hike.
Note that swim (a verb) can be changed to the noun swimming (a thing one can do in water) such that
I prefer swimming to hiking.
You can prove that swimming and hiking are nouns here by putting them with rather.
I would rather swimming than hiking.
This "rather" sentence is incorrect, although someone might infer that you must mean "rather GO swimming than GO hiking.