Mastering is often seen as a sort of "dark art" in the sense that it makes things sound better, but a lot of us aren't quite sure what it involves, or what the mastering engineer is doing. To me, mastering is a combination of philosophy and technical engineering, which I'll dive into below.
Whenever I begin mastering a song or collection of songs, I listen carefully to the music and try to figure out what the artist & mix engineer were aiming for. Words like "bright, smooth, aggressive, dark" etc. are different feelings that can all benefit from specific tools in achieving that vibe, and usually the music is already leaning in a particular direction sonically. My philosophy is that I'm here to enhance the original vision of the artist & mix engineer, and not to imprint my own sonic vision. I usually listen to a reference track (something that sounds great and is in the ballpark of how I want the final product to sound) and import that into my session as well, so I have a sort of grounding, or home base, sonically speaking.
The next part I'll describe is the technical process - by no means is this the exact sequence I use every time, but I've found this to be a common chain of processors/plugins that I use.
I generally begin with subtractive EQ. This is usually a bell curve with a narrow bandwidth, where I sweep up and down the frequency spectrum, listening for problem frequencies and unpleasant peaks. When I hear something that sounds harsh, tinny, or boomy, I pull that specific frequency down a bit, keeping the narrow bandwidth of the bell curve. I repeat this process a handful of times, and A/B my song with & without the EQ, so make sure I've cleaned up my song and haven't removed too much information.
The next step for me is often compression/saturation. I will run my track through a compressor with a slow attack and quick-medium release, to maintain the integrity of transient material in my song (usually drums). Sometimes compressors have soft clip features, which can have a pleasing result, or sometimes I will add a separate saturation plugin, usually around 5% to 20% wet (this is called parallel processing). I aim for anywhere from 2-5 dB of gain reduction with my compressor, depending on the material and how loud the mix was to begin with.
Next I will apply more EQ, but as a tonal shaper, and not as a surgical problem solver. In layman's terms, I'm trying to make broad strokes to make my song brighter overall, or bassier overall, or less mid-rangey overall, depending on what it needs. This looks like a bell curve with a wide bandwidth, or a shelf EQ. Now is a good time to listen to my reference track again to gauge where my song is tonally, and what it could use more or less of. Usually I will also apply a high-pass curve around 20-35 Hz, to reduce the super low end information that will not be audible on most playback systems.
I will usually apply more compression/saturation here, and maybe some OTT set around 5% to 10% wet (it's a free multi-band compressor, easy to find online). OTT, in small amounts, can make things brighter and a bit more exciting.
Lastly, I apply my master limiter. This is where we can push the volume a bit more and make sure that our song isn't clipping, or letting any stray peaks pass our ceiling. Many limiters now have an RMS or LUFS meter built in, which is very helpful in measuring the perceived loudness of your music. I would reference some different resources online about the best practices for target LUFS/RMS in regards to the different streaming platforms out there.
Throughout this whole process, I'm frequently A/B'ing the song with & without the plugins that I'm applying, to make sure that I'm actually improving the sound. When I'm done, I will export my master and listen over the next day or two on a variety of playback systems (phones, laptops, cars, home speakers etc.) to make sure I'm satisfied with how it's translating across the board. 9 times of 10 I will go back to tweak the master and make adjustment based on what I'm hearing.
This is just my personal process, but I've learned from some excellent teachers. I'm sure I omitted some finer details, but this should be a good place to start if you're intimidated by mastering on unsure of how to begin. Hope this helps & good luck!