Hi there, I think I can help answer your question. I have been playing guitar for 10+ and am currently working as a private instructor. The short answer to your question is yes, memorizing and learning one key at a time is probably the best way to eventually memorize and learn all of your keys.
The longer answer to your question is that there are basically two systems of music theory you are going to want to not just learn, but memorize. I am going to type out a lot but keep in mind that you should go slowly and not bite off more than you can chew at first. I think starting with the key of E major on guitar is a great place to start! I will use that key as an example.
The first system I am going to call the "letter system" is that which contains specific notes.
Key of E Major contains 4 sharps, F#, G#, C#, and D#
E major scale: E F# G# A B C# D#
The chords in E Major are: E, F#min, G#min, A, B, C#min, D#dim
Key of E minor contains 1 sharp: F#
E minor scale: E F# G A B C D
The chords in E Minor are: Emin, F#dim, G, Amin, Bmin, Cmaj, Dmaj
The "letter system" is one you are probably somewhat familiar with and have used before.
The second system I am going to call the "number system" because we use numbers for the intervals and chord qualities in any major or minor key. These numbers are called "scale degrees" and we use Arabic numerals for notes and Roman numerals for chords.
Major Scale Degrees: 1 2 3 4 5 6 7
Major Scale Chord Qualities: Imaj IImin IIImin IVmaj Vmaj VImin VIIdim
Minor Scale Degrees: 1 2 b3 4 5 b6 b7
Minor Scale Chord Qualities: Imin IIdim IIImaj IVmin Vmin VImaj VIImaj
The reason that the "number system" is so useful is because it works for ANY key major or minor respectively. The Major Scale Degrees and Chord Qualities that I listed are the exact same for EVERY major key. The Minor Scale Degrees and Chord Qualities that I listed are the exact same for EVERY minor key. It's important to keep this system in mind as you don't necessarily need to memorize every single note and chord in every single key right away. Sure, it's useful to do that eventually, but knowing this "number system" will help a lot early on and will continue to help as you make your way through the keys.
In terms of specifically learning on guitar, start by learning your notes on the low E and A strings. Memorize the dotted frets as well. Use your scale patterns/shapes and learn to move them around by starting from different notes but utilizing the same fingering.
Hope that helps!