Look at it like this :
1 bit can store 2^1 numbers (0 and 1)
2 bits can store 2^2 numbers (00, 01, 10, 11)
3 bits can store 2^3 numbers (000, 001, 010, 011, 100, 101, 110, 111)
and so on
so 32 bits can store 2^32 numbers (0 thru (2^32 - 1))
So if an address is 32 bits wide, you can potentially address 2^32 locations. An addressable location is a byte - its not a bit. We cannot have address of bit but the smallest addressable unit of memory is a single byte. Hence the 4GB.