We are seeing the light from a different portion of our universe. If we wanted to see light from 13 billion years ago from our location, we would have to be on a planet which lies on the outskirts of our current observable universe. In fact due to dark energy's expansion, the observable universe is much larger than 13B light years across (It's about 93B). The "source" of the burst is one of the biggest unanswered questions in astronomy, but everything we see, including ourselves originated from the material expelled from the big bang. Also ALL light, including gamma rays travel at the speed of light. The speed of light is a constant through a medium, whether it be through water, air, or the vacuum of space. So a gamma ray travels at precisely the same speed as a radio wave, the distinction between them is the frequency of the light (gamma is high frequency and radio is low frequency). The light from nearby objects blew by us eons ago for closer objects, such as the Andromeda galaxy. We would have seen 13B year old light from it ~2.5M years later, because that is how far away it is from us and the speed of light is the limiting factor (if Andromeda was the same distance from us 13B years ago, which it wasn't). We can only find 13B year old light from objects which are 13B light years away from us (before accounting for expansion of the universe) just as we can only find 500 year old light from objects which are 500 light years away from us. Hopefully that makes sense.
Also, regarding the universe being infinite or finite, this is determined by trying to find a curvature to our universe in its 3 dimensions. What this means is that if we were to observe 2 objects from Earth and calculated the angles between these objects, they should add up to 180° (because a triangle is made between the 2 stars and the Earth). If there is a measured curvature (which we have not found yet) then that means our universe is not "Flat" in the 4th dimension. So if we added up the angles and found them to be > 180° then the universe would be finite as that is characteristic of the angles one would find if they traced out a triangle on a sphere. If the angle is < 180° then the universe is saddle shaped. In the flat and saddle shaped universes, it is infinite whereas a spherical universe would be finite.