Since gas exchange occurs by diffusion alone, single-celled organisms such as bacteria do not need any specialised structures to achieve it for them. This is because, being so small, diffusion occurs readily as their surface area to volume ratio is high.
The distance between an oxygen molecule which must be taken in, and the place it must get to within a bacterium is short enough for diffusion to be a viable way of exchanging substances with the environment, without the need for additional structures. In mammals, for example, oxygen cannot simply diffuse into our bodies. We are too large, have a low surface area to volume ratio, hence the diffusion pathway is too long. The only way we can achieve gas exchange is through our lungs which provide a large surface area and alveoli with short diffusion pathways.
In fact, the difficulty of gas exchange as single-celled organisms get larger is a factor which leads to larger organisms being multicellular.
Before this goes any further, a few clarifications: