The exact date that the castle was built is not known, but was likely to have been built shortly after 1106. Within a short time after the Norman conquest of Gower, Oystermouth Castle was owned by the Chief Lord of Gower along with Swansea Castle. In the Courtyard (or Bailey), to the left of the entrance is a guard room and next to it is a flight of stone steps which leads to the chamber where the portcullis was worked, this room was also the room where the governor or his deputy worked. This portion of the castle is probably the latest addition to it. Standing with your back to the keep on the left is the kitchen and on the right the barracks, both now in ruins.

Beyond the kitchen is an extended kitchen built later on and has two cells beneath it, in the basement which is approached from the courtyard. There was also a ground floor and upper apartment. The basement was a store room. The upper floors had windows and fireplaces and may look strange now as there are no longer any upper floors. The upper floors were domestic apartments.

At the end of the open air corridor are the prisons, the oldest part of the castle. On the ground floor (really the basement) and partly underground is a room which was most likely a store room.

On the first floor which is level with the porch, the square Keep was divided into two parts. One half was called the courtroom where the assizes of the lordship were held, a notable feature was a large fireplace. Above this was an apartment with an east facing window (now blocked off). One of the doors of the apartment lead to the chapel.

When the castle was lived in, the rooms downstairs were very dark, the small secure windows designed to prevent an attack. Only the upstairs windows could afford to be be large and able to let in a reasonable amount of light.

The ground floor is known as the drawing room, above this was the banqueting hall. Leaving the Keep by the porch for the courtyard, on the left is a chapel annexe, decorated by Lady de Maubray. The two upper floors are reached by the circular stair case

After Oystermouth Castle was built a small but permanent settlement grew up around it, better known as "Mumbles Village"