It's very easy, tom. Just remove the core of the olive by pinching and scraping with a thumbnail or fingernail between the foam and core till it is almost free, then you should be able pull it out without tearing the foam.
Once you have the cores removed, find some OEM foams ...maybe some came with your UE4s, I don't remember...and easily remove their cores, mine usually fall out without having to even look at them, then put those larger cores into the olives, allowing the olives to come back to shape, take some super glue and carefully put a small dab between the olive and core at least 2 spots. The inner large side of the core will go to the flat side of the olive. The other end, the end that goes into the ear first, is the end to glue, to keep it from rolling up when you insert them. When you glue, kind of pull apart the olive from the core and put a small dab in there at 2 spots around the core.