The red thing stuffed in the middle of an olive is simply a piece of a fruit similar to a bell pepper called a pimento.
Also called pimiento in Spanish, the pimento is a variety of mild red chili similar to the bell pepper. The roughly heart-shaped pepper is about three to four inches long and about two to three inches wide. It is considered sweeter than a standard bell pepper.
Pimentos can be used fresh or pickled. They are an ingredient in pimento cheese, pimento loaf (a type of lunch meat) and prepared Spanish green olives.
Originally cooks would cut the pimentos into cubes and stuff them by hand into pitted olives as a complement to the olive's strong taste. In modern production facilities, the cut-up pimento is stuffed by a pitting and stuffing machine. The pimento is sometimes pureed and reconstituted as a gel.