Indeed, many have over the time speculated that the top quark might be a composite object. If that were true, then it would be natural to expect that along with the "fundamental state", the one discovered by CDF and D0 in 1995, there exist excited states of the top quark. That is the result of thinking the quark as a composite of smaller entities orbiting one around the other, and imagining that there can be different orbital energies corresponding to different configurations; this is in analogy with, e.g., the B meson. The B meson is a hadron formed by a heavy (anti)-bottom quark and a lighter quark. There exist four such states: the B+, the B0, the Bs, and the Bc, obtained by choosing as light partner the up quark, or the down, or the strange, or the charm.
The two quarks orbit around each other, or better, one can think of the lighter quark orbiting around the heavier one much like the Moon orbits around us (while in truth we both draw cicles around a center of gravity). Besides the "ground state" of each of these four hadrons, there exist excitations B*, B**, etcetera, characterized by a higher mass; they usually decay back to the ground state by emitting the extra energy in the form of, say, a light hadron (a pion).
In the case of the top quark, one may then imagine looking for excited t* states that decay back to the top quark which we know, plus a energetic gluon: that gluon can be visible as an independent jet of hadrons, given the large energy difference we may imagine separates the top from its excitations. If one then "adds" together the energy of the top quark and the detected jet energy, one may reconstruct the mass of the excited top.
At the LHC top quarks are produced in pairs with a large 200 picobarn cross section, which means that in the 20 inverse femtobarns of collisions collected by the CMS experiment there are a total of four million such events. Of course, not all of these events have been collected by the trigger system; but those where one of the tops decayed to an energetic electron or muon have indeed for the most part been identified and stored. CMS can thus look for events featuring all the objects characterizing a "single lepton" final state -a lepton, four hadronic jets, and missing transverse energy- plus one or two extra jets allegedly produced by the emitted gluon when the deexcitation of the t* took place.
[In case you do not know what a single lepton final state is, here is what I mean: each top decays to a W boson and a b-quark jet; the W boson can either decay to two further hadronic jets, or to a lepton-neutrino pair. If we then imagine that one t->Wb->jjj decay and one t->Wb->lνb decay took place, we have the final state we call "single lepton", which indeed included a single charged lepton, a neutrino, and four hadronic jets.]
The single lepton top quark dataset has been already used by CMS to extract the most precise individual measurement of the top quark mass, so you can well imagine that the technology to identify top quarks is quite advanced. In fact, the analysis is performed by first reconstruct the top decays, and then attach a jet not used as a top decay product and reconstruct a tentative t* mass. The study of the mass distribution of these t* candidates allows to search for the signal of this excited state. The distributions obtained by CMS in the sub-samples of data where the lepton is a muon (left) or an electron (right) are shown below (they are kept separate because the lepton characteristics as measured by the detector make it natural to treat them differently).
In the graphs, the experimental data are shown by black circles with error bars; the two coloured histograms show the contribution expected by theoretical calculations for a t* excitation with mass of 750 or 850 GeV; the data is fit to a smooth function (pink). As you can see, the excitation would be reconstructed as a broad peak which would contribute just a little bit to the falling distribution of reconstructed masses (for the dominant part due to true top quark pairs produced in association with a pair of gluon jets by QCD radiation). This is for masses just at the limit of the CMS sensitivity: for lower t* masses, the contribution would be much larger and quite visible.
From the featurelessness of the mass distribution, well fit by a simple functional form, CMS extracts a limit on the presence of the additional contribution of excited quarks in the sample. The upper limit on the signal rate can then be turned into a lower limit in the mass of these excitations by virtue of the fact that higher-mass excitations are expected to produce smaller number of such events. The graph which allows that inference to be drawn is the one shown below, where on the horizontal axis you see the excited top quark mass and the vertical scale describes the rate of the process. The yellow and green curve describes the upper limit on the excitation rate, and the intersection with theory curves (relative to two different hypotheses for the spin of the t* state, which yield different predicted production rates) identifies the lower limit on the mass of the t*.
In conclusion, CMS does not see any hints of excited top quarks in the large sample of top pair decays. Such particles, if they exist, must be heavier than 803 GeV, if the spin of the t* is 3/2. If you need more detail on this search, there is a fresh new paper in the Arxiv for you to read.