We do know that the fragments we've got so far are too different from our own counterpart segments to have been mixed in the same gene pool recently. The amount of similarity there is (only about twice as much as with chimpanzees) is only consistent with common ancestry, not with interbreeding.

There's also another way to look for genetic signs of interbreeding. The amount of variation on a given gene in the population tells the gene's age (the time elapsed since the original version of the gene that all modern versions are derived from) because mutations that don't affects its functions accumulate over the generations. A gene that's exclusive to this species starts its clock a couple of thousands of centuries ago or less, whereas one that at least some (or all) versions of came from another species starts its clock much longer ago, at the time of last common ancestry between the two species... so the former would have much less variation in the modern population than the latter. But there are no genes showing that much variation in modern humans; they're all recent, indicating exclusivity to this species. (I did read about one or a few that seemed variable enough to possibly be significantly older and thus at least partially of external origin, but that was years ago and I haven't heard of it since then, which makes me suspect it turned out not to be the case... and that was in eastern Asia anyway, not in or near Europe, so even if that were from interbreeding, it would be with a species nobody seems to eagerly want to say we've interbred with like they do with neanderthalensis: erectus!)