Personally, I agree that a conversation is always going to go off topic, given enough participation and enough time. In the other sections of the site, I'd allow a certain amount of leeway and most of the time someone has suggested getting back to the topic anyway, but in the Totally Off Topic section I'd probably allow more freedom.
I tend to see the Totally Off Topic section as being more the "social area" of the site... kinda like nipping down the pub after your astronomical society meeting has finished.
That's just my view on things.
Look at it this way - if you were to start a topic about something you were keen to learn more about, say black holes, and then people ended up talking about the acting in a certain Walt Disney film, wouldn't you want to get things back on track?