That description is consistent with having reverse dns lookup enabled for logging, first an initial delay while the ip number is resolved, then fast while it's the cached value that's used.
You might want to check that that isn't the cause.

__________________
An emperor without enemies, a king without a kingdom, supported in life by the willing tribute of a free people.