...it really handles gravel just fine as it is, once you get used to that loose feeling.
I do a little gravel-roading on mostly flat terrain now an then. It does feel dodgy but keep your weight in the right place and all is well. After renting an an adventure bike and doing about a quarter mile up a graveled hill on it, I'd prefer that feeling if I had to do it for any real distance. Do you use different tires?