It takes much too long for Giles to actually notice the hat, but...really, who expects to see a derby hat on a horse? There are things even Giles doesn't expect to find in a situation.

Getting a proper look at it, of course, is going to require getting as close as he can to Sif without actually passing through her again.

A derby hat on a horse...

"Do you mind if I take a look at that?"

He means for her to shake it off, maybe, or drop it on the ground.