Too many web pages load Facebook's scripts (for comments, single sign on, or just plain people stalking) so even if you didn't sign on to FB, you probably loaded a Facebook asset (single pixel gif, or javascript) and if you had any FB cookies, and there'd be enough info there on the basis of IP address to geolocate.

I'd recommend running the Ad Block Plus and Ghostery extensions and frequently updating them, they can stop sites from side loading Facebook assets.

Also, if you absolutely have to use Facebook, do it in a separate browser that is only used to look at Facebook, and delete all data (if you are feeling sufficiently burn-it-it with fire, use a virtual machine instance that gets deleted every time.)