How on earth would the flickr site know if someone looking at the picture was a friend of yours, or even you?

By using a similar cookie-based mechanism to LJ. If you don't have a cookie to authenticate you, or if wouldn't grant you access, then you don't get access. This takes care of the links-from-other-sites case, too.