If the file system is FFS or ext2(3) you can use file permissions, but it's impossible for FAT as it has no concept of Unix permissions.
This has been said in the past, FAT was designed for a single-user operating system..
EDIT: If you want to restrict access to files on ANY portable device, a better method would be encryption.. via OpenSSL for instance.