There are a few which I have hit, but there are more which are on
http://www.macwindows.com.
File name limitation
Invalid characters (/, \, *, ?, ;, etc.) which mac users keep trying to use
Network browsing sometime won't show SMB shares
Login to a SMB can be a nightmare (authentication problems)
a lot of "._filename" all over the server, which is the resource fork of the file you sent, so it keeps it preview, file type, what program made it etc etc. Usually they will delete when the file is deleted from a Mac, but if a PC deletes it or moves it, it stays there. (I had around 20000 "._" on one of my WinNT smb servers that I deleted, you can get "BlueHarvest" which helps you a bit, but be carefull on which file you delete the "._" on. I only do it to PDF, TIF, PSD, EPS, JPG, MPG, and a couple of other types. DON'T DO IT TO FONT FILES or you can kiss them good bye. If you use extensions on your files its OK, if not, learn to use them.
Some setups in a mixed PC Mac place, you can some files that lock, and you can't delete them untill you reboot the NAS (did have this problem for 10.4.x only where 10.3.9 was fine, so I have only got a couple of macs running 10.4.x atm, this problem did go away for some reason, but other ppl out there still have the problem, apple forums has others if you do get a problem)
I hope 10.5 is better.