some things to look into
I can probably help - but I would need some more information. You mentioned using xinet - which version of KA Share are you running on the O2? Also, you mentioned connecting to an AppleShare File server -- which server are you connecting to? AppleShare IP (old Apple server technology) or do you mean file sharing on a standard client (ie. enabling file sharing on OS X, for instance). I think you are probably running into an incompatibility with the versions of Appletalk that are being used. Or it could even be that AppleTalk is not on. OS X only does AFP over TCP/IP as opposed to the older AppleTalk protocol. Make sure your OS X client has Enable AppleTalk selected in the Sharing preference pane. If you have OS X on the Mac you have other options available for connecting to your SGI including NFS, and some easy to use command line tools, such as scp.
These just some things to try or think about. I would be glad to help you figure it out further.