to your eye only the tree exists on that layer, but to maya it sees the whole rectangular shape. the lights can't tell the difference, so it'll be a square shadow. i think you need to cut the tree out to get a shadow.

i could be wrong, if so i'd like to know how to do that too.