How do you attach them to a brick building. I think you want something that you can clean off easily..

It depends on the brick. Industrial strength Velcro works about 80% of the time just like duct tape. For the real crumbly brick, try Hercules Putty (available at WCS). Worse case scenarios is to use Tapcon's, but I normally don't recommend them unless you can cover up/hide/blend them in once you take the exclusion material down.

I've taken to using tapcon screws not to attach the net but to attach a ledger strip. Both wood lath strips and metal flashing work. Staple to wood and duct tape to metal.