The key is to believe that you're not doing anything wrong. If I'm seeing someone new, then in my mind I pretend I'm going on a first date. If I'm seeing someone I've seen before then I'm seeing a friend. Sometimes I talk with people on the elevator, sometimes I don't, just like I normally do. If someone has a dog I usually talk to them.
Once I arrived to find a crowd of people entering the building because there had been a small stove fire in one of the units. I buzzed in, but by the time I could get into the elevator (after missing a few because of the crowd) I couldn't press the floor button because of the time delay. I went out and buzzed in again and the same thing happened. Finally, the building manager saw my plight and asked me which floor and swiped the elevator pad so I could go up.