I think it was either
a) Banner was working right next to the Loki's staff which could have manipulated him when Loki was in the cell.
b) Loki had some weird eye contact with Banner when he was walking by his lab. If you noticed once Loki walked by Banner started rubbing his eye/eyebrows whatever like if his head was hurting. And maybe this prolly had nothing to do with anything but Hawkeye had on dark shades on in the end so maybe Loki wouldn't take control of him again?
Still makes no sense, he was protective of black widow all the way up till he transformed. It was definetly NOT loki controlling him.