If you keep going, you will come to overlook London Wall. Stay on the North side, don't cross over the road. Turn right, along 'Bastion Highwalk', & follow signs to the Museum of London. On your right you will see the remnants of the original Roman Wall. If you want to explore this further, the Museum is straight ahead.
If you don't feel visiting the museum, cross the footbridge & walk down the stairs to Noble Street. There are Roman remains all the way down the west side of this street, and gardens with benches if you fancy a sit down.Wikipedia - London Wall