It is possible to remove the starter without taking the exhaust off, but you will be working blind. The starter is held in place by two 12 mm headed bolts near the back. Remove them and the starter comes out of the cases. All that is left is to disconnect the battery lead.
If you are not comfortable working blind, you can remove the exhaust. It takes a bit of work, but it is not difficult.