Should I turn off the app when another driver uses my vehicle?

No. There is an option to indicate if you are the driver or if someone else is driving in the app. You need to exercise this option within 30 days after the trip.