AutoVoice How to terminate Autovoice listening on Google Home

I have an natural language intent where the toggle is set to make the assistant not finish the conversation. This is because the task that runs when the intent is triggered may ask for followup information, though only in certain circumstances.

The problem I have is that if there is no followup information required I need a way to stop the Autovoice conversation without saying something. Currently if I don't say anything Google Home will say "I couldn't hear anything" etc after a short while.

Is there a way to do this?