After helping Anders with his quest, Tranquility, you will have a conversation where it appears that you must initiate the romance if you want to avoid a rivalry gain with Anders. However, there are actually two ways to end this conversation without a rivalry gain and without an active romance.
The easiest way to avoid the romance without a rivalry gain is to simply select the first broken heart option available:
The downside to this option is that it will put an end to conversation right then without any increase in friendship or rivalry points. If you want to gain friendship, there is another way. Pick either the "I'm always ready to listen" option or the "It happens a lot" option:
On the following wheel, select the "You did the right thing":
This option will lead Anders into flirting with Hawke, and you will be able to respond with one of two heart options or one broken heart option. Picking the broken heart option will cut off the romance and end the conversation with a +10 rivalry gain. But, picking the very bottom heart option ("It's... unexpected") will not actually start the romance:
Even though the dialog may indicate that the romance does start, it does not. Perhaps it is a bug that causes the "It's... unexpected" option to not flag the romance, but it does provide a way to avoid the romance while still receiving 10 to 15 points of friendship.