The argument is that some victims are too scared to say no, therefore we should take the absence of yes as a no. Otherwise we as a society would be condoning rape every time a person jumps out of an ally and rapes a stranger walking by, right?
If something like that occurred, it would be very easy to prove rape. Remember, refusal to consent can be implied as well. It doesn't have to be a verbal statement. Physical wounds are the biggest evidence. Which is why rape tests are administered by hostpitals in these situations.