I am all for history and architecture but this doesn't seem to rank high on either.
Architecturally it has nothing going for it. Its not vernacular, doesn't offer anything new or inspiring. Honestly walked past this building dozens of times without making note of it.
Historically it was just one of Neylands 10 houses in Knoxville. If it had been his only one, sure. Brisco is interesting, but again not really note worthy. Doesn't sound like anything critical happened there. Hasn't been historically important, to the community, in the past. No moves to make it a historical building.
all that being said, tearing it down would have to be based on what they are planning to do with the lot/area. If its another parking garage I say no.