The first problem with affirmative action is that it is usually trying to fix the problem far too late. College admissions should be based entirely on merit and what the student can bring to the campus. It is certainly possible that the traditional measures of merit are fundamentally flawed, and should be reworked. This, however, is a problem with the measures of merit being inaccurate, not necessarily discriminatory. Rather than 'fix' the problem right before college, it would make a lot more sense to fix the problems with the education system before college. This way, the students who 'need' affirmative action now will be able to get in on their own merits.
It is also certainly possible that the admissions departments are biased, but there are other ways to get around this, rather than have a selection directly based on race. One would be to have all evaluations of the candidate take place without the name or any other identity markers (gender, for example). This can't solve discrimination at interviews, but it might help isolate the type of discrimination, and show how much is due to race and gender, and how much is due to merit.
One of the problems that those in favor of affirmative action tend to ignore is an effect later on, that changes the perceptions regarding the group benefiting from the discrimination. If people know that an institution practices affirmative action, then the people in the group receiving the benefits will be perceived by some as marginal. The phrase of 'You only got in because of ...' is something that friends of mine have heard, and part of the problem is that institutions publicize that they practice admitting traditionally lower qualified applicants based on race or gender. This sheds doubt on every member of the group that is benefiting from the affirmative action, as any of them could be below the requirements of what it would take a candidate who did not benefit from affirmative action. This hurts both those in the group who benefited from affirmative action who could have gotten in, and those who would have gotten if they were not displaced.
When affirmative action is done wrong, it can also have an even worse problem, by perpetuating the stereotypes that the program was supposed to oppose. If no candidates from the traditionally dominant group are admitted, and the traditional measures of merit are accurate, then what will tend to happen will be that the group that benefited from affirmative action will tend to be at the bottom of the class, rather than dispersed more evenly throughout the class. This will mean that the experiences of the students will generally be that the traditionally dominant group is better at academics than the group that benefited from affirmative action, leading to a perpetuation of racist or sexist stereotypes.
As I said before, this requires that the traditional measures of merit are accurate, and it is certainly possible that they are not. But I think it is highly unlikely that the measures of merit are completely wrong, or have no value, otherwise schools that didn't use them would be just as good as those that didn't, and I don't know of any school with an admissions policy that doesn't use at least some of the traditional measures of merit (GPA, SAT, ACT etc.). What makes more sense than adding something like race to the admissions would be to attempt to reform the traditional measures of merit so that they can more accurately measure the attributes that they intend.