Application of integer quadratic programming in detection of high-dimensional wireless systems