What are you expecting?
Transmit:
I[x] = A * sin(2PIx)
Q[x] = A * cos(2PIx)
You receive:
I[x] = i + A * b * sin(2 PI x + c)
Q[x] = j + A * d * cos(2 PI x + e)
While:
c, e are PHASE errors and
b, d are GAIN errors and
i, j are OFFSET/DC errors.
The AD9361 internal DC and IQ balance correction does an excellent job to keep the errors small.
However TX[i,q] will never be RX[i,q] !
Your digital modulation/demodulation scheme adds enough margins, so that the TX symbols can be recovered after the RX.
-Michael