Excellent schematic, glad you don't want to see any of mine ;-)
Having looked at the IA and ADC I cannot see anything wrong with your implementation.
I do think it worth looking at the +5V Vcc line with a scope for noise as Ceri indicated. I have found noise can cause all sorts of alias effects and even cause dc levels to shift.
I also presume the 5V VCC supply is derived from the +12V supply to maintain the break in GND and DGND. This again makes it worth checking the 5V VCC for noise as it can be transferred from the DC - DC converter.
Although I have never used the IL711 before (but may do now, thanks), again they look very good with minimal pulse distortion and are high speed, so should not be an issue.
So at the moment I feel that the VCC is suspect!
Regarding the decoupling I presume these are ceramic capacitors? I would also suggest adding a 10nF in parallel with the 1uF just to ensure good high frequency decoupling
Having seen that this is multi channel, are the results the same on other channels?
Also you have given us one result at 3.3V, can you supply more and at least 0V as well?
I use MBED to build a breadboard prototype for voltage measurement.
The hardware is like that
MCU -> Isolator -> ADC(LTC1864) -> Low Pass Filter - > Instrumental Amplifier - > voltage sense resistor I use SPI to communicate with ADC The circuit works and now I am calibrating the circuit. However, the error is a little bit too big. When I applied 3.3 regulated voltage to the voltage sense resistor, the reading from ADC only give me 3.03V.
Where is the 0.3 V error coming from?