Hi Meir,
I'm not sure what to tell you. They are trying a non-standard, proprietary format, we don't write scripts for that. We have no source that would give that and no way to verify it. The support we can give you is to suggest that they start with a standard script and tweak it for your format. It's work that we cannot do for them unfortunately.
If you have 0V on the output TMDS, they aren't even close. I would think if their hardware is wired up as expected even the basic SD script would produce some output on TMDS even if a display won't show it (which it's unlikely any HDMI TV will show this format in any case).
I don't really see that we can progress this any further without 1) A diagram of exactly how they have wired up every signal from their camera to the ADV7511 via the header and 2) The complete script they are programming into the part when they get no output. At least then we can compare how you are programming the part with how you have wired the circuit.
Dave