Be sure to set a filename to be produced. The filename extension controls what format file is produced.
Windows:The Preview Movie is produced by OpenGL, but it may be routed to Windows's software renderer, which can limit the maximum allowable output resolution.
Youmustselect a codec and compression options, Quicktime does not have a default.
Be sure to select the correct frame rate in Quicktime to match your shot.
Different codecs have different quirks and may need different settings. Cineform codecs may require the resolution to be a multiple of 16, for example.
The H.264 codec requires that the "Key frame every N frames" checkbox be OFF, and probably you should either turn off the "Limit data rate to..." checkbox, or select a rate much higher than the default 90 Kb/sec.
There's no Quicktime output from 64-bit Windows. On macOS, only a predefined set of codecs can be used for output.