If you notice that the only limit is file size — there are multiple file-size limit settings in the Nautilus Preferences window. Each of the settings is for a different kind of storage, e.g. local, USB-attached, and remote.
If you notice thumbnailing is format-limited, you need to install the appropriate Gstreamer codec (i.e. not plain ffmpeg codecs!). Nautilus video thumbnails are usually created via the Totem video player, so that needs to be installed too (there should also be a package named something like nautilus-totem-thumbnails [no guarantees on the package name]).