I can see two cases here:

1/ a compiler is not found, and build_ext can't proceed
2/ a compiler is found but it fails

For 2/, I don't think it's wise to fail silently if a compiler is found
and the build fails, because that can hide an unexpected problem.

For 1/ I would go for a binary distribution, or for a source
distribution that is specific to Linux in your case, if the extensions
are specific to other platforms. The latter could be done in the same
setup.py if needed by declaring the extensions only on matching platforms.

But I can see the benefit of a source distribution with optional
extensions in general. In that case, I think this option would have to
be declared optionally for each extension to be able to fail properly if
one extension is mandatory.