nixos

Overlays allow any modification you can think of, not just limited to what packages have as "override options". You could say firefox = final.hello and then building pkgs.firefox would then result in hello. You could also say firefox = final.callPackage { } ... to inject a whole new derivation you've written yourself. You could make firefox = "foo" if you wanted to (not that that'd make any sense to do).

firefox = prev.firefox.override { ... } is just the specific case where you set the firefox package to its overridden variant (.override returns a new derivation).

That's everything as far as overlays are concerned.

The other piece to this puzzle are overrides. Overrides are a function of each derivation that returns a modified version of the derivation. You can override parameters (a package definition is usually a function) or the attributes of the mkDerivation call. The former obviously relies on the derivation having some flags you could change in its parameters while the latter can do anything you could do in a mkDerivation call (adding additional patches, using a different src, running some more commands preInstall etc.).

You can use overrides without overlays. If all you want is a leaf package but slightly different in your systemPackages, you could do that directly where you set it:

environment.systemPackages = [
  (firefox.override { ... })
  hello
  whatever
  ...
];

Overlays come into play when you want to have dependants of the package to also use the overridden variant without having to explicitly inject it via another override every time or need to modify some service's package that you didn't set yourself (most offer an option to change the package used but some don't).