For some reason, this is disabled by default in all Domino installations (as of this FAQ, at least to 8.5.2FP1).
Check the domino/data/browser.cnf file. You'll see a section for the Java editor applet as follows:
## DisableEditorApplet - Boolean indicating whether browser can handle the Editor Applet
Property DisableEditorApplet Boolean False
Rule True MSIE [0-9].*\; Macintosh # IE on MAC platform
Rule True MSIE [0-9].*\; Mac_PowerPC # IE on MAC platform
Rule True ^Mozilla/[0-9].*\(Macintosh # Netscape on MAC platform
Rule True ^Mozilla/[0-9].*\(Mac_PowerPC # Netscape on MAC platform
Rule True Notes # Notes browser
Change the two highlighted values from True to False to enable Java applets for Firefox on the Mac.