It seems like the “Remove From Caption” entry, which occurs in the context menu, as well as in the “Image” menu, has been forgotten when the newer collection types were introduced and needs to be adjusted.
For the time being, there is the generic “delete” entry in the “Edit” menu, which reads “Remove Item From Collection” when an item within a taxonomy is selected. Pressing just the delete key will also remove the item from the taxonomy.