Include attachments with a custom taxonomy in search

I need to include attachments in the search. So far, I have managed this by hooking into pre_get_posts and checking if it's on the search results page, I add 'attachment' to the post_type and 'inherit' to the post_status, in order to also include attachments that are not attached to posts.

However, only attachments that have the custom taxonomy download_category set to any term should be included, attachments that do not have any download_category may not be seen in the search results.

How can I achieve that?

I have found this answer Custom Taxonomy WP_Query for All Terms in a Taxonomy? and tried it out. While this does really only show those attachments with any download_category set, it applies to posts and pages as well, and as those never have the download_category taxonomy set, no pages and posts are shown in the search results...

So to clarify again what I need in short terms: - include attachments in the search results, but only if they have taxonomy download_category set to any term - Keep normal search functionality for pages and posts!