This can be done by including extra criteria in the Action query:
  • In your action query go to Criteria
  • Select Events
  • Click Event ID
  • Select the operator as blank (So it says AND Event ID Blank)

This will exclude event actions from pulling in.