On the Application of Natural Language Processing for Advanced OSINT Analysis in Cyber Defence

Open Source Intelligence (OSINT), in addition to closed military sources, provides timely information on emerging cyber attack techniques, attacker groups, changes in IT products, policy updates, recent events, and much more. Often, dozens of analysts scour hundreds of sources to gather, categorize, cluster, and prioritize news items, delivering the most pertinent information to decision makers. However, the sheer volume of sources and news items is continually expanding, making manual searches increasingly challenging. Moreover, the format and presentation of this information vary widely, with each blog entry, threat report, discussion forum, and mailing list item appearing differently, further complicating parsing and extracting relevant data. The research projects NEWSROOM and EUCINF, under the European Defence Fund (EDF), focus on leveraging Natural Language Processing (NLP) and Artificial Intelligence (AI) to enhance mission-oriented cyber situational awareness. These EDF initiatives are instrumental in advancing Taranis AI, a tool designed to categorize news items using machine learning algorithms and extract pertinent entities like company names, products, CVEs, and attacker groups. This enables the indexing and labeling of content, facilitating the identification of relationships and grouping of news items related to the same events – a crucial step in crafting cohesive "stories." These stories enable human analysts to swiftly capture the most significant current "hot topics", alleviating them from the task of consolidating or filtering redundant information from various sources. Taranis AI further enhances its capabilities by automatically generating summaries of reports and stories, and implementing a collaborative ranking system, among other features. This paper serves as an introduction to Taranis AI, exploring its NLP advancements and their practical applications. Additionally, it discusses lessons learned from its implementation and outlines future directions for research and development.