Introduction
What is ciaops? ciaops is a lightweight, MIT-licensed Python library developed by Group-IB that provides a unified integration layer for the Group-IB Threat Intelligence, Digital Risk Protection, and Attack Surface Management APIs. It handles authentication, error management, cursor-based incremental polling, and data normalisation — abstracting the per-product API differences so integration engineers can focus on platform-specific logic rather than API plumbing. Available via pip install ciaops.
Modern cybersecurity infrastructure rarely exists in isolation. Enterprises use a broad range of TIPs, SOARs, SIEMs, ticketing platforms, and even custom solutions — and they expect seamless integration between them.
If you’re an integration engineer or a security architect who has spent more than a week wiring up a single API, you already know the problem isn’t authentication or rate-limiting in isolation. Every multi-product security stack has this challenge — different cursor formats, different filtering vocabularies, different edge cases per API. Together they turn a one-week job into a one-quarter slog. ciaops exists so that’s not your problem anymore.
At Group-IB, we don’t just “connect the dots. We design integrations that are robust, reusable, and built to last. In our previous post we walked through what goes wrong at the integration layer when teams build without a clear architecture: missed attribution, silent data loss, duplicate alerts.
This article introduces how our integrations team approaches this challenge, with a layered architecture. It is a powerful new version of an internal library, and a mindset focused on developer experience and scalability.
Three layers, one goal: seamless integrations
Our architecture is built on three logical layers. Each has a clear role, and together they form a scalable, predictable foundation for integrations.
Layer 1: Group-IB’s product APIs — three products, one abstraction problem
This is the source of truth — our cybersecurity products like:
These products are what our clients want to gain access to. But in each individual case they want it to be a tailored experience.
As every customer have their own requirements, our products are also quite different from each other:
- Threat Intelligence — A broad, continuously updated catalog of ~35 collections spanning APT and cybercrime threat reports (with MITRE ATT&CK mappings), IOCs, Suricata signatures and YARA rules, phishing infrastructure, compromised credentials and bank cards, dark web intelligence, and suspicious IP intel — built for ingestion into SIEM/TIP/SOAR platforms, threat hunting, and enrichment workflows.
- Digital Risk Protection — A continuously updated feed of brand-targeted threats (impersonation, fraud, phishing, counterfeit) paired with management endpoints to register brands, manage subscriptions, and approve, reject or submit violations — so detection and remediation can happen through the same API.
- Attack Surface Management — Continuous external-asset discovery delivered as three change feeds (assets, leaks, issues) with built-in status workflows (new/confirmed/false for assets; detected → solved/ignored for leaks), so external risk can be both observed and triaged programmatically.
Group-IB ships three products and three APIs that all follow the same bookmark-style delta pattern, but everything around the bookmark differs: GET vs. POST, query-string vs. JSON-body cursors, three filtering vocabularies, TTLs in one product but not the others. Each difference is small, together they turn a one-week integration into a one-quarter slog, and the bugs don’t loudly fail, they quietly produce incomplete data. ciaops is the bridge: one Poller → FeedGenerator → Parser flow across all three products, so you spend your time on your integration, not on three different API specs.
ciaops abstracts all of this. One Poller → FeedGenerator → Parser flow works across all three products, so you spend your time on your integration — not on three different API specs.
Current integrations built on ciaops include Splunk, Filigran OpenCTI, IBM QRadar, Cortex XSOAR, and more.
Layer 2: ciaops – the shared abstraction layer
This is the heart of our system: a public Python library that abstracts away the heavy lifting of API communication. It acts as a middle layer between our product APIs and the platform integrations.
What it handles:
- Authentication and safe credential storage – enter your credentials once, hit save and never think about them until they expire.
- Consistent request/response handling – library translates between human-friendly inputs and each product’s API, so your integration code reads naturally.
- Error management and retries – networks fail. The library transparently retries on transient errors and surfaces clear, actionable messages when something needs your attention.
- Basic JSON data parsing – a built-in “find by key” filter lets you trim large responses on the fly, before they hit your downstream systems, keeping memory use low.
- A system of adapters to customise library behaviour for a particular integration case; that’s what Layer 3 is built on.
Why it matters:
Instead of duplicating logic in every integration, we centralise it. That makes our code cleaner, safer, and easier to maintain, even as our products evolve. A single backward-compatible update in the library lets every downstream integration gain new features without rework.
Why a new library now?
An earlier version of this library was community-supported on PyPI as cyberintegrations. Rather than just a rename, ciaops is a complete redesign. We’ve broken the old monolithic structure into clean, modular components. This makes it easier to update individual features without breaking the rest of the system.
We’ve also replaced hardcoded strings with clear constants, improved error handling so bugs are caught immediately, and added support for ASM alongside TI and DRP.
What this means for you: You get a single, predictable import surface, smarter data validation, and a codebase that evolves with our products. New projects should use ciaops; the legacy cyberintegrations library will remain available but won’t receive new features.
Layer 3: Platform-specific integrations
The top layer contains the actual integrations with external platforms. Thanks to the library, they don’t have to worry about authentication quirks or payload formatting. Instead, they focus on:
- Mapping data to the platform’s needs
- Triggering actions or alerts
- Respecting each platform’s constraints and use cases
And if a platform has special requirements (e.g. flattened JSON, legacy auth), we use custom adapters in the library to tweak the behavior, without breaking the overall structure.
Why This Works
This architecture wasn’t built for elegance alone. It solves real challenges:
- Centralized Logic
- By moving shared logic into
ciaops, we eliminate duplicated code. One change in the library improves dozens of integrations at once.
- By moving shared logic into
- Fewer Bugs, More Predictability
- A shared validation layer catches issues early and keeps behavior consistent. Developers can rely on predictable patterns instead of hunting hidden edge cases.
- Flexibility with Adapters
- Each integration overrides only the parts it needs, without rewriting everything. The modularity is what lets us support many different platforms efficiently.
- Faster Development, Easier Maintenance
- Developers start from a solid foundation and focus only on what’s unique to their integration. This means faster delivery times and fewer long-term headaches.
Logic of iteration – how to guarantee delivery
When you build an integration that should not just “upload something,” but guarantee to receive all important events, sooner or later you realize that reliable integrations don’t happen by accident. They’re engineered.
According to our blog about integration mistakes, every missed event can be costly.
That’s why we have a clear iteration scheme via a cursor field.
Consistency instead of chaos
Every event in our database contains a cursor field – it’s a monotonically updated value with each new appearance of an entry. Think of it like a bookmark in a book: every response hands you a small cursor, you store it, and on the next call you give it back so the server picks up exactly where you left off. It looks like a technical triviality, but in practice it is the guarantee that you won’t lose anything.
It is our way of confidently answering the question that matters the most: “What has changed since the last request?”
The scenario: you’re pulling 1,500 events from a TI collection. Everything is going according to plan — and then on the 876th event, a network or process crashes.
Without seqUpdate: you either start over and get duplicates, or try to jump ahead and risk missing data.
With seqUpdate: you continue from the same place. No duplicates. No losses. No manual checks.
The logic in practice
- Get a starting seqUpdate via
/sequence_listor load a saved one. - Send a request to the
/updated-endpoint of the required collection:GET /api/v2/apt/threat/updated?seqUpdate=16172928022293 - Process the received events, save the new seqUpdate from the top-level response.
- Repeat the loop until an empty list arrives.
This is a classic iteration forward: strictly timed, with no unnecessary noise or re-requests. Even if the integration restarts – you’ll quietly continue from the last point.
Why it’s critical?
More than once we’ve seen systems try to “over-insure” by pulling everything again every hour. In practice, this leads to:
- API and platform overload
- Duplicate alerts in SIEM and SOAR
- Data loss from pagination failures
- Real-world incidents that remain out of sight
In contrast, the seqUpdate gives:
- Built-in resilience to failures
- A clear recovery point
- Efficient incremental delivery
And most importantly, a unified approach that works in all scenarios, from full synchronization to spot searches.
A recent example: an integration ingesting 100,000 events suffered three connection drops mid-run. Thanks to seqUpdate, we didn’t lose a single event – and we didn’t re-process any of them.
This logic seems simple – and that’s its power.
It allows us to build integrations that don’t fear failure, don’t lose data, and scale without fuss.
We don’t just connect to APIs, we guarantee delivery. Transparent. Predictable. Consistent.
ciaops as an integration framework, not just a library
Past the core request/response loop, ciaops is a framework rather than a helper script. It bundles in:
- Schema validation for incoming and outgoing data
- Unified logging and debugging tools
- Shared request models across all Group-IB APIs
- Extensibility via adapters and configuration
This is what lets us deliver real, maintainable integration components, not one-off scripts – and it is the foundation that the platform-specific integrations in Layer 3 stand on.
Where do we use STIX/TAXII (and where we don’t)
A reasonable follow-up question at this point is: why bother with a custom library and APIs when STIX/TAXII already exist as a standard? The honest answer is that we use them, but they are not the right tool for every case.
For readers unfamiliar with the standard: STIX (Structured Threat Information Expression) is a JSON data model for describing threat intelligence — indicators, TTPs, malware, threat actors, and the relationships between them. TAXII (Trusted Automated Exchange of Intelligence Information) is the transport protocol that moves STIX between producers and consumers. Both are maintained by OASIS. We work with STIX 2.0 and 2.1 across our feeds; the older 1.x XML-based versions are effectively retired.
So why not standardize on STIX for everything? Four reasons, in order of how often we run into them:
1. Inconsistent support across platforms
Many SIEM, SOAR, and TIP solutions claim STIX compliance, but in practice support is partial. References, relationships, and sightings are frequently ignored or stripped during ingestion. In some platforms, we see only the raw IOC retained — attribution, TTPs, and threat-actor context are silently dropped. We covered the downstream impact of that in our integration mistakes blog: losing attribution mid-pipeline has real consequences
2. Limited support for non-IoC data
A large share of our feeds is intentionally not IoC-shaped:
- Compromised or breached accounts
- Payment card data and suspicious payment details
- YARA rules and Suricata signatures
- Dark web infrastructure
STIX makes the most sense when representing structured threat intelligence, such as indicators, malware, campaigns, threat actors, attack patterns, and relationships between them. However, not all ciaops use cases are limited to traditional threat intelligence. On top of CTI, ciaops also covers areas such as Digital Risk Protection and Attack Surface Management, where the data often represents exposure, leaked assets, external infrastructure, identities, or risk findings rather than classic STIX objects.
These data types can be modeled with custom STIX objects or extensions, but most platforms either ignore custom objects or fail to map them correctly unless a native integration has been built for that specific data shape. As a result, forcing non-IoC, DRP, or ASM-related data into STIX/TAXII can reduce clarity, lose context, and create additional integration work without providing much practical benefit.
3. Overhead for simple use cases
For firewall rule feeds, NGFW integrations, and basic blocking lists, a full STIX object is overkill. These consumers typically want a flat list of indicators — anything richer adds complexity or breaks ingestion outright. In those cases we ship lightweight scripts built on ciaops.
4. STIX-native platforms
For platforms whose internal model is STIX — TIPs, primarily — we still use our own API and library layers underneath:
- Layers 1 & 2: data retrieval and transformation via
ciaops. - Layer 3: mapping and enrichment within the platform using its native STIX structures.
This way we get the flexibility of our own pipeline and the platform’s STIX-native semantics, without forcing one to bend to the other.
In short: STIX/TAXII is a useful complement to our integration stack, not a replacement for it. We are also actively working on a major refresh of our STIX/TAXII implementation to improve support for relationships, custom objects, and the non-IoC feeds above — more on that in a follow-up post and feature releases for TI customers and partners.
Get started or contribute
For customers and technology partners. ciaops is the library to build new Group-IB integrations against. It is publicly available on PyPI (pip install ciaops), MIT-licensed, and the public API surface is stable. New integrations should target ciaops rather than the older cyberintegrations package, which remains available for legacy code paths but will not receive new features.
For developers. The project is open to external contributions (soon go live on github). Adapters for new platforms, additional output schemas, bug reports, and documentation improvements are all welcome – see CONTRIBUTING.md in the repository and the issue tracker. If there’s an integration you’d like to see and the building blocks are already there, we’d love to hear about it.
Conclusion
A good integration doesn’t just connect to an API. It guarantees that what came in on the source side arrives intact on the other end; no duplicates, no silent drops, no missed events from a mid-run crash.
That’s what the three-layer architecture and ciaops were built to do. The library handles the authentication, the retry logic, the cursor management, the normalization. You handle the part that’s actually specific to your use case.
In cybersecurity, the data you miss is the data that costs you. ciaops exists so you don’t miss any.
Frequently asked questions: ciaops
What is ciaops and what does it do?
ciaops is a lightweight, MIT-licensed Python library designed to provide a unified abstraction layer for Group-IB’s Threat Intelligence, Digital Risk Protection, and Attack Surface Management APIs. Instead of requiring engineers to write separate, complex integration logic for each individual product, ciaops standardizes authentication, error management, data normalization, and cursor-based polling via the seqUpdate pattern. It is available for straightforward deployment via pip install ciaops.
How does ciaops handle connection stability, authentication, and error management?
Production networks are inherently unpredictable, which is why ciaops is built to resiliently manage the underlying transport and session layers. The library automatically handles credential validation, formats authentication tokens correctly per API requirements, and performs necessary pre-flight verifications. If a network blip, timeout, or transient server error occurs, ciaops executes built-in retry logic with appropriate back-offs. This ensures that integration scripts remain stable and don’t silently fail or crash due to temporary connectivity issues.
How does ciaops differ from STIX/TAXII for threat intelligence integration?
STIX/TAXII is an industry standard for structured threat intelligence exchange between platforms that natively support it. ciaops, by contrast, solves the upstream ingestion challenge: reliably extracting raw data from Group-IB’s three distinct product APIs, each of which utilizes different filtering vocabularies, cursor formats, and TTL (Time-to-Live) behaviors. For STIX-native platforms, ciaops manages the initial retrieval and normalization layers, allowing the target platform to handle the final STIX mapping cleanly. The two approaches are complementary, not mutually exclusive.
How does the seqUpdate cursor pattern prevent data loss in threat intelligence pipelines?
Data continuity is critical during high-volume ingestion. Every event in Group-IB’s database is assigned a seqUpdate cursor — a monotonically increasing value. After processing an API response, ciaops tracks the latest cursor value. On the subsequent request, it queries the API using this checkpoint, ensuring the server only returns new events. If an integration pipeline encounters an unexpected mid-run crash, it simply resumes from the last successfully stored cursor upon restart. This strict tracking prevents both data gaps and duplicate processing.
How long does it realistically take to build a new Group-IB integration using ciaops?
By abstracting the foundational complexities of authentication, retry logic, and cursor management, ciaops significantly compresses development timelines. For a standard SIEM or SOAR integration targeting specific Group-IB Threat Intelligence collections, writing the platform-specific code — such as schema mapping and alert triggering — typically takes a few focused development days. Without the library, building these foundational transport and synchronization mechanisms from scratch for each product generally requires two to three weeks of engineering effort.
What platforms does ciaops currently support?
It is important to note that ciaops is a foundational software development library, not a plug-and-play, pre-built connector. It serves as the programmatic backbone for any custom integration interfacing with Group-IB’s APIs. Group-IB utilizes this exact library to build and maintain its official, platform-specific integrations for major SIEM, SOAR, and TIP vendors.
