A Look Into How Intricately Collects Cloud Product Adoption, Usage, and Spend Data

The Intricately Global Sensor Network constantly maps and monitors Internet-based activities, actions, contributions, and communications. With thousands of physical deployments spread across six continents in over 100 countries, we're able to deliver a solution that addresses your biggest pain points.

At the heart of our Global Sensor Network are gateways: think of them as highway "on ramps" to the Internet. Gateways are operated by Internet Service Providers (ISPs) like Verizon, Comcast, and AT&T – and give your device the ability to send and receive information online.

The Global Sensor Network monitors the flow of information through these gateways, keeping track of changes as they occur. This helps our customers visualize the relationship between content owners (companies like Netflix, Facebook or Pinterest), the delivery platforms they choose (providers like AWS, Akamai or Google Cloud), and the volume of traffic each delivery platform serves.

Data sources

Intricately scours the digital divide for the following types of data:

• Applications. Once we get our hands on an application, we're able to do some profiling and investigate its dependencies. So, for all intents and purposes, applications are simply software that are running on some type of hardware that we can touch or "ping" or investigate via the public Internet.
• Public caches. Public caching infrastructure is a combination of the infrastructure that your ISP has in place, and the infrastructure that large content delivery providers operate – not just CDNs, but organizations like AWS, Microsoft Azure, and Google Cloud. These providers have loads of infrastructure tasked with caching and remembering the requests that were made, as well as the responses were sent. At Intricately, we tap into this and are able to extract a plethora of insightful usage and app-level data.
• Peering and network routing. This is a core aspect of how networks communicate with each other. If you're familiar with Internet infrastructure, it's a collection of autonomous networks. They share routing information, peering information, and we leverage that to understand relationships.
• Public records. We look at public records. Many public records aren't easy accessible or easy accessible to humans. The most common is the whois system – designed to help humans understand the owners of digital and physical internet resources. For example, if you were to Google "who is intricately.com?" – you'll get a registration record. You can do the same for thing IP addresses. So that's one example of public records that we monitor, tap into, and gives us a view into applications and IP space.
• Infrastructure. Another vital source, infrastructure alludes to the firewalls, routers, servers, peering exchanges – all of the backend infrastructure that powers the web. Intricately closely monitors this source of data, which ultimately allow us us to identify which applications running on certain IP spaces.
• Third-party traffic data. While we collect our own traffic data, we also purchase traffic data from third parties to supplement our own. We source the majority of our traffic data is from a service called Alexa.
• Published content. Published content is the most recent data source we've added, and includes webpages, social networks, and more. We've found there to be an increased need to monitor social media (for company and product updates) and even things like job postings, which can signal organizational expansion and growth.

Intricately's Global Sensor Network

Our intent behind building the Global Sensor Network was to create something incredibly purposeful and successful in tapping into very specific protocols and wavelengths that would allow us to extract the minimum about of data we needed. The network needed to be fault-tolerant, redundant, and available.

Looking at the sensor stack itself, there some common tools we leverage:

• DNS (dig)
• nmap
• whois
• phantomjs
• DNS resolver
• node

Data collection strategies: Achieving redundancy and scalability

1. Geo distribution and network diversity

After striking the right balance between network density and sensor location, our team realized that some sensors – depending on where they were deployed – could experience outages or hit resource limits.

To solve for these frequent outages and general instability, we started deploying many instances of the same sensor stack across a variety of locations. 

As of today's publication date, we can be found in over 50 locations around the world, and we are densely deployed – for any one location, there are typically 3-5 sensors. 

2. Application identification

Applications are truly the digital DNA of a business. At Intricately, try to identify as many applications as we can, in three ways:

• IP space. We crawl IP space, looking at signatures that are living on IP spaces that map to applications. Perhaps the most common type of this identification is an SSL certificate.
• DNS. We also crawl DNS constantly looking for applications. One technique we leverage is referred to as dictionary-based lookups. We have a list of approx. 200-300 application names we know companies use to publish other applications to, and we search for that app signature within DNS.
• Application dependencies. When we find an application from the first two sources, we profile it, and (when it runs) it eventually reveals other applications. Intricately uses this traffic that's generated to identify additional apps.

3. Capture steady state

There are countless variables that affect an application's behavior. Today's enterprises typically deploy a number of different infrastructure solutions that cause applications to behave differently depending on who you are, where you are, the time of day, or if you've visited the app before. They include:

• A/B testing
• Load balancing
• Geo-fencing
• Session-based logic

In order for Intricately to capture a comprehensive view of how an application behaves, we can't simply take a "snapshot" of an application. We have to take many snapshots over time, from lots of different locations, and lots of different networks.

Fortunately, if you're patient, you'll find that the collection of snapshots will start to repeat; signifying consistency within the app's behavior across all protocols – webpages, DNS, BGP, IP records, and more.

• Which companies use Amazon Web Services to host their online presence?
• How much are these companies spending, and how has that changed over time?
• Where are Facebook’s data centers located? Who operates them?

Intricately helps sales and marketing teams at cloud companies identify new target accounts based on usage and spend data

Schedule a demo with us to align your sales and marketing team with the right target accounts for your next go-to-market launch.