OSINT Framework and Top OSINT Tools Used By Pentesters

What is OSINT?

Open-Source Intelligence Tools, abbreviated as OSINT, is an art of gathering data from publicly available sources. This is commonly referred to as an Open Source. The process of collecting data can be in a manual or automated manner. The OSINT Framework makes this process much easier for us.

Commonly used OSINT engines include but are not limited to Google, Bing, DuckDuckgo, etc. In the past, there were not many choices to gather publicly known information. Most journalists and investigators relied on newspapers and word of mouth for that purpose. But as technology is progressing, platforms/tools for OSINT have drastically increased. One has to decide which tools/sources to be used depending upon the desired outcomes.

Why we use OSINT?

Motivation to use OSINT may depend on person to person. Journalists may use it to complement their research against a particular topic. Others may use OSINT to know the upcoming trends in the general public.

When it comes to cybersecurity, OSINT plays a very vital role. For example, for red teamers, OSINT may help them to know their target. With the help of OSINT, they can get themselves familiar with their target in a very efficient and easy manner. And can use the information gathered in this phase in the later stage of the exercise such as dictionary attack, reconnaissance, phishing, etc.

For blue teamers, OSINT tools may help them identify the new attack trends to take necessary steps to safeguard their environment. Also, OSINT can be used to uncover the geographical location of the threat actors or may potentially uncover the faces behind the black screens.

Top tools used for OSINT?

Now that we have discussed what OSINT is and why we use it, a rational human being may ask, “How do we use OSINT tools?”. Don’t worry; we got you covered on that part as well. However, you might want to keep your coffee ready because it will be a long list of tools we will discuss in the later detail of this article.

So we have curated some top-notch tools for OSINT used by industry experts; below is the list of them.

1- OSINT Framework

OSINT Framework is an open-source web application that queries multiple data sources, including but not limited to Free Search Engines, Scraped websites, etc. It is by far one of the go-to tools for every OSINT lover.

Its biggest attraction is that it’s a web-based tool, and you don’t need to install anything on your machine. The display is a visually pleasing format. Initially, the system was solely for Information Security purposes, but nowadays, folks from every field, be it journalism, engineers, or just random guys searching for fun, use it.

Most of the data sources used by OSINT Framework are free, and some may require charges for getting the information necessary.

2- CheckUsernames – OSINT tool

If you have a couple of usernames during the information-gathering phase of a Penetration test, you can gather more information regarding the collected usernames. You do this by seeing where on the internet this exact username is used. CheckUsernames is a valuable tool for checking usernames.

Another use case for CheckUsername can be to consider starting your brand and finalizing a username on various social media platforms for marketing purposes. Checkusername can help you locate the sites that are using the same username as you desire.

CheckUsername searches for the available usernames on 500+ social media sites and can save you a lot of time if you have to make all this effort manually by crawling all the 500+ social media sites.

3- HaveIbeenPwned

HavIbeenPwned is one of the go-to tools for every security researcher developed and maintained by one of the best in the industry, Troy Hunt. It maintains a database of leaked passwords, emails, and phone numbers over the past few years.

At the time of writing this, it has 11,253,394,522 pwned accounts which are increasing every passing minute. Security breaches are happening very frequently recently, which demands security professionals to be vigilant all the time. HaveIbeenPwned can be of great use when you need to check if the latest public breach has your credentials or not.

One of the attractive features in HaveIbeenPwned for the organization is Domain search. It will notify the domain owners every time the system finds data in the breach against a specific domain. But to use that feature, one has to first verify that he owns that particular domain. After that, HaveIbeenPwned will send automated emails every time the system finds data in the breach.

It can also be used as API if there is a need to check for breaches against specific email IDs frequently.

4- Censys

Censys is a web crawler that obtains recent and exact data about any web-related gadget. It allows searching the internet with both IP and domain names.

You can search complete geographic and specialized insights on port 80/443 of any server, HTTP/HTTPS protocol details, responses against the basic GET request of the objective site, TLS/SSL details, WHOIS data.

One of the most prominent features regarding census is that it’s free. You can do as many queries as you want, and it has up-to-date data on the internet. So it’s just like getting free shodan services.

5- BuiltWith

If you’re a full-time penetration tester or doing bug hunting and need to see the underlying technologies against a target website, then BuiltWith is for you.

BuiltWith provides a handsome amount of data regarding the target website that includes but not limited to:

• What target CMS is the host using like WordPress, Drupal, etc.
• Information regarding the Javascript and CSS libraries like jquery versions.
• Server and Reverse Proxy information like Apache, IIS, Nginx, etc.

Along with a very fancy GUI, BuiltWith comes with API access as well, with which you can automate all the recon process using the programming language of your choice and can display it the way you want.

Also, suppose you need to look for sites on the internet that uses particular technology. In that case, you can do it easily with one click in BuiltWith.

6- BeenVerified

BeenVerified is a market-competitive application that looks through individuals in open web records. It’s pretty helpful to inquire more important data regarding any individual on the planet when leading an IT Security investigation to find details regarding an obscure individual.

When you are done, the search results will consist of every individual that matches the given name, alongside their subtleties, geographic area, telephone number, and so on. After the system gathers the results, BeenVerified will generate a comprehensive report of the data.

The incredible thing about it is that BeenVerified additionally incorporates data from government sources and criminal records.

BeenVerified’s data sources incorporate data from different data sets, liquidation records, vocation history, web-based media profiles, and surprisingly online photographs.

So, BeenVerified can serve as your first stop when you’re on a hunt to get specific information about an individual that is available on the internet.

7- Google Dorks

Google Dorks is by far one of the most used techniques to search any information anywhere on the internet. However, Google dork is still underrated in its impact when used efficiently by an expert “Googler.”

It’s live since 2002 and can assist a great deal in your intel-gathering during the reconnaissance phase of the engagement. Google Dorks are uniquely formatted queries to question Google about specific data, which can be valuable for the ongoing investigation.

Word Wide Web searching tools or commonly known as “Search Engine,” file a massive amount of data constituting almost everything on the web. This includes individuals, organizations, and related information. 

Few common google dorks that can help you to get juicy information along with their brief explanation is below:

• Filetype:” pdf” – looks for all files having pdf format.
• Ext:” json” – looks for all JSON files
• Intext:” Confidential Information” – Searches for a specific keyword on the internet, in this case, “Confidential Information.”
• Title:” Sensitive Information” – instructs google search engine to look for web pages with “Senstive Information” present in their title tag.
• Inurl:”/admin/” – tells google search engine to look for specific URLs on the web with “/admin/” in there.

The Google Hacking Database section of exploit-DB features several interesting google dorks.

Now you have seen the power; it spits out numerous admin panels available on the internet within no time. An ethical hacker might use it as a starting point. They might narrow down the search to the specific organization and start its hacking from there.

8- Maltego

Maltego is a splendid utility to find the footprint of a target host in a pleasing graphical format. It comes as a default built-in utility in Kali Linux and was created by Paterva. Maltego allows its users to initiate information gathering against a target host, individual, or IP.

The best thing about Maltego, from my point of view, is its feature called ‘transforms’. It can be accessed freely without any cost involved once you’re signed up at Maltego. At the same time, on other tools, the only commercial version is available. Transforms runs parallel scans against a supplied hosts and tries to gather as much information as it can against, which includes but is not limited to corporate email addresses, Phone numbers, Shared hosting information, etc.

Initially, every new user has to sign up at the Maltego platform, and from that point forward, you can start gathering information against the desired host, domain or IP, etc. When the correct target has been set, Maltego will collect all the information present in their servers and transform them into a visually pleasing graphical format.

The final result of Maltego constitutes a visually pleasing graph. All the information against the supplied target is displayed and linked together, including IP address, Domain Information, Hosting information, etc.

9- Recon-Ng

It is a framework that conducts open-source reconnaissance smoothly and entirely with just using few CLI commands. Recon-ng is also a default tool of Kali Linux and one of the great tools for surveillance as it provides a complete framework for it.

It’s a Python-based tool and incorporates numerous features, comfort capacities, and intelligent assistance to manage you on the most proficient method to utilize it appropriately. Moreover, the straightforward interface permits you to run regular activities like connecting with a database, running web requests, and managing APIs.

Getting data about any objective is quite simple and should be possible inside the space of seconds. It incorporates fascinating modules for search engines like google and bing that can discover essential data related to the objective areas.

Although some modules of Recon-ng don’t contribute that much in the reconnaissance while others will collect an overwhelming amount of data from publicly available data sources.

10- the harvester -OSINT

The harvester is also a built-in facility of Kali Linux and can gather information, including but not limited to subdomains, open ports, virtual hosts against any supplied target.

Such information is precious when you are in the initial steps of a penetration test against your local network or the public target.

It utilizes numerous assets to bring the Information like PGP keys, Bing, Baidu, Yahoo, and Google web crawler, and social media platforms like Linkedin, Twitter, and Google Plus.

Automated Penetration Testing can be run against any supplied target using the harvester like DNS brute force, rDNS queries, and DNS TLD extension Bruteforce using curated dictionaries for the target. It is effortless to utilize yet incredible and effective tool intended to be used in the beginning phases of a penetration test or red teaming. Additionally, use it for open-source intelligence (OSINT) social events to help decide an organization’s outside danger scene on the web.

11- Shodan

Shodan is a search engine for security professionals that focus on gathering all the online devices under one roof. John Matherly created this masterpiece in 2009, and the idea behind its creation was to monitor all the publicly accessible devices, including Servers, Workstations, IoT, etc.

Commonly known as ‘hacker’s search engine,’ shodan allows you to discover and investigate alternate devices associated with the internet like servers, switches, webcams, etc.

Shodan is very similar to Google. However, instead of showing you extravagant pictures and rich substance/enlightening sites, it will deliver results that are more identified within the premium of IT security analysts like RTSP, SNMP, FTP, IMAP, SSH, Telnet, and HTTP standards and publicly available data. In addition, the user can filter results by country, working framework, organization, and ports.

It does not allow searches of live webcams, servers, IoT devices or routers, etc. Still, it can potentially check nearly whatever associates with the web. Examples include intelligent sensors, water plants, traffic lights, and more.

12- Jigsaw

Jigsaw accumulates data related to any publicly accessible servers. Therefore, this utility is pretty good if your target is organizations like LinkedIn, Microsoft, or Google, where one can get the Top Level Domain quickly (like linkedin.com). Afterward, we can further enumerate it in various domains, and Jigsaw can collect data from these subdomains.

The only disadvantage of using Jigsaw is that it collects data from one source only, and that is Jigsaw.com. Hence, we have to trust them for all the search results since we can’t cross-check the results.

If your target is big tech giants like Google, Microsoft, Linkedin, for any ethical reason, then Jigsaw will assist you greatly in the reconnaissance process. However, if you are investigating a not-so-well-known startup, you might be up the creek without a paddle.

13- SpiderFoot

SpiderFoot is a standout amongst other publicly available information-gathering tools. If you need to automate OSINT and want to have quick outcomes regarding information gathering, threat intel, and perimeter monitoring, SpiderFoot will come in handy.

Information Security expert Steve Micallef worked hard to structure this application.

It can assist with dispatching inquiries for more than 100 public information feeds to accumulate knowledge. For example, it can accumulate IP addresses, generic names, email, domain info, etc.

Spider foot is practically simple; indicate the objective, pick which modules you need to run. Then, it will perform the tedious tasks for you in gathering the intel information from the modules while you relax.

14- Creepy

Creepy is a geographical location identifier tool that information security experts and federal investigators use. Creepy can access complete geolocation information about any person by querying social media platforms like Facebook, Twitter, Flickr, and so forth.

Suppose anybody uploads a picture to any of these social media platforms with geolocation feature actuated. In that case, creepy can dig out the exact location coordinates for us. This allows us to know the precise geographical location where the user took the image.

It allows users to filter the data based on geographical location or even by date. From that point onward, you can extract the results in the desired format like KML or CSV.

It comes in handy to investigate the geographical location of an individual in this era where everyone is using VPNs. Often, we cannot get the location statistics based on IP addresses.

15- NMAP

Nmap is the most common tool in the information security industry. Calling it a “hearbeat” of any Information Security professional will not be an exaggeration. “Network Mapper,” commonly known as Nmap, comes with various distinct features, including port scan, vulnerability scan, OS scan, etc.

A portion of the primary highlights include:

• Host Discovery: It can discover live hosts in a public or private network by sending a bunch of ICMP packets. It can also analyze the response of each packet and enumerate SMB as well.
• DNS and IP Data Gathering: This allows you to gather DNS and IP-related information against the target host like domain name, SSL details, etc.
• Ports: With NMAP, one can easily find the open ports against a public or private target. Not only it does TCP scan, but it does UDP scanning as well to make sure no ports are left.
• Operating System Discovery: It can grab full OS details and underlying hardware details of any associated host like MAC address, Vendor name, etc.
• Version Scan: Nmap is also ready to get version details of the applications running on the target host and their names.

16- WebShag -OSINT

WebShag is a beautiful server scanning tool. Just like other tools, it’s a built-in feature for Kali Linux. Also, it can help you a great deal in your IT security research and penetration testing.

We can run a basic ping sweep scan or utilize progressed techniques like a proxy or over HTTP auth.

It’s developed in Python and serves as a best friend when you are auditing client systems.

Fundamental highlights are:

• Web Crawl to get all the paths
• Discovering common files
• Scanning for open ports
• Scanning the particular URL

It also prevents ISPs and servers from blocking you by utilizing a keen IDS avoidance framework. This way, you can keep examining the target host without the fear of raising any flags.

17- OpenVAS

Open Vulnerability Assessment System, commonly known as OpenVAS, is an open-source tool.

It’s an open-source version of Nessus. The creators of Nessus created it for the community when Nessus became a commercial tool.

Like Nessus, it also has the below features:

• Vulnerability Scanning
• Port Scanning
• Scheduled San
• Credentialed Scanning

For enterprises that need to upgrade their security with a smaller budget, OpenVAS will come in handy for them. OpenVAS has probably all the features that Nessus has to offer.

Although setting up OpenVAS can be cumbersome, it adds value to the Vulnerability and Patch Management process.

After it scans the target host, it generates a professional report including executive summary, finding details, and their proposed fixes. It can also be stored in a separate database.

18- Fierce -OSINT

Fierce is PERL based reconnaissance tool for DNS and IP, popular for aiding IT sec experts to discover IP-related domain names.

RSnake was the original author alongside different individuals from the old http://ha.ckers.org/. It is mostly used to target organizations.

Whenever you have characterized your objective organization, it will perform subdomain discovery against the particular TLD domain and perform various scans to assess the health of domains.

It is super fast and generates a lot more shorter duration than other tools like Nessus, OpenVAS, unicorn, etc.