Fetch URLs with GetAllUrls (GAU)

Getting as much information about the target as possible is very important during penetration testing. If you are a seasoned bug bounty hunter you know that sometimes there are low hanging fruits. Hidden subdomain with exposed secret, or a password might result in a big payout. That’s why it is very important to find everything that it is possible to find about the target. So, to the help comes GetAllUrls, also known was GAU. It is a tool for fetching URLs.

Currently GAU has over 1700 stars on GitHub.

What GAU Can Be Used Of

This tool can be used for fetching known URLs about the target. Unlike the Waybackurls, it is capable of fetching links from a few different sources.

GAU is a CLI tool that works simply by entering its name to the terminal:

gau example.com

We will get into details how to install GetAllUrls on Kali Linux, but for new lets see where does the tool searches for the data:

• AlienVault’s Open Threat Exchange. A crowd-sourced platform is a great source to gather information about malware. It was created to find potential threats on the internet. This is one of the sources you should definitely look into if you are performing reconnaissance.
• Wayback Machine. This is a respectable source that can be used to find historic URLs. If the website you are looking at, was there for long enough, there might be copies from years ago. Using this source would help you to find debug information, or even login credentials, that might still be relevant.
• Common Crawl. This source might be less known to the public than Web Archive, but it sure does provide a lot of value. The project has many data crawled about different websites. And the good news is that it is a non-profit and all data is free.

How to Install GetAllUrls, (Or Gau) on Kali Linux

There are different ways how you can install the GetAllUrls:

• Installing directly from the source
• Downloading the binary and adding it to your system $PATH
• Creating a Docker container

If you are willing to install it directly from the source, you must have a Golang on your system:

sudo apt install golang-go

Now add the path of Golang to your .bashrc file. Write the path (export PATH=”$PATH:$HOME/go/bin”) to the file:

sudo nano .bashrc

Now you are ready to install GetAllUrls:

go install github.com/lc/gau/v2/cmd/gau@latest

To make sure that the installation went fine, you may check the help menu:

gau -h

If you prefer a Docker container, you may create one:

docker run --rm sxcurity/gau:latest --help

This command will immediately download the image and output the Gau help menu. If you want to use the tool, you must enter the Docker container and invoke it from the CLI.

And finally, the third way is to download the binary and move it to the path:

tar xvf gau_2.0.6_linux_amd64.tar.gz
mv gau /usr/bin/gau

GAU Flags

• –blacklist. This one can be used to skip a list of unwanted extensions, for example png, svg, etc. (ex. “gau –blacklist png”).
• –fc. Used to filter unwanted status code (for example, command “gau -fc 404” filters URLs with 404 response code).
• –from. Can be used for setting date of the URLs that shall be fetched (ex. gau –from 202202).
• –ft. Can be used for excluding mime-types (ex. gau –ft text/plain).
• -h. Used for printing the help information of GetAllUrls.
• –json. Used for outputting results to JSON file.
• –mc. If specified with -mc flag and a status codes, Gau will look for the URLs with the given response statuses (ex. “gau –mc 200“).
• –mt. Can be used for looking for URLs with specific mime-types (ex. “gau -mt text/html“).
• –o. This flag can be used to set output file (ex. “gau -o results.txt“).
• –providers. If you do want to look only in one source (or a few), you can specify a provider(s) (ex. “gau –providers wayback“).
• –proxy. Can set proxy URL with the flag (ex. “gau –proxy http://localhost:8080“).
• –retries. Used for setting number of retries for HTTP client (ex. “gau –retries 10“).
• –subs. This can be used to include subdomains to the search (ex. “gau example.com –subs“).
• –threads. By setting this parameter you are able to control how many threads you want running (ex. “gau example.com –threads“).
• –to. Used to specify the date until which you want URLs to be fetched (ex. “gau –to 202202“).
• –verbose. To get more information information you can use the verbose flag (ex. “gau –verbose example.com“).
• –version. Used for checking version of GAU (ex. “gau —version”).

Final Words

GetAllUrls is a great tool that is capable of finding existing URLs of your target. While this tool is handy, do not forget that only by using a full spectrum of tools and methodologies you will be able to find as much security flaws, as possible.