There's a new tool I'm using as part of my security assessments and it is
DirBuster. Developed by James Fisher and now available from OWASP, DirBuster's purpose is to sniff out the existence of directories and files on a server. Nothing more and nothing less. I did not have high expectations when grabbing a copy of DirBuster from the OWASP site. I was shocked at how many features it provides and how well it performs! Don't let the version number of 0.12 fool you. It's a very capable and polished tool.
DirBuster is written in Java (requires v1.6 or above), and the user interface is simple and intuitive. Even the look and feel is top-notch - much better than WebScarab or Burp Suite, for example, which use the default Swing look and feel. James wisely chose to use
JGoodies, an open source library designed to make a more esthetically-pleasing Java user interface.
Once you start a scan, DirBuster goes to work. It lists directories and files as they are found. Since some servers don't return 404/Not Found for non-existent directories and files, DirBuster identifies positive hits by comparing each response to a base response for a known, non-existent resource.
Some of the bells and whistles in DirBuster include:
- configurable number of threads and ability to throttle up or down while a scan is running
- ability to scan for directories, files, or both
- file checks are done using extension(s) of your choice
- ability to narrow a scan to a particular subdirectory
- ability to do recursive scanning
- ability to load payloads from a file or to configure pure brute forcing
- customizable request headers
- support for basic, digest, and NTLM authentication
- fuzzing capability for resources that are referenced by URL parameter
- ability to send traffic through a proxy
- an informative scan status screen
- report generation (text files)
- automatic update feature
For being so polished, DirBuster has some quizzical misspellings. Another small quibble is that the tool's "advanced options" are not saved after you close it down. The author told me there is a new version in the works, and both of these issues will be addressed.