Palo Alto Networks - Querying Multiple URL Categories at Once

Palo Alto Networks offers a way to query a URL via https://urlfiltering.paloaltonetworks.com/. However, if you want to query multiple URLs at the same time, then you need to leverage the API.

• Generate the API key for the firewall. See this link for instructions.
• Create a text file with all the URLs you want to check. For example, create a file called urls.txt with the URLs entered in the following format:

www.google.com
www.facebook.com
www.yahoo.com
purple.com
chase.com

• Create a bash file to query PAN-DB using the text file created in the previous step. For example, create a file called url_checker with the following details:

#!/bin/bash

for url in $(cat urls.txt); 

do curl -k 'https://{firewall ip}/api/?type=op&cmd=<test><url>'$url'</url></test>&key={api key here}'; 

done

• From the command line, run the bash script. For example, in OSX, ./url_checker
• The output should give you a list of all URLs in the file and their corresponding categories.

SJCMACF0UPG8WM:Desktop smitchell$ ./url_checker

<response cmd="status" status="success"><result>www.google.com search-engines (Base db) expires in 3000 seconds

www.google.com search-engines (Cloud db)

</result></response><response cmd="status" status="success"><result>www.facebook.com social-networking (Base db) expires in 3000 seconds

www.facebook.com social-networking (Cloud db)

</result></response><response cmd="status" status="success"><result>www.yahoo.com internet-portals (Base db) expires in 3000 seconds

www.yahoo.com internet-portals (Cloud db)

</result></response><response cmd="status" status="success"><result>purple.com home-and-garden (Base db) expires in 0 seconds

purple.com home-and-garden (Cloud db)

</result></response><response cmd="status" status="success"><result>chase.com financial-services (Base db) expires in 9000 seconds

chase.com financial-services (Cloud db)

</result></response>SJCMACF0UPG8WM:Desktop smitchell$

Stand-alone vs. Integrated URL Filtering

In the field, I hear a lot of administrators talk about how they prefer to use a point-product for URL filtering rather than collapse that service into a Palo Alto Networks firewall. I somewhat understand the value of this approach because its possible that a stand-alone URL filter is going to have some capabilities that are required by the organization (i.e. a specific type of report). However, the biggest issue with that approach is that there is no integration with other security products (i.e firewall, IPS, anti-malware, etc.). This means that not only any logs from security devices would need to be aggregated and correlated with yet another security device (i.e. SIEM), but more importantly it would be very difficult to provide automated outcomes due to the disparate nature of each point product. At the end of the day, we want to make our lives as administrators easier, right? I know I do! Here is an example of what I'm talking about:

One of the most prevalent ways that malware is delivered to an endpoint is via drive-by download over an SSL connection. This type of attack is typically used in conjunction with some other form of attack (i.e. spear phishing, watering hole, etc.). A drive-by download is a download that occurs without a user's knowledge when visiting a website. The access method (i.e. browser, application, etc.) is exploited to automate the process so that the user is unable to take action until it is too late. Attackers will register new domains that have yet to be categorized by URL filtering products in attempts to bypass URL categories traditionally categorized as malicious, phishing, etc. Palo Alto Networks firewalls can intercept drive-by downloads because its URL filtering, file blocking, and SSL decryption capabilities are natively-integrated.

PAN-DB URL filtering within Palo Alto Networks firewalls have an unknown category to match newly registered domains by attackers, but more importantly, URL categories (whether pre-defined or custom) can be leveraged as match criteria within a security policy.



The security policy above signifies that any traffic going from domain users on the internal network to any website categorized as unknown on the outside internet will be allowed, but have specific Content-ID profiles applied. In this particular policy, we specify a File Blocking profile called Drive-by.



The File Blocking profile above signifies that any application or file type that is seen will result in a continue action. This means that if a user accesses an unknown URL and a file download or upload is attempted, then a response page will be generated forcing the user to either confirm or deny the download. Other options include alert (log and allow) and block (log and block). This is even when the session is encrypted over SSL. The response page can be customized, of course.



In summary, as the number of devices connected to the internet continue to rise, the more threats will also continue to rise. Its almost impossible to keep up with disparate products, logs, etc. and no correlation. Leveraging a platform with native URL filtering integration allows us to automate outcomes with very little to no manual intervention on our part.