Changelog

Synapse-URLScan Changelog

v1.11.0 - 2024-05-15

Features and Enhancements

  • Add a --download-responses option to the urlscan.enrich, urlscan.search, and urlscan.scan commands to download available HTTP response bodies and downloaded files from results.

v1.10.0 - 2024-03-13

Features and Enhancements

  • Update $lib.bytes usage with $lib.axon APIs.

Bugfixes

  • Fix an issue where characters were not correctly escaped when using an inet:url node as input to the urlscan.search.incidents command.

v1.9.1 - 2024-02-20

Features and Enhancements

  • Update deprecated $lib.dict() usage to JSON style syntax.

v1.9.0 - 2024-01-23

Features and Enhancements

  • Add .seen timestamps to inet:url nodes seen during URL redirects.

v1.8.0 - 2023-11-27

Deprecations

  • Caching has been removed from the urlscan.enrich, urlscan.search, urlscan.search.certificates, and urlscan.search.hostnames commands, so the --asof argument has been deprecated and will no longer have any effect.

This release contains an automatic cache cleanup that will run when the package is first upgraded. This will remove existing cached API response data from the jsonstor.

v1.7.0 - 2023-10-11

Features and Enhancements

  • Add urlscan.search.incidents for searching the URLScan Pro incidents datasource.

v1.6.0 - 2023-08-25

Features and Enhancements

  • Add urlscan.search.hostnames and urlscan.search.certificates commands for searching additional urlscan Pro datasources.

  • Update error messages for unsuccessful HTTP requests to include the reason phrase in addition to the response code.

v1.5.1 - 2023-06-14

Bugfixes

  • Remove the use of a deprecated Storm function.

v1.5.0 - 2023-04-05

Features and Enhancements

  • Add --country option to urlscan.scan to allow specifying the country to perform the scan from.

  • Update ingest to populate inet:http:request:response:body when a SHA256 hash of the HTTP response body is available.

  • Update ingest to populate it:exec:url:http:request with the inet:http:request for the URL.

  • Update command help to clarify the use of inet:urlredir nodes for scans where the actual URL returned is not the same as the initial scan URL.

  • Set file:bytes:mime property on downloaded screenshots.

v1.4.0 - 2023-03-22

Features and Enhancements

  • Update Power-Up to build with code signing.

v1.3.0 - 2022-09-28

Features and Enhancements

  • Update urlscan.search and urlscan.scan to create it:exec:query nodes with -(found)> edges.

v1.2.0 - 2022-05-12

Features and Enhancements

  • Cached API responses are now stored in the JsonStor instead of in nodedata.

v1.1.1 - 2022-02-10

Bugfixes

  • Fix various spelling errors in command help output.

v1.1.0 - 2022-01-19

Features and Enhancements

  • Download screenshots when ingesting results with the --detail flag.

  • Add --dom and --screenshot flags to urlscan.enrich and urlscan.search. These can be used to download the DOM trees or screenshots of results instead of the full result as with --detail.

  • Add inet:urlfile nodes for files downloaded by pages.

  • Add Optic node action for urlscan.scan.

  • Add inet:url nodes to allowed forms for the urlscan.enrich node action.

Notes

  • DOM trees are now modeled using the :page:html property on it:exec:url nodes instead of a -(dom)> edge. The following Storm command can be used to migrate existing search results:

    > $lib.import(urlscan).migrateDomEdges()

v1.0.4 - 2021-11-17

Bugfixes

  • Add -(found)> edges to link results back to the inet:search:query node.

v1.0.3 - 2021-11-15

Features and Enhancements

  • Print clear warning when ingesting invalid search results

  • Default search size parameter moved to 10k

Bugfixes

  • Account for pagination of search queries with <10k results

v1.0.2 - 2021-11-12

Bugfixes

  • Fix pagination based ingest of URLScan search queries.

v1.0.1 - 2021-10-13

Bugfixes

  • Fix an issue where results containing an empty request object could raise an error.

v1.0.0 - 2021-08-13

Features and Enhancements

  • Initial release of Synapse-URLScan v1.0.0.