User Guide
Synapse-Google Search User Guide
Synapse-Google Search adds new Storm commands to allow you to query the
Google Search API using the /customsearch your existing API key.
Additionally, each URL in the search results is also downloaded to the Axon, and the corresponding inet:urlfile
node is created.
Getting Started
Check with your Admin to enable permissions and find out if you need a personal API and CX key.
Examples
Setting your personal API key
To set-up a personal use API key and CX key:
> google.search.setup.apikey --self myapikey
Setting Synapse-Google Search API key for the current user.
> google.search.setup.cx --self mycxkey
Setting Synapse-Google Search CX key for the current user.
Performing Google Searches
The google.search.enrich command will perform a search based on the primary property of a node. In this case, we’re
searching for the IP address 1.2.3.4.
> inet:ipv4=1.2.3.4 | google.search.enrich
inet:ipv4=1.2.3.4
.created = 2024/12/20 18:03:11.343
:type = unicast
You can also yield the search results as well.
> inet:ipv4=1.2.3.4 | google.search.enrich --yield | limit 3
inet:search:result=a6c2b569b635be4b3d54e6c8d145f82f
.created = 2024/12/20 18:03:11.577
:query = 4508c01380c0663bcb7fe523ae1c5003
:rank = 0
:title = kegg enzyme: 1.2.3.4
:url = https://www.genome.jp/dbget-bin/www_bget?ec:1.2.3.4
inet:search:result=5bebbfeecad04b9ac0e854689cb15062
.created = 2024/12/20 18:03:11.592
:query = 4508c01380c0663bcb7fe523ae1c5003
:rank = 1
:title = 1.2.3.4 ip address geolocation lookup demo | ip2location
:url = https://www.ip2location.com/demo/1.2.3.4
inet:search:result=63c8e5a6b0484f364d8cfcb6ea70708c
.created = 2024/12/20 18:03:11.606
:query = 4508c01380c0663bcb7fe523ae1c5003
:rank = 2
:title = 1.2.3.4 oxalate oxidase - enzyme
:url = https://enzyme.expasy.org/EC/1.2.3.4
From those results, you can pivot directly to the URLS.
> inet:ipv4=1.2.3.4 | google.search.enrich --yield | limit 3 | -> inet:url
inet:url=https://www.genome.jp/dbget-bin/www_bget?ec:1.2.3.4
.created = 2024/12/20 18:03:11.579
:base = https://www.genome.jp/dbget-bin/www_bget
:fqdn = www.genome.jp
:params = ?ec:1.2.3.4
:path = /dbget-bin/www_bget
:port = 443
:proto = https
inet:url=https://www.ip2location.com/demo/1.2.3.4
.created = 2024/12/20 18:03:11.594
:base = https://www.ip2location.com/demo/1.2.3.4
:fqdn = www.ip2location.com
:params =
:path = /demo/1.2.3.4
:port = 443
:proto = https
inet:url=https://enzyme.expasy.org/EC/1.2.3.4
.created = 2024/12/20 18:03:11.608
:base = https://enzyme.expasy.org/EC/1.2.3.4
:fqdn = enzyme.expasy.org
:params =
:path = /EC/1.2.3.4
:port = 443
:proto = https
The google.search command can be used to perform arbitrary searches based on user provided strings.
> google.search "cloud computing" --yield | limit 3
inet:search:result=7da0738fe552baebd7bf02636a752f06
.created = 2024/12/20 18:03:13.719
:query = a8b7c54a2e74d0509cb4d1c0c2bb9c2e
:rank = 0
:title = cloud computing - wikipedia
:url = https://en.wikipedia.org/wiki/Cloud_computing
inet:search:result=0d1f1d6f82b4be291b3bbaf1762a8b55
.created = 2024/12/20 18:03:13.732
:query = a8b7c54a2e74d0509cb4d1c0c2bb9c2e
:rank = 1
:title = what is cloud computing? | ibm
:url = https://www.ibm.com/cloud/learn/cloud-computing
inet:search:result=5e5f1254b42698940ac874c7f4c540ea
.created = 2024/12/20 18:03:13.744
:query = a8b7c54a2e74d0509cb4d1c0c2bb9c2e
:rank = 2
:title = what is cloud computing? a beginner's guide | microsoft azure
:url = https://azure.microsoft.com/en-us/overview/what-is-cloud-computing/
If you wanted to do exact matches in Google searches, you need to enclose the string with quotes. You can do the following as an example:
> google.search '"vertex project synapse"' --yield | limit 2
inet:search:result=aff538824f8e5f6190b295d94f4ed46c
.created = 2024/12/20 18:03:13.906
:query = 4021d7a29af52a33150189a12611e8f8
:rank = 0
:title = synapse documentation release 2.54.0 the vertex project
:url = https://synapse.docs.vertex.link/_/downloads/en/stable/pdf/
inet:search:result=25a0239f69ee632c7a611e8e94371197
.created = 2024/12/20 18:03:13.923
:query = 4021d7a29af52a33150189a12611e8f8
:rank = 1
:title = the vertex project - synapse is a versatile central intelligence ...
:url = https://lu.ma/vertexproject
Since this is a user provided string, it can also be a variable. The following example searches for file hash and word malware together.
> file:bytes#cno.mal +:md5 $text=$lib.str.format('{m} malware', m=:md5) google.search $text --debug --yield | limit 3
Searching google for [a38a367d6696ba90b2e778a5a4bf98fd malware]
inet:search:result=aa4a73ee1f6a76ed122b5867c19e078b
.created = 2024/12/20 18:03:14.132
:query = 38f532a995e1322e3f624380c081d04c
:rank = 0
:title = apt1 static malware analysis: webc2-cson - vitali kremez
:url = http://vkremez.weebly.com/cyber-security/apt1-static-malware-analysis-webc2-cson_sample
Both of the Google commands create inet:search:query nodes. These have a :engine=google secondary property.
The inet:search:result nodes are linked back to these queries.
> inet:search:query
inet:search:query=38f532a995e1322e3f624380c081d04c
.created = 2024/12/20 18:03:14.070
:engine = google
:text = a38a367d6696ba90b2e778a5a4bf98fd malware
inet:search:query=4021d7a29af52a33150189a12611e8f8
.created = 2024/12/20 18:03:13.845
:engine = google
:text = "vertex project synapse"
inet:search:query=4508c01380c0663bcb7fe523ae1c5003
.created = 2024/12/20 18:03:11.499
:engine = google
:text = 1.2.3.4
inet:search:query=a8b7c54a2e74d0509cb4d1c0c2bb9c2e
.created = 2024/12/20 18:03:13.658
:engine = google
:text = cloud computing
By default, search result URLs will not have their content retrieved. This behavior can
be enabled with the --get-content option:
> google.search '"vertex project synapse"' --yield --get-content | limit 2
inet:search:result=aff538824f8e5f6190b295d94f4ed46c
.created = 2024/12/20 18:03:13.906
:query = 4021d7a29af52a33150189a12611e8f8
:rank = 0
:title = synapse documentation release 2.54.0 the vertex project
:url = https://synapse.docs.vertex.link/_/downloads/en/stable/pdf/
inet:search:result=25a0239f69ee632c7a611e8e94371197
.created = 2024/12/20 18:03:13.923
:query = 4021d7a29af52a33150189a12611e8f8
:rank = 1
:title = the vertex project - synapse is a versatile central intelligence ...
:url = https://lu.ma/vertexproject
This will retrieve the content of the URLs and store it in the Axon that the Cortex is configured to use. There is a
five minute timeout on this download, to account for pages which may not be online or accessible to the Cortex. You can
view the file:bytes created from a given search with the following pivot:
> inet:search:query:text='"vertex project synapse"' -> inet:search:result -> inet:url -> inet:urlfile -> file:bytes
file:bytes=sha256:33f882b1246a6e7bb71d8b78097a662f31e7682c8b1b86b2681676da04d5d275
.created = 2024/12/20 18:03:14.457
:md5 = a2d670d783515d3a1b1aeaa155bbfa32
:mime = text/html
:sha1 = b99f9ae0e19dc3771c0558913d29db18fb9f0300
:sha256 = 33f882b1246a6e7bb71d8b78097a662f31e7682c8b1b86b2681676da04d5d275
:size = 31
file:bytes=sha256:e0189b491f87bbce7a7bdc10ab8bb3123bfa0064e7d3dd2ea41b1727593908b0
.created = 2024/12/20 18:03:14.365
:md5 = de65eecf40f3d44eb96cd100f44e0c73
:mime = application/pdf
:sha1 = 995afe53a8ab8aefe40f1110bab9271693cfcd86
:sha256 = e0189b491f87bbce7a7bdc10ab8bb3123bfa0064e7d3dd2ea41b1727593908b0
:size = 62
Use of meta:source nodes
Synapse-Google Search uses a meta:source node and -(seen)> light
weight edges to track nodes observed from the Google Search API.
> meta:source=a753bb4cf2ff32f6f894e52624ec392c
meta:source=a753bb4cf2ff32f6f894e52624ec392c
.created = 2024/12/20 18:03:11.483
:name = google search api
Storm can be used to filter nodes to include/exclude nodes which have been observed by Synapse-Google Search. The following example shows how to filter the results of a query to include only results observed by Synapse-Google Search:
> #cool.tag.lift +{ <(seen)- meta:source=a753bb4cf2ff32f6f894e52624ec392c }