The Botify API uses BQL (Botify Query Language) to perform incisive operations on analyzed URLs. BQL lets you define fields that you can group, filter, select, or compute metrics for, giving you complete control to manipulate data the way you want to. This lets developers utilize some of the most powerful functions in Botify, including:
Use BQL to build apps that harness some of Botify’s premier functions, all while gaining unfettered access to your Botify data! Botify clients can gain full access by contacting your Botify CSM today.
Permissions
Describes the list of operations the field can be used for. The rights may include the following:
- select: can be used in BQLQuery fields.
- filters: can be used in BQLFilter.
- filters_exist: can only be used with a exist predicate in a BQLFilter.
- sort: can be used in BQLFilter sort.
- agg:categorical: can be used in BQLAggsQuery groupby.
- agg:numerical: can be used in BQLAggsQuery range groupby.
Type
Describes the basic type of the data as it is stored. It can be one of: integer, long, float, double, date, datetime, string, boolean or tree. It defines which predicates are available and which type of input to use.
SubType
Describes the concept represented by the field, such as url_status or time_millisec. If the field doesn’t represent any specific concept, subtype is the same as type. List of subtypes may depend on the datamodel. It allows to know how to render the field data.
url
Standalone URL
string
image_url
URL of an image
string
url_status
URL with its crawled state. If not crawled, no information can be obtained using the URLs endpoint.
{
"url": string,
"crawled": boolean
}
url_link_status
URL with its crawled state and its follow/nofollow status
{
"status": string,
"url": {
"url": string,
"crawled": boolean
},
}
url_http_code
URL with its HTTP status code
[string, string] // [URL, HTTP Status Code]
string_nb_map
List of strings with their number of occurrences.
[
{
"text": string,
"nb": integer,
}
]
time_millisec
Number of milliseconds (duration)
integer
hreflang_valid_values
{
"url": {
"url": string,
"crawled": boolean
},
"warning": Array<string>,
"value": string,
},
| Warning |
Description |
| WARNING_DEST_BLOCKED_ROBOTS_TXT |
URL referenced by hreflang tag is blocked by (virtual) robots.txt. |
| WARNING_DEST_BLOCKED_CONFIG |
URL referenced by hreflang tag was not crawled due to project settings. |
| WARNING_DEST_NOT_CRAWLED |
URL referenced by hreflang tag was not crawled. |
hreflang_error_values
{
"url": {
"url": string,
"crawled": boolean
},
"errors": Array<string>,
"value": string,
},
| Error |
Description |
| ERROR_LANG_NOT_SET |
Language is not set in the HTML tag. |
| ERROR_NOT_COMPLIANT |
URL is not compliant |
| ERROR_LANG_NOT_EQUAL |
Language in hreflang tag does not match the language of referenced URL. |
| ERROR_LANG_NOT_RECOGNIZED |
Language in hreflang tag does not exist or is not supported. |
| ERROR_REGION_NOT_RECOGNIZED |
Region in hreflang tag does not exist or is not supported. |
| ERROR_REGION_NOT_EQUAL |
Region in hreflang tag does not match the region of referenced URL. |
| ERROR_DEST_NOT_COMPLIANT |
URL referenced by hreflang tag is not compliant. |
Multiple
Some fields can contain a list of values, they are called multiple fields. For instance, query_string_keys could be equal to ['page', 'length'] on a URL with pagination.
Area
In almost every analysis operation dealing with URL Data model, an area parameter is present (its default value is current). It refers to the subset of URL to compute on:
URLs distribution chart available in movements tab of the analysis report.
- current: URLs that were crawled in the current analysis (blue and purple part)
- new: URLs crawled in the current analysis but not in the previous analysis (blue part)
- disappeared: URLs crawled in the previous analysis but not in the current analysis (red part)
