Botify API | Analysis | URLs Datamodel

The Botify API uses BQL (Botify Query Language) to perform incisive operations on analyzed URLs. BQL lets you define fields that you can group, filter, select, or compute metrics for, giving you complete control to manipulate data the way you want to. This lets developers utilize some of the most powerful functions in Botify, including:

Use BQL to build apps that harness some of Botify’s premier functions, all while gaining unfettered access to your Botify data! Botify clients can gain full access by contacting your Botify CSM today.

Analysis Datamodel

Feature	Id	Name	Permissions	Type	SubType	Multiple	Group

Edit this page

Permissions

Describes the list of operations the field can be used for. The rights may include the following:
- select: can be used in BQLQuery fields.
- filters: can be used in BQLFilter.
- filters_exist: can only be used with a exist predicate in a BQLFilter.
- sort: can be used in BQLFilter sort.
- agg:categorical: can be used in BQLAggsQuery groupby.
- agg:numerical: can be used in BQLAggsQuery range groupby.

Type

Describes the basic type of the data as it is stored. It can be one of: integer, long, float, double, date, datetime, string, boolean or tree. It defines which predicates are available and which type of input to use.

SubType

Describes the concept represented by the field, such as url_status or time_millisec. If the field doesn’t represent any specific concept, subtype is the same as type. List of subtypes may depend on the datamodel. It allows to know how to render the field data.

url

Standalone URL

string

image_url

URL of an image

string

url_status

URL with its crawled state. If not crawled, no information can be obtained using the URLs endpoint.

{
  "url": string,
  "crawled": boolean
}

url_link_status

URL with its crawled state and its follow/nofollow status

{
  "status": string,
  "url": {
    "url": string,
    "crawled": boolean
  },
}

url_http_code

URL with its HTTP status code

[string, string] // [URL, HTTP Status Code]

string_nb_map

List of strings with their number of occurrences.

[
  {
    "text": string,
    "nb": integer,
  }
]

time_millisec

Number of milliseconds (duration)

integer

hreflang_valid_values

{
    "url": {
        "url": string,
        "crawled": boolean
    },
    "warning": Array<string>,
    "value": string,
},

Warning	Description
WARNING_DEST_BLOCKED_ROBOTS_TXT	URL referenced by hreflang tag is blocked by (virtual) robots.txt.
WARNING_DEST_BLOCKED_CONFIG	URL referenced by hreflang tag was not crawled due to project settings.
WARNING_DEST_NOT_CRAWLED	URL referenced by hreflang tag was not crawled.

hreflang_error_values

{
    "url": {
        "url": string,
        "crawled": boolean
    },
    "errors": Array<string>,
    "value": string,
},

Error	Description
ERROR_LANG_NOT_SET	Language is not set in the HTML tag.
ERROR_NOT_COMPLIANT	URL is not compliant
ERROR_LANG_NOT_EQUAL	Language in hreflang tag does not match the language of referenced URL.
ERROR_LANG_NOT_RECOGNIZED	Language in hreflang tag does not exist or is not supported.
ERROR_REGION_NOT_RECOGNIZED	Region in hreflang tag does not exist or is not supported.
ERROR_REGION_NOT_EQUAL	Region in hreflang tag does not match the region of referenced URL.
ERROR_DEST_NOT_COMPLIANT	URL referenced by hreflang tag is not compliant.

Multiple

Some fields can contain a list of values, they are called multiple fields. For instance, query_string_keys could be equal to ['page', 'length'] on a URL with pagination.

Area

In almost every analysis operation dealing with URL Data model, an area parameter is present (its default value is current). It refers to the subset of URL to compute on:

URLs distribution between areas
URLs distribution chart available in movements tab of the analysis report.

current: URLs that were crawled in the current analysis (blue and purple part)
new: URLs crawled in the current analysis but not in the previous analysis (blue part)
disappeared: URLs crawled in the previous analysis but not in the current analysis (red part)

Datamodel