Relational Data - Functions - Url Functions

From FojiSoft Docs
Revision as of 18:52, 28 August 2024 by Chris.Hansen (talk | contribs) (Import ClickHouse Docs: Wed Aug 28 2024 14:52:41 GMT-0400 (Eastern Daylight Time))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)


’ ’ | ✗ |✗ |
  ✗ |✗ |
 < | ✗ |✗ |
 > | ✗ |✗ |
 % | ✗ |✔* |
 { | ✗ |✗ |
 } | ✗ |✗ |
 | | ✗ |✗ |
 \  | ✗ |✗ |
 ^ | ✗ |✗ |
 ~ | ✗ |✔* |
 [ | ✗ |✗ |
 ] | ✗ |✔ |
 ; | ✗ |✔* |
 = | ✗ |✔* |
 & | ✗ |✔* |

symbols marked * are sub-delimiters in RFC 3986 and allowed for user info following the @ symbol.

Functions that Extract Parts of a URL

If the relevant part isn’t present in a URL, an empty string is returned.

protocol

Extracts the protocol from a URL.

Examples of typical returned values: http, https, ftp, mailto, tel, magnet.

domain

Extracts the hostname from a URL.

Syntax 

domain(url)

Arguments

The URL can be specified with or without a protocol. Examples: 

svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/

For these examples, the domain function returns the following results: 

some.svn-hosting.com
some.svn-hosting.com
clickhouse.com

Returned values

  • Host name if the input string can be parsed as a URL, otherwise an empty string. String.

Example 

SELECT domain('svn+ssh://some.svn-hosting.com:80/repo/trunk');
┌─domain('svn+ssh://some.svn-hosting.com:80/repo/trunk')─┐
│ some.svn-hosting.com                                   │
└────────────────────────────────────────────────────────┘

domainRFC

Extracts the hostname from a URL. Similar to domain, but RFC 3986 conformant.

Syntax 

domainRFC(url)

Arguments

Returned values

  • Host name if the input string can be parsed as a URL, otherwise an empty string. String.

Example 

SELECT
    domain('http://user:password@example.com:8080/path?query=value#fragment'),
    domainRFC('http://user:password@example.com:8080/path?query=value#fragment');
┌─domain('http://user:password@example.com:8080/path?query=value#fragment')─┬─domainRFC('http://user:password@example.com:8080/path?query=value#fragment')─┐
│                                                                           │ example.com                                                                  │
└───────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────┘

domainWithoutWWW

Returns the domain without leading www. if present.

Syntax 

domainWithoutWWW(url)

Arguments

Returned values

  • Domain name if the input string can be parsed as a URL (without leading www.), otherwise an empty string. String.

Example 

SELECT domainWithoutWWW('http://paul@www.example.com:80/');
┌─domainWithoutWWW('http://paul@www.example.com:80/')─┐
│ example.com                                         │
└─────────────────────────────────────────────────────┘

domainWithoutWWWRFC

Returns the domain without leading www. if present. Similar to domainWithoutWWW but conforms to RFC 3986.

Syntax 

domainWithoutWWWRFC(url)

Arguments

Returned values

  • Domain name if the input string can be parsed as a URL (without leading www.), otherwise an empty string. String.

Example

Query: 

SELECT
    domainWithoutWWW('http://user:password@www.example.com:8080/path?query=value#fragment'),
    domainWithoutWWWRFC('http://user:password@www.example.com:8080/path?query=value#fragment');

Result: 

┌─domainWithoutWWW('http://user:password@www.example.com:8080/path?query=value#fragment')─┬─domainWithoutWWWRFC('http://user:password@www.example.com:8080/path?query=value#fragment')─┐
│                                                                                         │ example.com                                                                                │
└─────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────┘

topLevelDomain

Extracts the the top-level domain from a URL. 

topLevelDomain(url)

Arguments

The URL can be specified with or without a protocol. Examples: 

svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/

Returned values

  • Domain name if the input string can be parsed as a URL. Otherwise, an empty string. String.

Example

Query: 

SELECT topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk');

Result: 

┌─topLevelDomain('svn+ssh://www.some.svn-hosting.com:80/repo/trunk')─┐
│ com                                                                │
└────────────────────────────────────────────────────────────────────┘

topLevelDomainRFC

Extracts the the top-level domain from a URL. Similar to topLevelDomain, but conforms to RFC 3986. 

topLevelDomainRFC(url)

Arguments

The URL can be specified with or without a protocol. Examples: 

svn+ssh://some.svn-hosting.com:80/repo/trunk
some.svn-hosting.com:80/repo/trunk
https://clickhouse.com/time/

Returned values

  • Domain name if the input string can be parsed as a URL. Otherwise, an empty string. String.

Example

Query: 

SELECT topLevelDomain('http://foo:foo%41bar@foo.com'), topLevelDomainRFC('http://foo:foo%41bar@foo.com');

Result: 

┌─topLevelDomain('http://foo:foo%41bar@foo.com')─┬─topLevelDomainRFC('http://foo:foo%41bar@foo.com')─┐
│                                                │ com                                               │
└────────────────────────────────────────────────┴───────────────────────────────────────────────────┘

firstSignificantSubdomain

Returns the “first significant subdomain”. The first significant subdomain is a second-level domain for com, net, org, or co, otherwise it is a third-level domain. For example, firstSignificantSubdomain (‘https://news.clickhouse.com/’) = ‘clickhouse’, firstSignificantSubdomain (‘https://news.clickhouse.com.tr/’) = ‘clickhouse’. The list of “insignificant” second-level domains and other implementation details may change in the future.

Syntax 

firstSignificantSubdomain(url)

Arguments

Returned value

  • The first significant subdomain. String.

Example

Query: 

SELECT firstSignificantSubdomain('http://www.example.com/a/b/c?a=b')

Result: 

┌─firstSignificantSubdomain('http://www.example.com/a/b/c?a=b')─┐
│ example                                                       │
└───────────────────────────────────────────────────────────────┘

firstSignificantSubdomainRFC

Returns the “first significant subdomain”. The first significant subdomain is a second-level domain for com, net, org, or co, otherwise it is a third-level domain. For example, firstSignificantSubdomain (‘https://news.clickhouse.com/’) = ‘clickhouse’, firstSignificantSubdomain (‘https://news.clickhouse.com.tr/’) = ‘clickhouse’. The list of “insignificant” second-level domains and other implementation details may change in the future. Similar to firstSignficantSubdomain but conforms to RFC 1034.

Syntax 

firstSignificantSubdomainRFC(url)

Arguments

Returned value

  • The first significant subdomain. String.

Example

Query: 

SELECT
    firstSignificantSubdomain('http://user:password@example.com:8080/path?query=value#fragment'),
    firstSignificantSubdomainRFC('http://user:password@example.com:8080/path?query=value#fragment');

Result: 

┌─firstSignificantSubdomain('http://user:password@example.com:8080/path?query=value#fragment')─┬─firstSignificantSubdomainRFC('http://user:password@example.com:8080/path?query=value#fragment')─┐
│                                                                                              │ example                                                                                         │
└──────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────┘

cutToFirstSignificantSubdomain

Returns the part of the domain that includes top-level subdomains up to the “first significant subdomain”.

Syntax 

cutToFirstSignificantSubdomain(url)

Arguments

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain if possible, otherwise returns an empty string. String.

Example

Query: 

SELECT
    cutToFirstSignificantSubdomain('https://news.clickhouse.com.tr/'),
    cutToFirstSignificantSubdomain('www.tr'),
    cutToFirstSignificantSubdomain('tr');

Result: 

┌─cutToFirstSignificantSubdomain('https://news.clickhouse.com.tr/')─┬─cutToFirstSignificantSubdomain('www.tr')─┬─cutToFirstSignificantSubdomain('tr')─┐
│ clickhouse.com.tr                                                 │ tr                                       │                                      │
└───────────────────────────────────────────────────────────────────┴──────────────────────────────────────────┴──────────────────────────────────────┘

cutToFirstSignificantSubdomainRFC

Returns the part of the domain that includes top-level subdomains up to the “first significant subdomain”. Similar to cutToFirstSignificantSubdomain but conforms to RFC 3986.

Syntax 

cutToFirstSignificantSubdomainRFC(url)

Arguments

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain if possible, otherwise returns an empty string. String.

Example

Query: 

SELECT
    cutToFirstSignificantSubdomain('http://user:password@example.com:8080'),
    cutToFirstSignificantSubdomainRFC('http://user:password@example.com:8080');

Result: 

┌─cutToFirstSignificantSubdomain('http://user:password@example.com:8080')─┬─cutToFirstSignificantSubdomainRFC('http://user:password@example.com:8080')─┐
│                                                                         │ example.com                                                                │
└─────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────┘

cutToFirstSignificantSubdomainWithWWW

Returns the part of the domain that includes top-level subdomains up to the “first significant subdomain”, without stripping www.

Syntax 

cutToFirstSignificantSubdomainWithWWW(url)

Arguments

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain (with www) if possible, otherwise returns an empty string. String.

Example

Query: 

SELECT
    cutToFirstSignificantSubdomainWithWWW('https://news.clickhouse.com.tr/'),
    cutToFirstSignificantSubdomainWithWWW('www.tr'),
    cutToFirstSignificantSubdomainWithWWW('tr');

Result: 

┌─cutToFirstSignificantSubdomainWithWWW('https://news.clickhouse.com.tr/')─┬─cutToFirstSignificantSubdomainWithWWW('www.tr')─┬─cutToFirstSignificantSubdomainWithWWW('tr')─┐
│ clickhouse.com.tr                                                        │ www.tr                                          │                                             │
└──────────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────┴─────────────────────────────────────────────┘

cutToFirstSignificantSubdomainWithWWWRFC

Returns the part of the domain that includes top-level subdomains up to the “first significant subdomain”, without stripping www. Similar to cutToFirstSignificantSubdomainWithWWW but conforms to RFC 3986.

Syntax 

cutToFirstSignificantSubdomainWithWWW(url)

Arguments

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain (with “www”) if possible, otherwise returns an empty string. String.

Example

Query: 

SELECT
    cutToFirstSignificantSubdomainWithWWW('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy'),
    cutToFirstSignificantSubdomainWithWWWRFC('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy');

Result: 

┌─cutToFirstSignificantSubdomainWithWWW('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy')─┬─cutToFirstSignificantSubdomainWithWWWRFC('http:%2F%2Fwwwww.nova@mail.ru/economicheskiy')─┐
│                                                                                       │ mail.ru                                                                                  │
└───────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────┘

cutToFirstSignificantSubdomainCustom

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain. Accepts custom TLD list name. This function can be useful if you need a fresh TLD list or if you have a custom list.

Configuration example 

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
    <!-- https://publicsuffix.org/list/public_suffix_list.dat -->
    <public_suffix_list>public_suffix_list.dat</public_suffix_list>
    <!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax 

cutToFirstSignificantSubdomain(url, tld)

Arguments

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain. String.

Example

Query: 

SELECT cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list');

Result: 

┌─cutToFirstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐
│ foo.there-is-no-such-domain                                                                   │
└───────────────────────────────────────────────────────────────────────────────────────────────┘

See Also

cutToFirstSignificantSubdomainCustomRFC

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain. Accepts custom TLD list name. This function can be useful if you need a fresh TLD list or if you have a custom list. Similar to cutToFirstSignificantSubdomainCustom but conforms to RFC 3986.

Syntax 

cutToFirstSignificantSubdomainRFC(url, tld)

Arguments

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain. String.

See Also

cutToFirstSignificantSubdomainCustomWithWWW

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. Accepts custom TLD list name. It can be useful if you need a fresh TLD list or if you have a custom list.

Configuration example 

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
    <!-- https://publicsuffix.org/list/public_suffix_list.dat -->
    <public_suffix_list>public_suffix_list.dat</public_suffix_list>
    <!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax 

cutToFirstSignificantSubdomainCustomWithWWW(url, tld)

Arguments

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. String.

Example

Query: 

SELECT cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list');

Result: 

┌─cutToFirstSignificantSubdomainCustomWithWWW('www.foo', 'public_suffix_list')─┐
│ www.foo                                                                      │
└──────────────────────────────────────────────────────────────────────────────┘

See Also

cutToFirstSignificantSubdomainCustomWithWWWRFC

Returns the part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. Accepts custom TLD list name. It can be useful if you need a fresh TLD list or if you have a custom list. Similar to cutToFirstSignificantSubdomainCustomWithWWW but conforms to RFC 3986.

Syntax 

cutToFirstSignificantSubdomainCustomWithWWWRFC(url, tld)

Arguments

Returned value

  • Part of the domain that includes top-level subdomains up to the first significant subdomain without stripping www. String.

See Also

firstSignificantSubdomainCustom

Returns the first significant subdomain. Accepts customs TLD list name. Can be useful if you need fresh TLD list or you have custom.

Configuration example: 

<!-- <top_level_domains_path>/var/lib/clickhouse/top_level_domains/</top_level_domains_path> -->
<top_level_domains_lists>
    <!-- https://publicsuffix.org/list/public_suffix_list.dat -->
    <public_suffix_list>public_suffix_list.dat</public_suffix_list>
    <!-- NOTE: path is under top_level_domains_path -->
</top_level_domains_lists>

Syntax 

firstSignificantSubdomainCustom(url, tld)

Arguments

Returned value

  • First significant subdomain. String.

Example

Query: 

SELECT firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list');

Result: 

┌─firstSignificantSubdomainCustom('bar.foo.there-is-no-such-domain', 'public_suffix_list')─┐
│ foo                                                                                      │
└──────────────────────────────────────────────────────────────────────────────────────────┘

See Also

firstSignificantSubdomainCustomRFC

Returns the first significant subdomain. Accepts customs TLD list name. Can be useful if you need fresh TLD list or you have custom. Similar to firstSignificantSubdomainCustom but conforms to RFC 3986.

Syntax 

firstSignificantSubdomainCustomRFC(url, tld)

Arguments

Returned value

  • First significant subdomain. String.

See Also

port

Returns the port or default_port if the URL contains no port or cannot be parsed.

Syntax 

port(url [, default_port = 0])

Arguments

  • url — URL. String.
  • default_port — The default port number to be returned. UInt16.

Returned value

  • Port or the default port if there is no port in the URL or in case of a validation error. UInt16.

Example

Query: 

SELECT port('http://paul@www.example.com:80/');

Result: 

┌─port('http://paul@www.example.com:80/')─┐
│                                      80 │
└─────────────────────────────────────────┘

portRFC

Returns the port or default_port if the URL contains no port or cannot be parsed. Similar to port, but RFC 3986 conformant.

Syntax 

portRFC(url [, default_port = 0])

Arguments

  • url — URL. String.
  • default_port — The default port number to be returned. UInt16.

Returned value

  • Port or the default port if there is no port in the URL or in case of a validation error. UInt16.

Example

Query: 

SELECT
    port('http://user:password@example.com:8080'),
    portRFC('http://user:password@example.com:8080');

Result: 

┌─port('http://user:password@example.com:8080')─┬─portRFC('http://user:password@example.com:8080')─┐
│                                             0 │                                             8080 │
└───────────────────────────────────────────────┴──────────────────────────────────────────────────┘

path

Returns the path without query string.

Example: /top/news.html.

pathFull

The same as above, but including query string and fragment.

Example: /top/news.html?page=2#comments.

protocol

Extracts the protocol from a URL.

Syntax 

protocol(url)

Arguments

  • url — URL to extract protocol from. String.

Returned value

  • Protocol, or an empty string if it cannot be determined. String.

Example

Query: 

SELECT protocol('https://clickhouse.com/');

Result: 

┌─protocol('https://clickhouse.com/')─┐
│ https                               │
└─────────────────────────────────────┘

queryString

Returns the query string without the initial question mark, # and everything after #.

Example: page=1&lr=213.

fragment

Returns the fragment identifier without the initial hash symbol.

queryStringAndFragment

Returns the query string and fragment identifier.

Example: page=1#29390.

extractURLParameter(url, name)

Returns the value of the name parameter in the URL, if present, otherwise an empty string is returned. If there are multiple parameters with this name, the first occurrence is returned. The function assumes that the parameter in the url parameter is encoded in the same way as in the name argument.

extractURLParameters(url)

Returns an array of name=value strings corresponding to the URL parameters. The values are not decoded.

extractURLParameterNames(url)

Returns an array of name strings corresponding to the names of URL parameters. The values are not decoded.

URLHierarchy(url)

Returns an array containing the URL, truncated at the end by the symbols /,? in the path and query-string. Consecutive separator characters are counted as one. The cut is made in the position after all the consecutive separator characters.

URLPathHierarchy(url)

The same as above, but without the protocol and host in the result. The / element (root) is not included. 

URLPathHierarchy('https://example.com/browse/CONV-6788') =
[
    '/browse/',
    '/browse/CONV-6788'
]

encodeURLComponent(url)

Returns the encoded URL.

Example: 

SELECT encodeURLComponent('http://127.0.0.1:8123/?query=SELECT 1;') AS EncodedURL;
┌─EncodedURL───────────────────────────────────────────────┐
│ http%3A%2F%2F127.0.0.1%3A8123%2F%3Fquery%3DSELECT%201%3B │
└──────────────────────────────────────────────────────────┘

decodeURLComponent(url)

Returns the decoded URL.

Example: 

SELECT decodeURLComponent('http://127.0.0.1:8123/?query=SELECT%201%3B') AS DecodedURL;
┌─DecodedURL─────────────────────────────┐
│ http://127.0.0.1:8123/?query=SELECT 1; │
└────────────────────────────────────────┘

encodeURLFormComponent(url)

Returns the encoded URL. Follows rfc-1866, space() is encoded as plus(+).

Example: 

SELECT encodeURLFormComponent('http://127.0.0.1:8123/?query=SELECT 1 2+3') AS EncodedURL;
┌─EncodedURL────────────────────────────────────────────────┐
│ http%3A%2F%2F127.0.0.1%3A8123%2F%3Fquery%3DSELECT+1+2%2B3 │
└───────────────────────────────────────────────────────────┘

decodeURLFormComponent(url)

Returns the decoded URL. Follows rfc-1866, plain plus(+) is decoded as space().

Example: 

SELECT decodeURLFormComponent('http://127.0.0.1:8123/?query=SELECT%201+2%2B3') AS DecodedURL;
┌─DecodedURL────────────────────────────────┐
│ http://127.0.0.1:8123/?query=SELECT 1 2+3 │
└───────────────────────────────────────────┘

netloc

Extracts network locality (username:password@host:port) from a URL.

Syntax 

netloc(url)

Arguments

Returned value

  • username:password@host:port. String.

Example

Query: 

SELECT netloc('http://paul@www.example.com:80/');

Result: 

┌─netloc('http://paul@www.example.com:80/')─┐
│ paul@www.example.com:80                   │
└───────────────────────────────────────────┘

Functions that remove part of a URL

If the URL does not have anything similar, the URL remains unchanged.

cutWWW

Removes leading www. (if present) from the URL’s domain.

cutQueryString

Removes query string, including the question mark.

cutFragment

Removes the fragment identifier, including the number sign.

cutQueryStringAndFragment

Removes the query string and fragment identifier, including the question mark and number sign.

cutURLParameter(url, name)

Removes the name parameter from a URL, if present. This function does not encode or decode characters in parameter names, e.g. Client ID and Client%20ID are treated as different parameter names.

Syntax 

cutURLParameter(url, name)

Arguments

Returned value

  • url with name URL parameter removed. String.

Example

Query: 

SELECT
    cutURLParameter('http://bigmir.net/?a=b&c=d&e=f#g', 'a') as url_without_a,
    cutURLParameter('http://bigmir.net/?a=b&c=d&e=f#g', ['c', 'e']) as url_without_c_and_e;

Result: 

┌─url_without_a────────────────┬─url_without_c_and_e──────┐
│ http://bigmir.net/?c=d&e=f#g │ http://bigmir.net/?a=b#g │
└──────────────────────────────┴──────────────────────────┘
Retrieved from "https://docs.foji.io/index.php?title=Relational_Data_-_Functions_-_Url_Functions&oldid=2323"