View source for Relational Data - Aggregate Functions - Combinators

The name of an aggregate function can have a suffix appended to it. This changes the way the aggregate function works.

<span id="if"></span>
== -If ==

The suffix -If can be appended to the name of any aggregate function. In this case, the aggregate function accepts an extra argument – a condition (Uint8 type). The aggregate function processes only the rows that trigger the condition. If the condition was not triggered even once, it returns a default value (usually zeros or empty strings).

Examples: <code>sumIf(column, cond)</code>, <code>countIf(cond)</code>, <code>avgIf(x, cond)</code>, <code>quantilesTimingIf(level1, level2)(x, cond)</code>, <code>argMinIf(arg, val, cond)</code> and so on.

With conditional aggregate functions, you can calculate aggregates for several conditions at once, without using subqueries and <code>JOIN</code>s. For example, conditional aggregate functions can be used to implement the segment comparison functionality.

<span id="array"></span>
== -Array ==

The -Array suffix can be appended to any aggregate function. In this case, the aggregate function takes arguments of the ‘Array(T)’ type (arrays) instead of ‘T’ type arguments. If the aggregate function accepts multiple arguments, this must be arrays of equal lengths. When processing arrays, the aggregate function works like the original aggregate function across all array elements.

Example 1: <code>sumArray(arr)</code> - Totals all the elements of all ‘arr’ arrays. In this example, it could have been written more simply: <code>sum(arraySum(arr))</code>.

Example 2: <code>uniqArray(arr)</code> – Counts the number of unique elements in all ‘arr’ arrays. This could be done an easier way: <code>uniq(arrayJoin(arr))</code>, but it’s not always possible to add ‘arrayJoin’ to a query.

-If and -Array can be combined. However, ‘Array’ must come first, then ‘If’. Examples: <code>uniqArrayIf(arr, cond)</code>, <code>quantilesTimingArrayIf(level1, level2)(arr, cond)</code>. Due to this order, the ‘cond’ argument won’t be an array.

<span id="map"></span>
== -Map ==

The -Map suffix can be appended to any aggregate function. This will create an aggregate function which gets Map type as an argument, and aggregates values of each key of the map separately using the specified aggregate function. The result is also of a Map type.

'''Example'''

<syntaxhighlight lang="sql">CREATE TABLE map_map(
    date Date,
    timeslot DateTime,
    status Map(String, UInt64)
) ENGINE = Log;

INSERT INTO map_map VALUES
    ('2000-01-01', '2000-01-01 00:00:00', (['a', 'b', 'c'], [10, 10, 10])),
    ('2000-01-01', '2000-01-01 00:00:00', (['c', 'd', 'e'], [10, 10, 10])),
    ('2000-01-01', '2000-01-01 00:01:00', (['d', 'e', 'f'], [10, 10, 10])),
    ('2000-01-01', '2000-01-01 00:01:00', (['f', 'g', 'g'], [10, 10, 10]));

SELECT
    timeslot,
    sumMap(status),
    avgMap(status),
    minMap(status)
FROM map_map
GROUP BY timeslot;

┌────────────timeslot─┬─sumMap(status)───────────────────────┬─avgMap(status)───────────────────────┬─minMap(status)───────────────────────┐
│ 2000-01-01 00:00:00 │ {'a':10,'b':10,'c':20,'d':10,'e':10} │ {'a':10,'b':10,'c':10,'d':10,'e':10} │ {'a':10,'b':10,'c':10,'d':10,'e':10} │
│ 2000-01-01 00:01:00 │ {'d':10,'e':10,'f':20,'g':20}        │ {'d':10,'e':10,'f':10,'g':10}        │ {'d':10,'e':10,'f':10,'g':10}        │
└─────────────────────┴──────────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────┘</syntaxhighlight>
<span id="simplestate"></span>
== -SimpleState ==

If you apply this combinator, the aggregate function returns the same value but with a different type. This is a [[Relational_Data_-_Data_Types_-_Simpleaggregatefunction|SimpleAggregateFunction(…)]] that can be stored in a table to work with [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/aggregatingmergetree AggregatingMergeTree] tables.

'''Syntax'''

<syntaxhighlight lang="sql"><aggFunction>SimpleState(x)</syntaxhighlight>
'''Arguments'''

* <code>x</code> — Aggregate function parameters.

'''Returned values'''

The value of an aggregate function with the <code>SimpleAggregateFunction(...)</code> type.

'''Example'''

Query:

<syntaxhighlight lang="sql">WITH anySimpleState(number) AS c SELECT toTypeName(c), c FROM numbers(1);</syntaxhighlight>
Result:

<syntaxhighlight lang="text">┌─toTypeName(c)────────────────────────┬─c─┐
│ SimpleAggregateFunction(any, UInt64) │ 0 │
└──────────────────────────────────────┴───┘</syntaxhighlight>
<span id="state"></span>
== -State ==

If you apply this combinator, the aggregate function does not return the resulting value (such as the number of unique values for the [[Relational_Data_-_Aggregate_Functions_-_Reference_-_Uniq#agg_function-uniq|uniq]] function), but an intermediate state of the aggregation (for <code>uniq</code>, this is the hash table for calculating the number of unique values). This is an <code>AggregateFunction(...)</code> that can be used for further processing or stored in a table to finish aggregating later.

<div class="note">

Please notice, that -MapState is not an invariant for the same data due to the fact that order of data in intermediate state can change, though it doesn’t impact ingestion of this data.


</div>
To work with these states, use:

* [https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/aggregatingmergetree AggregatingMergeTree] table engine.
* [[Relational_Data_-_Functions_-_Other_Functions#function-finalizeaggregation|finalizeAggregation]] function.
* [[Relational_Data_-_Functions_-_Other_Functions#runningaccumulate|runningAccumulate]] function.
* [[#-merge|-Merge]] combinator.
* [[#-mergestate|-MergeState]] combinator.

<span id="merge"></span>
== -Merge ==

If you apply this combinator, the aggregate function takes the intermediate aggregation state as an argument, combines the states to finish aggregation, and returns the resulting value.

<span id="mergestate"></span>
== -MergeState ==

Merges the intermediate aggregation states in the same way as the -Merge combinator. However, it does not return the resulting value, but an intermediate aggregation state, similar to the -State combinator.

<span id="foreach"></span>
== -ForEach ==

Converts an aggregate function for tables into an aggregate function for arrays that aggregates the corresponding array items and returns an array of results. For example, <code>sumForEach</code> for the arrays <code>[1, 2]</code>, <code>[3, 4, 5]</code>and<code>[6, 7]</code>returns the result <code>[10, 13, 5]</code> after adding together the corresponding array items.

<span id="distinct"></span>
== -Distinct ==

Every unique combination of arguments will be aggregated only once. Repeating values are ignored. Examples: <code>sum(DISTINCT x)</code>, <code>groupArray(DISTINCT x)</code>, <code>corrStableDistinct(DISTINCT x, y)</code> and so on.

<span id="ordefault"></span>
== -OrDefault ==

Changes behavior of an aggregate function.

If an aggregate function does not have input values, with this combinator it returns the default value for its return data type. Applies to the aggregate functions that can take empty input data.

<code>-OrDefault</code> can be used with other combinators.

'''Syntax'''

<syntaxhighlight lang="sql"><aggFunction>OrDefault(x)</syntaxhighlight>
'''Arguments'''

* <code>x</code> — Aggregate function parameters.

'''Returned values'''

Returns the default value of an aggregate function’s return type if there is nothing to aggregate.

Type depends on the aggregate function used.

'''Example'''

Query:

<syntaxhighlight lang="sql">SELECT avg(number), avgOrDefault(number) FROM numbers(0)</syntaxhighlight>
Result:

<syntaxhighlight lang="text">┌─avg(number)─┬─avgOrDefault(number)─┐
│         nan │                    0 │
└─────────────┴──────────────────────┘</syntaxhighlight>
Also <code>-OrDefault</code> can be used with another combinators. It is useful when the aggregate function does not accept the empty input.

Query:

<syntaxhighlight lang="sql">SELECT avgOrDefaultIf(x, x > 10)
FROM
(
    SELECT toDecimal32(1.23, 2) AS x
)</syntaxhighlight>
Result:

<syntaxhighlight lang="text">┌─avgOrDefaultIf(x, greater(x, 10))─┐
│                              0.00 │
└───────────────────────────────────┘</syntaxhighlight>
<span id="ornull"></span>
== -OrNull ==

Changes behavior of an aggregate function.

This combinator converts a result of an aggregate function to the [[Relational_Data_-_Data_Types_-_Nullable|Nullable]] data type. If the aggregate function does not have values to calculate it returns [[Relational_Data_-_Syntax#null-literal|NULL]].

<code>-OrNull</code> can be used with other combinators.

'''Syntax'''

<syntaxhighlight lang="sql"><aggFunction>OrNull(x)</syntaxhighlight>
'''Arguments'''

* <code>x</code> — Aggregate function parameters.

'''Returned values'''

* The result of the aggregate function, converted to the <code>Nullable</code> data type.
* <code>NULL</code>, if there is nothing to aggregate.

Type: <code>Nullable(aggregate function return type)</code>.

'''Example'''

Add <code>-orNull</code> to the end of aggregate function.

Query:

<syntaxhighlight lang="sql">SELECT sumOrNull(number), toTypeName(sumOrNull(number)) FROM numbers(10) WHERE number > 10</syntaxhighlight>
Result:

<syntaxhighlight lang="text">┌─sumOrNull(number)─┬─toTypeName(sumOrNull(number))─┐
│              ᴺᵁᴸᴸ │ Nullable(UInt64)              │
└───────────────────┴───────────────────────────────┘</syntaxhighlight>
Also <code>-OrNull</code> can be used with another combinators. It is useful when the aggregate function does not accept the empty input.

Query:

<syntaxhighlight lang="sql">SELECT avgOrNullIf(x, x > 10)
FROM
(
    SELECT toDecimal32(1.23, 2) AS x
)</syntaxhighlight>
Result:

<syntaxhighlight lang="text">┌─avgOrNullIf(x, greater(x, 10))─┐
│                           ᴺᵁᴸᴸ │
└────────────────────────────────┘</syntaxhighlight>
<span id="resample"></span>
== -Resample ==

Lets you divide data into groups, and then separately aggregates the data in those groups. Groups are created by splitting the values from one column into intervals.

<syntaxhighlight lang="sql"><aggFunction>Resample(start, end, step)(<aggFunction_params>, resampling_key)</syntaxhighlight>
'''Arguments'''

* <code>start</code> — Starting value of the whole required interval for <code>resampling_key</code> values.
* <code>stop</code> — Ending value of the whole required interval for <code>resampling_key</code> values. The whole interval does not include the <code>stop</code> value <code>[start, stop)</code>.
* <code>step</code> — Step for separating the whole interval into subintervals. The <code>aggFunction</code> is executed over each of those subintervals independently.
* <code>resampling_key</code> — Column whose values are used for separating data into intervals.
* <code>aggFunction_params</code> — <code>aggFunction</code> parameters.

'''Returned values'''

* Array of <code>aggFunction</code> results for each subinterval.

'''Example'''

Consider the <code>people</code> table with the following data:

<syntaxhighlight lang="text">┌─name───┬─age─┬─wage─┐
│ John   │  16 │   10 │
│ Alice  │  30 │   15 │
│ Mary   │  35 │    8 │
│ Evelyn │  48 │ 11.5 │
│ David  │  62 │  9.9 │
│ Brian  │  60 │   16 │
└────────┴─────┴──────┘</syntaxhighlight>
Let’s get the names of the people whose age lies in the intervals of <code>[30,60)</code> and <code>[60,75)</code>. Since we use integer representation for age, we get ages in the <code>[30, 59]</code> and <code>[60,74]</code> intervals.

To aggregate names in an array, we use the [[Relational_Data_-_Aggregate_Functions_-_Reference_-_Grouparray#agg_function-grouparray|groupArray]] aggregate function. It takes one argument. In our case, it’s the <code>name</code> column. The <code>groupArrayResample</code> function should use the <code>age</code> column to aggregate names by age. To define the required intervals, we pass the <code>30, 75, 30</code> arguments into the <code>groupArrayResample</code> function.

<syntaxhighlight lang="sql">SELECT groupArrayResample(30, 75, 30)(name, age) FROM people</syntaxhighlight>
<syntaxhighlight lang="text">┌─groupArrayResample(30, 75, 30)(name, age)─────┐
│ [['Alice','Mary','Evelyn'],['David','Brian']] │
└───────────────────────────────────────────────┘</syntaxhighlight>
Consider the results.

<code>John</code> is out of the sample because he’s too young. Other people are distributed according to the specified age intervals.

Now let’s count the total number of people and their average wage in the specified age intervals.

<syntaxhighlight lang="sql">SELECT
    countResample(30, 75, 30)(name, age) AS amount,
    avgResample(30, 75, 30)(wage, age) AS avg_wage
FROM people</syntaxhighlight>
<syntaxhighlight lang="text">┌─amount─┬─avg_wage──────────────────┐
│ [3,2]  │ [11.5,12.949999809265137] │
└────────┴───────────────────────────┘</syntaxhighlight>
<span id="argmin"></span>
== -ArgMin ==

The suffix -ArgMin can be appended to the name of any aggregate function. In this case, the aggregate function accepts an additional argument, which should be any comparable expression. The aggregate function processes only the rows that have the minimum value for the specified extra expression.

Examples: <code>sumArgMin(column, expr)</code>, <code>countArgMin(expr)</code>, <code>avgArgMin(x, expr)</code> and so on.

<span id="argmax"></span>
== -ArgMax ==

Similar to suffix -ArgMin but processes only the rows that have the maximum value for the specified extra expression.

<span id="related-content"></span>
== Related Content ==

* Blog: [https://clickhouse.com/blog/aggregate-functions-combinators-in-clickhouse-for-arrays-maps-and-states Using Aggregate Combinators in ClickHouse]
[[Category:Relational_Data]]