Chances are that some­day the data you bring into Domo won’t be in the per­fect form. It might be denor­mal­ized or have the incor­rect lev­el of aggre­ga­tion, but when a busi­ness ques­tion requires this imper­fect data you just might need to rely on DataFlows.

As defined by Domo, a DataFlow is “a script that trans­forms input datasets and out­puts new datasets” So DataFlows are sim­ply a way to man­age your Extract Trans­form Load (ETL) jobs with­in your Domo instance. This enables the user to lever­age the pow­er of Domo’s cloud to do the trans­for­ma­tions with­out anoth­er sys­tem in place. For enter­prise orga­ni­za­tions with larg­er datasets the advan­tage of high­ly scal­able com­put­ing resources on demand to process data is very excit­ing. Very, very excit­ing.

There are two gener­ic types of DataFlows avail­able with­in Domo today: GUI- based and SQL- based.

The graph­i­cal user inter­face based ETL pro­cess­ing tool is called Mag­ic ETL and has a very low bar­ri­er to entry. As long as you under­stand basic data orga­ni­za­tion and can drag icons onto a field, you should be able to fig­ure out how to use Mag­ic. This tools comes with capa­bil­i­ties to join data, stack data, and so some col­umn- based math all through an easy-to-use GUI.

How­ev­er, what Mag­ic lacks is true SQL trans­la­tion capa­bil­i­ties. So if you are a SQL jock­ey look­ing to do a sub­s­e­lect, you’ll be more com­fort­able using MySQL or Red­Shift

The two SQL- based options are MySQL and Red­shift, with MySQL being the stan­dard SQL envi­ron­ment. Each pro­vide access to a SQL edi­tor where you have access to the func­tion­al­i­ty of either data­base. Hav­ing trou­ble decid­ing between MySQL or Red­shift? We sug­gest that MySQL be used when 1–3 mil­lion rows are being trans­formed and Red­shift when over 3 mil­lion rows are being trans­formed.

Both MySQL and Red­shift sup­port the sys­tem func­tions of their respec­tive data­bas­es, allow­ing you to do cool stuff like this:


SELECT usersales.Surname
, user
sales.Sales
, usersales.CAC
, @curRow := @curRow + 1 AS RowNumber
FROM user
sales
JOIN (
SELECT @curRow := 0
) r;

Sum­ma­ry

Mag­ic ETL and the SQL sys­tems allow trans­for­ma­tion of data inside the Domo cloud, lever­ag­ing their scal­able com­put­ing resources. While the SQL sys­tems offer more flex­i­bil­i­ty in terms of their func­tion­al­i­ty, Mag­ic ETL cer­tain­ly is more approach­able to a less tech­ni­cal audi­ence.

At Search Dis­cov­ery we live to work with clients to get effi­cient and main­tain­able process­es in place, if you’re get­ting start­ing with Domo you may con­tact us at contact@searchdiscovery.com.