Chances are that someday the data you bring into Domo won’t be in the perfect form. It might be denor­mal­ized or have the incor­rect level of aggre­ga­tion, but when a busi­ness ques­tion requires this imper­fect data you just might need to rely on DataFlows.

As defined by Domo, a DataFlow is “a script that trans­forms input datasets and outputs new datasets” So DataFlows are simply a way to manage your Extract Trans­form Load (ETL) jobs within your Domo instance. This enables the user to lever­age the power of Domo’s cloud to do the trans­for­ma­tions without another system in place. For enter­prise orga­ni­za­tions with larger datasets the advan­tage of highly scal­able comput­ing resources on demand to process data is very excit­ing. Very, very excit­ing.

There are two generic types of DataFlows avail­able within Domo today: GUI- based and SQL- based.

The graph­i­cal user inter­face based ETL process­ing tool is called Magic ETL and has a very low barrier to entry. As long as you under­stand basic data orga­ni­za­tion and can drag icons onto a field, you should be able to figure out how to use Magic. This tools comes with capa­bil­i­ties to join data, stack data, and so some column- based math all through an easy-to-use GUI.

However, what Magic lacks is true SQL trans­la­tion capa­bil­i­ties. So if you are a SQL jockey looking to do a subs­e­lect, you’ll be more comfort­able using MySQL or RedShift

The two SQL- based options are MySQL and Redshift, with MySQL being the stan­dard SQL envi­ron­ment. Each provide access to a SQL editor where you have access to the func­tion­al­ity of either data­base. Having trouble decid­ing between MySQL or Redshift? We suggest that MySQL be used when 1–3 million rows are being trans­formed and Redshift when over 3 million rows are being trans­formed.

Both MySQL and Redshift support the system func­tions of their respec­tive data­bases, allow­ing you to do cool stuff like this:


SELECT usersales.Surname
, user
sales.Sales
, usersales.CAC
, @curRow := @curRow + 1 AS RowNumber
FROM user
sales
JOIN (
SELECT @curRow := 0
) r;

Summary

Magic ETL and the SQL systems allow trans­for­ma­tion of data inside the Domo cloud, lever­ag­ing their scal­able comput­ing resources. While the SQL systems offer more flex­i­bil­ity in terms of their func­tion­al­ity, Magic ETL certainly is more approach­able to a less tech­ni­cal audi­ence.

At Search Discovery we live to work with clients to get effi­cient and main­tain­able processes in place, if you’re getting start­ing with Domo you may contact us at contact@searchdiscovery.com.