Business Intelligence / Data Visualization
State of the Art
There’s something my first boss told me that stuck with me through my career in finance. My first role at a Canadian financial institution was in a capital markets business that performed currency conversions on payments inbound to Canada. We had made deals with many of the largest retail banks in the U.S. and Europe so that any payment sent from their bank to a Canadian bank account would first get routed through us. We would perform a conversion into Canadian dollars, charge a small conversion fee, share the revenue with the originating bank, and then route the payment to its final destination. We were intermediating millions of dollars of foreign payments and siphoning all of the currency conversion revenue that would otherwise have gone to whichever Canadian bank was the payment’s final destination.
I once asked our Managing Director how it was that the other Canadian banks were allowing us to get away with this. Surely our competitors noticed that a disproportionate number of payments were coming via our bank? The revenues were significant, and risk-free! We weren’t even the largest of the Canadian banks; why couldn’t our competitors leverage their global relationships and strike similar deals?
His answer surprised me; it was our reporting. We were providing our clients dashboards and data visualizations that showed them exactly how the business was performing. They could see trends and project revenues. We were better automated and faster. Essentially, we had driven larger banks out of this business because we were the best at data visualization and business intelligence. As a business manager, it’s critical that you get this aspect of your business right. It can make or break a business.
Choosing the Right Tool or Service
There’s a tradeoff between how much flexibility and creative freedom your technology gives you, and the expertise it takes to use it. No-code tools like Excel and Tableau let you jump into making data visualizations quickly, but they constrain you to using their particular toolbox of templates, formulas and data structures, whereas coding something yourself allows for user experiences that truly stand out.
No-Code Tools
- Minimal technical expertise required
- Least flexibility with regard to data structure and visual design
- Ongoing software licensing costs
Python and R Packages
- Write code in a familiar language
- Generally visual design is constrained by templates
- Some packages can generate web applications
JavaScript Libraries
- JavaScript has state of the art data visualization and graphics libraries
- Frameworks such as React and Vue accelerate UI development and deployment
- Native web applications are ideal for sharing across all kinds of devices
Full Service Solutions
- Get expertise to solve data engineering challenges and satisfy unique design requirements
- Design from scratch to suit business needs
- Own your application code (no software licenses or hosting restrictions)
No-Code Tools
No-code tools like Excel help you make data visualizations and reports with very little expertise, but they are also very rigid about what templates are available and how you interact with them. They are designed for business analysts as opposed to developers. The purpose of these tools is to abstract away the implementation details so that you can point and click your way to creating a basic data visualization. These are good options when you want to do it yourself, but they have limitations as well.
Tableau is most effective when used by a subject matter expert to visually explore their data quickly. It’s convenient to connect to a database or CSV file, pick a chart template, and pivot dimensions as needed to explore relationships. However, it’s less effective as a reporting tool or for complex analytics. According to the Tableau Support Forums, “Tableau isn't meant to print 20 page reports with page numbers and headers like older BI products - just not our thing.”
After working with Tableau for some time, certain drawbacks become apparent: doing anything outside-the-box often demands unintuitive and “hacky” solutions. Depending on how your database is structured, certain analytics can be difficult, or impossible, to engineer in Tableau without restructuring the database itself. Feature engineering in Tableau often requires an awkward mix of database queries in a language such as SQL, and then using proprietary Tableau formulas that are similar to the Basic language Excel uses. This requires Tableau-centric expertise that can send you down a rabbit hole of reading whitepapers on level-of-detail calculations, which is a bit counter to the purpose of using a no-code tool to begin with. When choosing Tableau, you need to consider the cost of training and labor to develop your dashboards, along with the monthly cost of licensing the software itself.
Excel has been around since 1987 and is still widely used in the financial industry today. Combining data and charts together in spreadsheets can make it easy to work with small datasets, but there are drawbacks as well. Workbooks can become bloated and turn into haphazardly arranged databases themselves. Formulas and data get scattered about and make maintenance a real chore. The built-in chart templates are generic and have limited customizability. Automation is limited by the old fashioned VBA language, which few new developers want to learn, and which will probably see limited support in the future. If your organization is still using Excel to automate routine reporting tasks, you're past due for an update.
Power BI is a Microsoft product similar to Excel, but with additional options for connecting to databases and publishing dashboards. It shares many of the same advantages and disadvantages of Excel and Tableau. The user interface will be familiar to those who work with other MS Office products, but you’re stuck with built-in templates which make it difficult to make something that really grabs attention. It’s also costly to scale up for enterprises, demands some expertise to build the dashboards, and requires user licenses.
Web-based tools such as Datawrapper and Flourish offer an API for generating and hosting charts based on their provided templates. These options tend to be cheaper than premium software like Tableau, but are particular about how you structure your data and interact with their API. They can be a good fit for journalists and NGO’s with some IT and data visualization expertise but smaller budgets.
Python and R Packages
These are packages for people who are comfortable coding in a particular environment (namely Python or R), but lack the expertise (or desire) to build a full-stack web application. Some frameworks try to bridge that gap by letting you build with Python, and translate that code into a JavaScript application for display on the web.
One of the big advantages of using a programming language over no-code tools is that you get the convenience of Python or R for data engineering and pre-processing data. This may fit better with your existing technology stack without the added complication of installing plugins for Tableau or Excel. On the other hand, these tools also impose certain constraints. You generally need to work within certain chart templates, and can’t do anything too far outside the box. These packages are ideal for programmers in the scientific and academic domains to share data and research.
Examples: Plotly (Dash), Seaborn, Bokeh, Rayshader
JavaScript Libraries
Building business intelligence and visualization tools that really stand out requires coding and data visualization expertise. Custom applications built for the web give complete control over the presentation of your data and how the user interacts with it. That includes unique chart types, custom animation, and interactivity that’s perfectly suited to your use case. This requires more expertise than with other options, but all the development tools are free and open source, and you don’t get stuck licensing any particular software. You have ownership of your application. Here are some the most important JavaScript libraries for integrating data visualizations in state of the art web applications today.
D3 has been the state of the art in data visualization on the web for 10 years. It is an open source JavaScript library that gives you complete control over rendering visual elements in a web browser down to the smallest detail. Many low-code tools and frameworks are actually built on top of D3 because it’s the most flexible toolbox for data visualization on the web. For developers serious about data visualization, learning the ins and outs of this library is a must.
It’s worth noting that as web technology has evolved, the way we use D3 has changed as well. With front-end frameworks like React and Vue designed to manage and control a webpage’s document-object-model (DOM) it’s sometimes better to relegate rendering logic to your front-end framework, while using D3’s toolbox as a helper for working with data, scales, and force-based layouts. Part of the strength of D3 is that it’s an open toolbox that let’s you pick and choose exactly which functions you want to use. There’s almost always something that D3 can help you with when visualizing data, even if you’re using a larger framework in tandem.
This is one of the most exciting libraries in web development today. Working with 3D graphics has always been a complex and mathematically heavy endeavor. ThreeJS has taken major strides in abstracting away much of that complexity, bringing the kind of 3D graphics to the web that we used to only see in video games. The learning curve is long and steep, but the graphics and performance it provides are unparalleled.
ThreeJS uses WebGL technology, which leverages a device’s graphics processor, allowing for rendering in 3D, or animating large numbers of 2D elements that would be too much for a browser to handle using more conventional graphic types such as SVG. Although this isn’t necessary in all applications, it opens up amazing possibilities for data visualization. With a graphics processor (GPU) in almost every modern device (including smartphones), to ignore ThreeJS would be like having a Ferrari in the garage and never taking it for a spin. This is the future of the web.
React Three Fiber is a library for bringing ThreeJS into React applications. It allows you to create 3D components declaratively, making it easy to interact with 3D objects in much the same way you would with 2D components in a standard webpage. This opens up all sorts of possibilities for interacting with complex multidimensional data while maintaining a smooth and familiar user interface. See an example.
Nivo is a library that packages D3 visualizations in convenient React components. It sacrifices some of the control you get from building from scratch in D3 alone, but it can get you up and running with aesthetically pleasing chart designs very quickly. It’s also flexible in that you can layer your own customizations on top of an established template, although there is a bit of a learning curve to this - see a tutorial to get started. It’s a good choice for React developers who want to make dashboards, but aren’t so familiar with the finer details of D3.
Full Service Solutions
Chartfleau exists to provide businesses with the expertise they need to build state of the art data-driven applications on the web. We save businesses the cost of licensing and development associated with commercial business intelligence software, and code custom web applications with open source libraries. Our clients keep ownership of all the code we develop, and are free to modify, white-label, or commercialize it however they like. Learn more about our services.
Social Media
