A performance investigation into JavaScript visualization libraries with the focus on render time and memory usage PDF Free Download

Name: A performance investigation into JavaScript visualization libraries with the focus on render time and memory usage PDF
Author: im_alyssa

1 / 87

0 views•87 pages

A performance investigation into JavaScript visualization libraries with the focus on render time and memory usage PDF Free Download

A performance investigation into JavaScript visualization libraries with the focus on render time and memory usage PDF free Download. Think more deeply and widely.

Bachelor Degree Project

A performance investigation into

JavaScript visualization libraries

with the focus on render time

and memory usage

A performance measurement of diﬀerent libraries and

statistical charts

Bachelor Degree Project in Information Technology

Basic level 30 ECTS

Spring 2022

Fredrik Boström, Alexander Dahlberg, William

Linderoth

Supervisor: Maurice Lamb

Examiner: Joe Steinhauer

Abstract

Visualizing data is important to make it easier to understand. Due to the common accessibility

and the popularity of the web, a growth within web-based visualization is seen. One common and

easy way to do this is by using an already implemented JavaScript library. When developing such

a website some important properties to keep in mind are the render time and the memory usage.

This study measured both the render time with diﬀerent sizes of the dataset, and the render

time when rendering diﬀerent number of charts with the same dataset size. The memory usage

was also measured with diﬀerent sizes of the dataset. Six libraries were chosen to be included in

this study: D3, Echarts, CanvasJS, Chartist, Highcharts, and Plotly. An experiment has been

performed to test the libraries render times and memory usage. The results of the experiment

show that D3 has the overall lowest render times whilst CanvasJS had the lowest memory usage.

keywords: Visualization, JavaScript, Memory usage, Render time

Contents

1 Introduction 1

2 Background 2

2.1 Visualization ...................................... 2

2.2 Dataset ......................................... 2

2.3 Charts.......................................... 3

2.4 HTML,CSSandJavaScript.............................. 3

2.5 JavaScript Libraries Included in the Study . . . . . . . . . . . . . . . . . . . . . . 4

2.5.1 D3........................................ 4

2.5.2 ECharts..................................... 4

2.5.3 CanvasJS .................................... 4

2.5.4 Chartist..................................... 4

2.5.5 Highcharts ................................... 4

2.5.6 Plotly ...................................... 4

3 Problem 5

3.1 ProblemDeﬁnition................................... 5

3.2 ResearchQuestions................................... 6

3.3 Hypotheses ....................................... 6

3.4 Objectives........................................ 7

4 Related Work 8

5 Method and Approach 10

5.1 Method ......................................... 10

5.1.1 FindingLibraries................................ 10

5.1.2 Setting up library criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5.1.3 Experiment................................... 12

5.1.4 AlternativeMethods.............................. 12

5.2 Approach ........................................ 13

6 Results 15

6.1 Scalability on increasing data points . . . . . . . . . . . . . . . . . . . . . . . . . 15

6.1.1 LineChart ................................... 15

6.1.2 AreaChart ................................... 16

6.1.3 ScatterPlot................................... 17

6.2 Scalability of multiple charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

6.2.1 LineChart ................................... 18

6.2.2 AreaChart ................................... 19

6.2.3 ScatterPlot................................... 20

6.3 MemoryUsage ..................................... 21

6.3.1 LineChart ................................... 21

6.3.2 AreaChart ................................... 22

6.3.3 ScatterPlot................................... 23

7 Analysis 24

7.1 Scalability of increasing data points . . . . . . . . . . . . . . . . . . . . . . . . . 24

7.1.1 LineChart ................................... 24

7.1.2 AreaChart ................................... 26

7.1.3 ScatterPlot................................... 28

7.2 Scalability of multiple charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

7.2.1 LineChart ................................... 29

7.2.2 AreaChart ................................... 31

7.2.3 ScatterPlot................................... 33

7.3 Scalability of memory usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

7.3.1 LineChart ................................... 36

7.3.2 AreaChart ................................... 37

7.3.3 ScatterPlot................................... 39

7.4 Conclusions....................................... 41

8 Discussion 42

8.1 Ethics .......................................... 42

8.2 ThreatstoValidity................................... 42

8.2.1 InternalValidity ................................ 42

8.2.2 ExternalValidity................................ 43

8.3 ValuetoSociety .................................... 43

8.4 FutureWork ...................................... 44

8.5 Conclusions....................................... 44

A Appendix - Scalability on datapoints for Line chart I

B Appendix - Scalability on datapoints for Area chart V

C Appendix - Scalability on datapoints for Scatter plot IX

D Appendix - Scalability on Number of Charts for Line chart XIII

E Appendix - Scalability on Number of Charts for Area chart XVII

F Appendix - Scalability on Number of Charts for Scatter plot XXI

G Appendix - Scalability on Memory usage for Line chart XXV

H Appendix - Scalability on Memory usage for Area chart XXIX

I Appendix - Scalability on Memory usage for Scatter plot XXXIII

1|Introduction

Visualization is a powerful tool which can help with understanding and making sense of a large

amount of data. Visualizing data on the web in the form of charts is something that is often

done today.

When designing a new website to display charts, an already existing JavaScript library to render

these chart can be used to make the process easier for the developers. When choosing such a

library, there are some factors that are important to consider, e.g., render time and memory

usage. When a user waits for a website to load, Nah (2004) shows that a tolerated waiting time

where users start to leave the web page is after 2 seconds.

This study benchmarks the selected libraries on their render time, both when rendering a single

chart with diﬀerent number of data points, as well as several charts where the number of data

points stays consistent, and memory usage when rendering a single chart with diﬀerent number

of data points. Since the dataset chosen for this study is a time-series dataset, the chart types

that are evaluated are: Line chart, Area chart and Scatter plot.

To answer the research questions in this study, a controlled experiment was performed where

the measurements were taken, changes were made to the dataset and more measurements were

taken (Wohlin et al. 2012). This method gives better control over the independent variables and

the results produced are therefore more accurate.

The experiment is split in to three parts, where each of the measurement of render time and

memory usage based on number of data points have their own set of datasets, and the measure-

ment of render time based on number of charts rendered has its own set of datasets. The websites

are created one by one and the the measurements are taken. The results from each experiment

is analyzed to be able to answer all of the research questions in this study.

2|Background

2.1 Visualization

There are several diﬀerent ways to present data, e.g., by showing raw data as numbers; talk-

ing and giving the information via voice; and visualizing data in a picture, chart, or something

similar. People might have diﬀerent opinions on which way of presenting data is the easiest to

understand, but according to Aparicio & Costa (2015), the human mind is very visual, and that

data visualization is an ancient need. Therefore the easiest way for most people to take in new

information is by visualization, which helps both learning and remembering what one has learned.

Visualization of data is actively used on websites to represent data by using, for example, a chart

or a graph. Visualizing data changes the way people experience the information (Aparicio &

Costa 2015). Transforming raw data to something more readable, like a chart, can make it easier

to comprehend. Visualization techniques can let the reader recognize patterns that are otherwise

hard to see when only looking at, e.g., numbers.

2.2 Dataset

To test the libraries charting capabilities, a dataset with suitable data is needed. This dataset

needs to be large enough for this research, and also somewhat representative of real life data

as to make the results more generalizable. The dataset chosen for this study is made up of

conﬁrmed Covid-19 infection cases in Sweden, taken from Folkhälsomyndigheten (2022). The

dataset covers each reported case in the time span between the 4th of February 2020 and the

26th of January 2022. Each row in the dataset represents one date, and includes that date’s

conﬁrmed cases per region, see Table 2.1.

Statistikdatum Blekinge Dalarna Gotland Gävleborg Halland

9/14/2021 13 70 0 36 37

9/15/2021 4 22 4 13 28

Table 2.1: A small excerpt from the dataset that shows how the dataset is built up, each

column that contains a number is a datapoint

2.3 Charts

Since the chosen dataset is a time-series dataset, the chart type that suits the dataset the most

is a line chart (Statistiska Centralbyrån 2022). The line chart excels at displaying series of data

by connecting datapoints with a continuous line. Therefore, the line chart is good at visualizing

how a variable changes over time. Two other chart types representing a time series well are

area charts and scatter plots. An area chart is a line chart where the area beneath the line is

ﬁlled, this chart type is mostly used when a part-to-whole analysis is needed. This means that

it is easier to see the diﬀerence between two series in an area chart compared to a line chart.

A scatter plot marks out every data point with a point in the chart, but it does not connect

them with a line as the line chart does. The scatter plot is therefore better to use for showing

correlation between two variables, since a . These two chart types are also included in the study.

A comparison of these diﬀerent chart types can be seen in Figure 2.1

Figure 2.1: From left to right: Line chart, Area chart, Scatter plot.

2.4 HTML, CSS and JavaScript

Every website on the internet has one thing in common, a hypertext ﬁle. By creating multiple

ﬁles using HyperText Markup Language (HTML) and linking them together under the same

domain name, you can create a website. HTML is essential and creates the foundation of what

will get displayed on a web page. It contains the text that the website displays, the containers

in which it structures and holds information, and many other so-called DOM (Document Object

Model) elements that make up the content for each web page (MDN HTML 2022).

The HTML ﬁle allows the developer to enrich a web page using other languages such as Cas-

cading Style Sheets (CSS) and JavaScript. The CSS language’s main objective is to style and

change the appearance of the DOM elements created by the HTML document. CSS has the

power to alter colors, reposition elements, change fonts, and many other attributes linked to

each element’s appearance (MDN CSS 2022).

The ﬁnal part, which brings a website from static and non-interactable to dynamic, is by writing

JavaScript. The JavaScript language has much potential and should be seen as a tool that

allows controlling the behavior of the web page, such as - what happens when a button is clicked,

fetching remote data to render dynamic content, or in this case, taking data and rendering a

chart. What makes displaying charts possible is Application Programmable Interfaces (API)

built on top of the JavaScript language. Speciﬁcally, the Canvas API and the SVG API provide

the developer with a whole new toolset to implement 2D graphics onto the web page (MDN

Javascript 2022).

2.5 JavaScript Libraries Included in the Study

2.5.1 D3

D3.js is an open-source JavaScript library licensed under the BSD-3-Clause (2022). The D3

library helps the developer draw interactive charts to visualize the data using SVG, HTML, and

CSS. According to their website (D3.js 2022), the D3 library is extremely fast and supports large

datasets.

2.5.2 ECharts

Echarts.js is an open-source JavaScript library licensed under Apache License 2.0 (GitHub

ECharts 2022). According to their website, ECharts (2022) provides more than 20 chart types

and a dozen components, each of which can be arbitrarily combined. They also mention that

ECharts has progressive rendering and stream loading, making it possible to render 10 million

data in real-time.

2.5.3 CanvasJS

CanvasJS (2022) is an HTML Canvas Charting library developed by Fenopix, which specializes

in the development of data visualization. The library has rich integration with both front-end

and back-end, such as jQuery, React, Angular, and many more. CanvasJS falls under a royalty-

free license based on the number of employees. This means that it is possible to pay a one-time

fee to use their library. However, they do provide the possibility of a fully-featured trial version

that lasts up to 30 days.

2.5.4 Chartist

Chartist (2022) is a lightweight data visualization library that displays the charts in an inline-

SVG format instead of HTML Canvas. The library is open-source and encourages individuals to

contribute.

2.5.5 Highcharts

Highcharts (2022) is another SVG data visualization library written purely in native JavaScript.

The library has an array of possible data formats, including JSON and CSV ﬁles. It also supports

integrating native operating systems such as iOS and Android and popular JavaScript frameworks

like React, Angular, and Vue. Highcharts is a commercial library that allows developers to freely

use their code as long as development is in action. As soon as the developed software is ready

for commercial use, it is required through their license that payment for the product has to be

done.

2.5.6 Plotly

Plotly is a free and open-source data visualization library released under the MIT License, which

renders its charts using SVG. Plotly (2022) contains over 40 diﬀerent chart types and includes

the possibility of rendering 3D graphs. Compared to other data visualizations libraries, Plotly

only supports the integration of the React JavaScript framework.

3|Problem

3.1 Problem Deﬁnition

Earlier digital visualizations would often be written as native software. However, this contributes

to a limited accessibility in general, since you would be required to download and run software

locally on your computer. Instead we see a growth within web-based data visualization due to

the popularity as well as the common accessibility of the web (Lee et al. 2014). One common

and easy way to do web-based visualization is by using JavaScript.

Websites that want to display diﬀerent charts may want to use an already existing JavaScript

library to make the process easier. Some essential properties of these libraries that a developer

has in mind when choosing a library are the render time, and the memory usage used when

rendering charts. The research done by Nah (2004) shows that a tolerated waiting time for any

website seems to start at 2 seconds, whereas users begin to leave the web page after that amount

of time.

Visualizing big data in a meaningful way can be diﬃcult, and it can be a tough challenge to keep

it readable for the user (Ali et al. 2016). Ali et al. (2016) mentions two major problems of big

data visualization as visual noise and information loss.

When a webpage with embedded JavaScript code is loaded, the code is downloaded and run

locally on the client. Therefore, any memory used by the webpage needs to be allocated by the

client. If a webpage requires a signiﬁcant portion of the client’s working memory, the client may

experience long loading times, and the webpage can seem unresponsive to the user.

This study aims to determine if the size of the dataset aﬀects the render time and memory usage,

both within the library itself and between all the selected JavaScript charting libraries. It also

aims to determine if the number of charts rendered, when the total number of data points stays

consistent, aﬀects the render time, both within the library itself and between all the selected

JavaScript charting libraries.

3.2 Research Questions

RQ1.1 Is there a signiﬁcant diﬀerence in render time when increasing the number of data points?

RQ1.2 Is there a signiﬁcant diﬀerence in render time between the chosen libraries?

RQ2.1 Is there a signiﬁcant diﬀerence in render time when rendering a single chart compared to

rendering several charts when rendered data points stay consistent?

RQ2.2 Is there a signiﬁcant diﬀerence in render time between the selected JavaScript libraries

when rendering a single chart compared to rendering several charts when rendered data

points stay consistent?

RQ3.1 Is there a signiﬁcant diﬀerence in allocated working memory depending on the number of

data points?

RQ3.2 Is there a signiﬁcant diﬀerence in allocated working memory depending on the implemented

library?

3.3 Hypotheses

When looking at previous research done in this area, both Carlström (2018) and Persson (2021)

found signiﬁcant diﬀerences in render time between diﬀerent libraries. They also found a signif-

icant diﬀerence in render time when increasing the number of data points. Another study (Lee

et al. 2014) measured both the render time and memory usage is of the libraries. The results

from this study shows that there is a diﬀerence in render time and memory usage depending on

what library is used. Bonald et al. (2020) have also done a study where memory usage have been

measured when rendering graphs. This study also shows that there is a diﬀerence in memory

usage depending on what algorithm is used to render a graph. Even though this study is done

with a diﬀerent programming language (Python) this at least shows that there is a diﬀerence in

memory usage. The hypotheses for this study is therefore concluded as following:

H1.1 There will be a signiﬁcant diﬀerence in render time as the data points rendered increase.

H1.2 There will be a signiﬁcant diﬀerence in render time between the selected libraries.

H2.1 There will be a signiﬁcant diﬀerence in render time when the number of charts rendered

increases while maintaining a ﬁxed number of data points.

H2.2 There will be a signiﬁcant diﬀerence in render time between the selected libraries when the

number of charts rendered increases while maintaining a ﬁxed number of data points.

H3.1 There will be a signiﬁcant diﬀerence in memory usage when increasing the number of data

points in the chart.

H3.2 There will be a signiﬁcant diﬀerence in memory usage between the selected libraries.

3.4 Objectives

To be able to complete this study, the following objectives have to be completed:

1. Select a dataset to be used in the study

2. Collect popular JavaScript visualization libraries that can render charts to include in the

study

3. Set up criteria for the libraries

4. Select six suitable libraries for the study based on the criteria.

5. Implement the selected libraries in all of the chart types

6. Design how the experiment should be performed

7. Perform the benchmark of render time and memory usage on all the libraries and their

chart types

8. Gather all of the results and present the data

9. Analyze the results and data from the previous step

4|Related Work

Performance testing on visualization tools has been performed many times before. This is true

for JavaScript visualization libraries as well. Below are some previously done researches in this

area.

Carlström (2018) performed an experiment in which three JavaScript libraries were tested and

compared on their render time. The libraries being: Chart.js, Google Charts, and Plotly.js.

The author measured the render time of a few chart types using diﬀering dataset sizes for each

library. The work set out to compare if the diﬀerent libraries performed signiﬁcantly diﬀerently

and if the size of the dataset mattered for render time. Both of which turned out to be true.

The results shows that the Chart.js library had the overall lowest render time, Plotly.js had the

second fastest render time, and Google Charts had the overall slowest render time.

Persson (2021) performed a similar experiment in which the author compared four JavaScript

libraries on their render time. The libraries being: ApexCharts, Frappe charts, Google Charts,

and TeeCharts JS. The authors’ work here emphasizes the scalability of the libraries in terms of

render time, which is what one part of this study will look at. The author concludes that both

the size of the dataset and the type of chart aﬀect the render time. The ﬁnal results showed that

TeeChartJS performed best followed by Frappecharts in second place. Google charts were able

to handle four out of ﬁve dataset sizes while Apex charts only managed to render two out of ﬁve.

Lee et al. (2014) wanted to ﬁnd the best visualization method for a large volume of data. They

tested diﬀerent visualization tools, Google Charts, Flex, OFC, D3, and JfreeChart. They mea-

sured processing time as well as memory performance. Their research found that each visual-

ization tool has strength in some areas, e.g., Google Charts was best for general users with a

user-friendly interface with somewhat limited functionality. D3 is suitable for scientiﬁc visualiza-

tion because of its low processing delay. For render time speciﬁcally, the D3 library performed

the best in their experiment, while the Flex library performed best in terms of memory usage.

Shahzad et al. (2016) presents and compares four diﬀerent JavaScript libraries, i.e., Highcharts,

GoJS, D3, and JSmol. They also show their custom visualization solutions where they have

utilized the technologies and libraries they presented, Visual-Netsim and Visual-Flow.

Bonald et al. (2020) compares the graph analysis algorithm scikit-network in Python with other

similar algorithms, i.e., NetworkX, iGraph, and graph-tool. They compare them based on exe-

cution time as well as memory usage, and their results show that the scikit-network algorithm

is highly competitive compared to the other algorithms.

Netek et al. (2019) performed a comparison study on ﬁve JavaScript libraries, Leaﬂet.marketcluster,

OpenLayers, Supercluster, MapBox GL JS, and PruneCluster. The aim of their study were to

test and compare the libraries capabilities in terms of rendering big data. Nine datasets contain-

ing between 10,000 and 3,000,000 datapoints were used to test the loading times of the libraries

clustering visualization methods. The included libraries showed vastly diﬀerent results, with

some libraries not being able to render all datasets. The render time was signiﬁcantly diﬀerent

between the diﬀerent dataset sizes. It was concluded that PruneCluster had the fastest render

time of all the libraries.

5|Method and Approach

5.1 Method

5.1.1 Finding Libraries

An informal search on the web was performed to gather the initial set of libraries to choose

from. A search for the "most common Javascript data visualization libraries" and "most popular

Javascript data visualization libraries" yielded a couple of websites (Monterail 2021, Codersera

2020, Logrocket 2021, Duomly 2022, Openbase 2022) listing their take on the best libraries. The

majority of these websites contained libraries which were listed multiple times across multiple

websites. Cross referencing the most frequently occurred libraries against its downloads showed

that - the more downloads a library possessed, the more likely it was to occur in a list.

5.1.2 Setting up library criteria

After collecting potential libraries, a ﬁltering process was applied based on criteria.

•Chart types

•Stand-alone libraries

•Access to the full version for free

The following list seen in Figure 5.1 contains all of the 26 initial libraries selected before going

through the criteria process. The green color represent a Yes while as the red color displays

No. Libraries which were having obvious necessity of external libraries and frameworks such as

React or Vue, were left out because of this speciﬁc property.

Figure 5.1: Initially selected libraries before criteria process

Firstly, due to the nature of the dataset being a time-series, a shift towards chart types that con-

tains the capability of displaying time-series data was necessary. Therefore, libraries that could

draw Line charts, Scatter plots, and Area charts were chosen. Secondly, a standalone version

of the library is crucial since implementing additional libraries could potentially alter the time

it takes to render a chart, thus - invalidating the collected data for that experiment. Finally,

it is crucial to have full access to a library and its functions before launching a ready product.

Thereof the ﬁnal criterion.

Since the time for this study is limited, there is not enough time to perform the experiment on

all of the libraries shown above in Figure 5.1. Therefore, only the six libraries that were the

easiest to implement were kept. The ﬁnal list of selected libraries can be seen in the list below.

•D3

•ECharts

•CanvasJS

•Chartist

•Highcharts

•Plotly

5.1.3 Experiment

Because the measuring procedure needs to be both reproducible and highly controllable, an

experiment is the most suitable research (Wohlin et al. 2012). Since three diﬀerent measurements

are collected, the experiment is split into three distinct parts, which can be seen in table 5.1.

The three parts of the experiment are all similar but diﬀer slightly regarding dependent and

independent variables. For the ﬁrst experiment concerning RQ1.1 and RQ1.2, the dependent

variable measured is the render time of the chart, and the independent variables changed is the

number of data points included in the dataset and the chart type. For the second experiment

concerning RQ2.1 and RQ2.2, the dependent variable is once again the chart render time, and

the independent variables are the number of charts rendered and the chart type. For the third

experiment concerning RQ3.1 and RQ3.2, the dependent variable is the memory usage, and the

independent variables are the number of data points rendered and the chart type. Aside from

these variables, there should be as little variance as possible for the experiment to give accurate

and valuable results.

Experiment 1 Experiment 2 Experiment 3

Independent

Variables

•Number of data points

•Charting library

•Chart Type

•Number of charts

•Charting library

•Chart Type

•Number of data points

•Charting library

•Chart Type

Dependent

Variables •Render time •Render time •Memory usage

Table 5.1: Independent and dependent variables for each experiment

5.1.4 Alternative Methods

An alternative method that could have been considered is a case study. A case study could have

been performed by measuring the performance of already existing websites using the included

libraries. However, ﬁnding working and representative websites for each library would prove diﬃ-

cult. Even harder would be comparing the performance amongst the libraries as it would require

diﬀerent websites using diﬀerent libraries to display the same or similar data. There would be

too many uncontrollable variables which would make an accurate comparison of libraries almost

impossible.

Another alternative method that could have been used is a survey. A survey would probably have

been the second best option, right behind an experiment. A survey could have been performed

by ﬁrst creating a similar web page for each library. Participants in the survey would get to test

and play around with each website and give their input on what website felt the best too use.

The participants could have been asked what web page felt the most responsive or which one

felt the best to use overall. A survey like this could also have been able to test more subjective

aspects, like the visual design of the libraries for example. However, the problem with a survey

is that all results would be gathered from the perception of the survey participants, and thus no

objective results would be gathered. Another major problem with this method is that memory

usage would be essentially impossible to test, as that is something that can not be experienced

in the same way render time can.

5.2 Approach

To be able to perform the experiment on diﬀerent dataset sizes, the creation of diﬀerent dataset

sizes has to be done. The choice of suitable sizes for these datasets was made by taking the last

month, the last six months, and the last year and creating a new dataset for each. These datasets

contain 704, 4070, and 8052 data points, respectively. The complete data set with 15906 data

points was also used to get the largest dataset possible from this dataset. These speciﬁc datasets

were chosen because they represent reasonable timespans to display in a chart on a website that

shows statistics for the covid-19 pandemic. Their sizes also give a good spread of sizes for the

experiment.

For each experiment an implementation of each chosen library is required to gather test results.

Each implementation consist of multiple HTML ﬁles which was written to render diﬀerent type

of charts as well as multiple charts. The choice was made to embed the written JavaScript into

each HTML ﬁle in order to keep the number of ﬁles as low and organized as possible. Since

experiment 1 and 3 requires the size of the data to alter, a global variable was created which

controls what dataset size is used while rendering. This variable keeps the number of ﬁles low

and provides an easy way to switch between dataset sizes. As for experiment 2 which does not

depend on the dataset sizes to change, instead used multiple ﬁles for each variation, meaning

that - each chart type produced 3 HTML ﬁles which would render either 1, 11 or 22 charts

on a single page. The underlying JavaScript then used a global variable to automatically as-

sign each rendered chart the correct amount of data depending on the number of rendered charts.

The computer that was used to conduct these experiments was located on the campus at

Högskolan i Skövde. The computer had the following specs:

•NVIDIA GeForce RTX 2080

•Intel Core i7-9700k 3.60GHz

•16GB DDR4 RAM

In Experiment 1 the render time when rendering a single chart is observed and measured.

One measurement is taken for each combination of dataset and chart type, resulting in 12 mea-

surements per library. Each measurement is performed ﬁve times, and an average is calculated.

The initial approach to measuring the time taken was to place the inbuilt functionality of Con-

sole.time both before and after the line of code which executes and renders the chart. A render

time was gathered by taking the time diﬀerence between the two timestamps. However, it was

noticed that for some of the libraries, the Console.time line of code would execute before the

chart had ﬁnished rendering, resulting in invalid results. A workaround was instead adapted.

Every library requires a DOM element container to be created before a chart is rendered. Once

the chart is fully rendered, it would become a child node of the container. This opens the op-

portunity to check whether the container contains a child node by using a timer that runs every

10th millisecond. Once a child node is found, the timer stops, and a render time is gathered and

displayed in the Google Chrome browser console window.

In Experiment 2 the render time is once again observed and measured. The complete dataset

is always used in this experiment, and the number of rendered charts and chart types is modiﬁed.

One measurement is taken for each combination of chart type and the predetermined number

of charts, totaling nine measurements per library. The render time is measured the same way

as in experiment 1. However, since several DOM element containers are created, only the DOM

element that is created last is used to check when a child node appears. The timer stops as

this last DOM element has a child node, and the render time is displayed in the Google Chrome

browser console window.

In Experiment 3 the memory usage when rendering a single chart is observed and measured.

The peak memory usage on page load-in is collected once for each dataset and chart type combi-

nation, resulting in 12 measurements per library. To monitor the memory usage, Google chromes

built-in task manager was used. The option "Memory footprint" was monitored as it showed

how much memory was currently in use for any speciﬁc tab. To collect a measurement, a tab

loads one of the HTML documents, and the highest recorded memory footprint of that tab was

noted. This was done ﬁve times for each measurement, and an average was calculated. All tests

were run in an incognito tab to ensure any caching would not disrupt the experiment. It was also

made sure that the tab got some time to rest between each reload and measurement until the

memory footprint settled. This was to prevent a problem noticed early on, where the memory

footprint would peak much higher if it were reloaded too frequently.

6|Results

6.1 Scalability on increasing data points

The ﬁrst experiment is testing how the render time is aﬀected by varying the data set size. Four

diﬀerent data set sizes will be used and to easier reference these, the following will be used.

•704 datapoints: Category 1

•4070 datapoints: Category 2

•8052 datapoints: Category 3

•15906 datapoints: Category 4

6.1.1 Line Chart

Figure 6.1: Render time of a line chart for all libraries with diﬀerent dataset sizes

The D3 library has the lowest render time in all four categories. It reached a maximum average

of 44.42ms in category 4. The ECharts Library is the only library where an unusual pattern can

be seen. ECharts, on average, performed better in category 2 than in category 1. Its maximum

render time was measured at 111.12ms in category 4. The CanvasJS library had the second-

best performance in all categories except category 1. It reached a maximum render time of

75.48ms in category 4. The Chartist library saw the highest measured maximum render time

of any library, measuring 334.24ms in category 4. Despite this, the library also produced the

second-lowest render time in the category 1, measuring 26.34ms. The Highcharts library saw

a maximum render time of 112.88ms in category 4. The Plotly library saw a maximum render

time of 118.92ms in category 4. To get a closer look at all the separate diagrams for each library,

see Appendix A.

6.1.2 Area Chart

Figure 6.2: Render time of an area chart for all libraries with diﬀerent dataset sizes

The D3 library has the lowest render time in all four categories for the area chart as well. It

reached a maximum render time of 75.36ms in category 4. The ECharts library showed the

same nonlinear pattern for the area chart as it did for the line chart, category 2 rendering a

bit faster than category 1. Its maximum render time was measured at 119.78ms in category 4.

The CanvasJS library once again performed the second-best in all categories except for category

1. The maximum render time for the area chart is 78.5ms in category 4. The Chartist library

also performed in line with previous patterns, reaching the highest measured render time of any

library with 366.26ms in category 4, but also had the second-lowest render time in category 1.

The Highcharts library reached a maximum render time of 133.84ms in category 4. The Plotly

library reached a maximum render time of 120.4ms in the 15906 category. To get a closer look

at all the separate diagrams for each library, see Appendix B.

6.1.3 Scatter Plot

Figure 6.3: Render time of a scatter plot for all libraries with diﬀerent dataset sizes

The D3 library had the best render time in all categories except for category 4. It reached a

maximum render time of 197.64ms in category 4. The ECharts library did not show the same

nonlinear pattern in the scatter plot as in the other two chart types. The library also performed

worse compared to the other libraries than it did for the other chart types, reaching the highest

measured render time of any library of 691.78ms in the 15906 category. The CanvasJS library

performed following the same pattern as earlier charts for categories 1, 2, and 3. However, it

showed the best performance out of all libraries in category 4, reaching a render time of 143.38ms.

The Chartist library performed signiﬁcantly better in contrast to the other charts types, achieving

the second-lowest render time in category 1 and the third-best in all other categories. It reached

a maximum render time of 300.72ms in category 4. The Highcharts library followed a similar

pattern to how it performed in the other chart types, where it performed worse than average

with a maximum render time of 463.3ms in category 4. The Plotly library had the slowest render

time in category 1 and the second slowest render time in all other categories. The library reached

a maximum render time of 655.92ms. To get a closer look at all the separate diagrams for each

library, see Appendix C.

6.2 Scalability of multiple charts

Within the second experiment, three diﬀerent amounts of charts were used. To easier reference

these, the following will be used.

•1 chart: Category 1

•11 charts: Category 2

•22 charts: Category 3

6.2.1 Line Chart

Figure 6.4: Render time of diﬀerent number of line charts rendered

The D3 library had the lowest render time in all categories, where it reached a maximum render

time of 64.3ms when rendering 22 charts. The ECharts library had a maximum render time

of 185.7ms in category 3, making it the second-best performing library in that category. The

CanvasJS library had the second-best performance in categories 1 and 2, then slightly fell behind

ECharts for category 3, where the highest render time of 209.88ms was recorded for the CanvasJS

library. The Chartist library was by far the worst-performing in this experiment. It had more

than twice the render time of the second-worst library in all categories. The maximum render

time being 1320.36ms in category 3. The Highcharts library had the fourth-lowest render time of

all libraries in all categories. Reaching a maximum render time of 357.18ms in category 3. The

Plotly library had the ﬁfth-lowest render time in all categories. Reaching a maximum render

time of 481.98ms in category 3. To get a closer look at all the separate diagrams for each library,

see Appendix D

6.2.2 Area Chart

Figure 6.5: Render time of diﬀerent number of area charts rendered

The D3 library, just like for line charts, had the lowest render time in all categories. Category

2 had a lower render time than category 1, while category 3 had the highest render time of

89.82ms. The ECharts library had the second-lowest render time in category 3, where it also

saw its highest render time at 205.14ms. The CanvasJS had the second-lowest render time in

categories 1 and 2. Its highest render time can be seen in category 3, measuring 212.94ms. The

Chartist library, just like for line charts, had by far the worst performance in all categories. Once

again, it had twice the render time of the second-highest render time in all categories. Its highest

render time measured 1374.1ms in category 3. The Highcharts and Plotly libraries had worse

than average render times in all categories. Both show either the fourth or ﬁfth highest render

time in each category. The libraries reached the highest render time of 374.24ms and 484.64ms,

respectively. To get a closer look at all the separate diagrams for each library, see Appendix E

6.2.3 Scatter Plot

Figure 6.6: Render time of diﬀerent number of scatter plot charts rendered

The D3 performed well overall, but was beaten out by the CanvasJS library in categories 1 and

2. The D3 library had similar render times in all categories, reaching its highest render time in

category 2 with 200.76ms. The ECharts library had the highest render time of any library in

category 1 but had one of the lowest three render times in both categories 2 and 3. Its highest

render time, measured in category 1, reached 691.78ms. The CanvasJS library, as mentioned,

had the lowest render times in both categories 1 and 2. In category 3, it had the second-lowest

render time, measuring 264.9ms which was also its highest render time. The Chartist library

performed decently well in category 1 compared to previous patterns. It had the third-lowest

render time in that category at 300.72ms. However, in categories 2 and 3, it had the second-

highest and absolute highest render times, respectively. Its highest render time was measured

at 1300.16ms in category 3. The Highcharts and Plotly libraries once again had above average

render times. None of the libraries performed better than fourth place in any category. The

highest render times for the two libraries were measured at 659.5ms and 977.76ms, respectively.

To get a closer look at all the separate diagrams for each library, see Appendix F

6.3 Memory Usage

Within the third experiment, four dataset sizes were used. To easier reference these, the following

will be used.

•704 datapoints: Category 1

•4070 datapoints: Category 2

•8052 datapoints: Category 3

•15906 datapoints: Category 4

6.3.1 Line Chart

Figure 6.7: Memory usage of a line chart for all libraries with diﬀerent dataset sizes

The D3 library had the second-lowest memory footprint in all categories besides category 1, where

it had the third-lowest. The highest memory footprint measured for the library was 53828kB, in

category 4. The ECharts library had the highest memory footprint of all libraries in category 1.

In the other categories, it performed average compared to the other libraries, having either the

third or fourth lowest memory footprint in all of them. Its highest memory footprint recorded

was 72325.8kB in category 4. The CanvasJS library had the lowest memory footprint in all

categories. Its highest memory footprint recorded was 40732.8kB in category 4. The Chartist

library had the second-lowest memory footprint in category 1. However, it performed the worst

and had the highest memory footprint in all other categories. The highest memory footprint

recorded for the library was 108142.4kB in category 4. The Highcharts performed quite poorly

compared to the other libraries, having the fourth-lowest memory footprint in category 2 but the

ﬁfth-lowest in the rest, only beating out Chartist. The highest memory footprint recorded for the

library was 80817.6kB in category 4. The Plotly library performed a little better than Highcharts

but had a higher than average memory footprint overall. The highest memory footprint recorded

for the library was 64192.8kB in category 4. To get a closer look at all the separate diagrams for

each library, see Appendix G

6.3.2 Area Chart

Figure 6.8: Memory usage of an area chart for all libraries with diﬀerent dataset sizes

The D3 library showed the same pattern as it did for the line chart. Second lowest memory

footprint in all categories except category 1, where it had the third-lowest. The library’s highest

memory footprint for the area chart was 60241.6kB in category 4. The ECharts library showed

the same pattern for the area chart as it did for the line chart. Out of all the libraries, it had the

highest memory footprint in category 1 and average in the other categories. Its highest memory

footprint recorded was 65789.6kB in category 4. The CanvasJS library once again had the

lowest memory footprint in all categories. Its highest memory footprint recorded was 37161.6kB

in category 4. The Chartist library showed the same pattern as it did for the line chart. It

had the second-lowest memory footprint in category 1 and the absolute highest in the rest.

Its highest recorded memory footprint was 107580.8kB in category 4. The Highcharts library

followed a similar pattern for the area chart as it showed for the line chart. Having a higher than

average memory footprint in the majority of categories. Its highest memory footprint recorded

was 77022.8kB in category 4. The Plotly library had an overall higher than average memory

footprint. However, it had the smallest diﬀerence in memory footprint between the lowest and

highest recorded points. Its highest memory footprint recorded was 62426.4kB in category 4. To

get a closer look at all the separate diagrams for each library, see Appendix H

6.3.3 Scatter Plot

Figure 6.9: Memory usage of a scatter plot for all libraries with diﬀerent dataset sizes

The D3 library showed a similar pattern to the other chart types. It had the third-lowest memory

footprint in categories 1 and 4 while having the second-lowest memory footprint in categories

2 and 3. Its highest memory footprint recorded was 124651.2kB. The ECharts library had an

above-average memory footprint in all categories. It either had the fourth- or ﬁfth-lowest memory

footprint in all categories. Its highest recorded memory footprint was 135801.6kB in category 4.

The CanvasJS library once again had the lowest memory footprint in all categories. Its highest

memory footprint recorded was 38953.6kB in category 4. This is almost a third of the second-

lowest memory footprint in category 4. The Chartist library performed better than average in

all categories. It had the second-lowest memory footprint in categories 1 and 4, while it had

the third-lowest memory footprint in categories 2 and 3. Its highest memory footprint recorded

was 103827.2kB in category 4. The Highcharts library followed a similar pattern here as it did

for the other chart types. It had a higher than average memory footprint in all categories. Its

highest recorded memory footprint was 174936kB in category 4. The Plotly library had the

highest recorded memory footprint in all categories. Its highest recorded memory footprint was

220227.2kB in category 4. To get a closer look at all the separate diagrams for each library, see

Appendix I

7|Analysis

In order to answer the research questions with conﬁdence, and to decide whether the hypotheses

hold or not, ANOVA tests were performed on the gathered data. In each case where a signiﬁcant

diﬀerence occured, a Tukey-test was performed in order to see where the signiﬁcant diﬀerence

occurred. Each of the tests were conducted with a 95% conﬁdence interval. In all of the ﬁgures

seen below, a green cell indicates that there is a signiﬁcant diﬀerence, while a red cell indicates

that there is no signiﬁcant diﬀerence.

7.1 Scalability of increasing data points

For the ﬁrst experiment, the render time was measured based on the number of data points in a

single chart. This experiment helps answer the following research questions:

RQ1.1 Is there a signiﬁcant diﬀerence in render time when increasing the number of data points?

RQ1.2 Is there a signiﬁcant diﬀerence in render time between the chosen libraries?

7.1.1 Line Chart

RQ1.1 is answered by looking at how the render time of each library is aﬀected by the diﬀerent

dataset sizes. As presented in ﬁgure 6.1, for all cases besides one, there was an increase in

render time whenever the datapoints increased. The one case where the render time did not

increase was between categories 1 and 2 for the ECharts library. A likely explanation for this

anomaly was discovered when the experiment was performed. When rendering the smallest

dataset, each individual point is rendered. But when the dataset in category 2 is rendered, some

of the individual points does not get rendered, and instead only the line is rendered. This made

the render time decrease for category 2 compared to category 1. To establish if these changes

in render time were scientiﬁcally signiﬁcant, the Tukey-tests were performed on each library,

comparing the render times for each category. The results of these tests can be seen in ﬁgure

7.1.

Figure 7.1: Combined Tukey-test results within each library for Line chart

As shown in ﬁgure 7.1, the render time sees a signiﬁcant diﬀerence in all cases but two. These two

cases are between ECharts with 704 data points and 8052 data points and between Highcharts

with 704 data points and 4070 data points. As mentioned earlier, the reason why the render

time does not signiﬁcantly increase could be because both Highcharts and ECharts render all the

data points in the chart when using the smallest dataset, but when rendering a slightly larger

dataset, not all data points can be rendered exactly where they are and instead only the line is

rendered.

Figure 7.2: Combined Tukey-test results between every library for Line chart

RQ1.2 can be answered by looking at - how the render time between the libraries is aﬀected

by the diﬀerent dataset sizes. In ﬁgure 7.2 all but one instance shows that there is a signiﬁcant

diﬀerence in terms of render time between each other for line chart.

7.1.2 Area Chart

RQ1.1 Looking at the collected Tukey-tests for area chart in ﬁgure 7.3, a similar pattern of

EChart reveals itself. However with area chart, no signiﬁcant diﬀerence is seen between cate-

gories 1 and 2. There is also no signiﬁcant diﬀerence between categories 1 and 2 for CanvasJS.

All remaining combinations tested with Tukey-tests shows a signiﬁcant diﬀerence between the

diﬀerent data set sizes.

Figure 7.3: Combined Tukey-test results within each library for Area chart

RQ1.2 is answered for Area chart by evaluating ﬁgure 7.4 below. Within the table, there are

four occurrences that show no signiﬁcant diﬀerences. Three out of these involve the Plotly

library. For both Category 1 and Category 4 there is no signiﬁcant diﬀerence between Plotly

and ECharts. Then there is another in Category 3 between Plotly and Chartist as well as one

ﬁnal between CanvasJS and D3 in Category 4. The remaining t-tests resulted in a signiﬁcant

diﬀerence between each other.

Figure 7.4: Combined Tukey-test results between every library for Area chart

7.1.3 Scatter Plot

Answering RQ1.1 is very straightforward. As seen in ﬁgure 7.5, every tested instance between

data points shows a signiﬁcant diﬀerence as the data points increased.

Figure 7.5: Combined Tukey-test results within each library for Scatter plot

Similarly to ﬁgure 7.5 a combination of all Tukey-tests between each library for each data set,

seen in ﬁgure 7.6 shows that there are only signiﬁcant diﬀerences. Thereby giving an answering

to research question RQ1.2 for Scatter plot.

Figure 7.6: Combined Tukey-test results between every library for Scatter plot

7.2 Scalability of multiple charts

For the second experiment, the render time was measured based on the number of charts rendered.

This experiment helps answer the following research questions:

RQ2.1 Is there a signiﬁcant diﬀerence in render time when rendering a single chart compared to

rendering several charts when rendered data points stay consistent?

RQ2.2 Is there a signiﬁcant diﬀerence in render time between the selected JavaScript libraries

when rendering a single chart compared to rendering several charts when rendered data

points stay consistent?

7.2.1 Line Chart

RQ2.1 can be answered by looking at how the render times for each library are aﬀected by how

many charts are being rendered.

For the line chart, RQ2.1 can be answered by looking at how the render times for each library

are aﬀected by how many charts are being rendered. By looking at Figure 6.4 the render time

increased for each library as the number of charts increased. The Tukey-test table shown in Figure

7.7 shows that the render times are signiﬁcantly diﬀerent from each other when increasing the

number of rendered charts within each of the libraries.

Figure 7.7: Combined Tukey-test results within each library for Line chart

RQ2.2 for the line chart can be answered by looking at the Tukey-test tables in Figure 7.8.

This table shows no signiﬁcant diﬀerence in render time between the libraries in three cases.

The ﬁrst case is when rendering a single chart, where there is no signiﬁcant diﬀerence in render

time between the ECharts library and the Highcharts library. The other cases where there is

no signiﬁcant diﬀerence are between the ECharts library and the CanvasJS library, both when

rendering 11 charts and 22 charts. By looking at Figure 6.4 the CanvasJS library seems to

perform better in terms of render time with a smaller dataset, but the render time seems to

increase faster when rendering more charts compared to the ECharts library. This library does

not seem to increase as much in render time when more charts are rendered.

Figure 7.8: Combined Tukey-test results between every library for Line chart

7.2.2 Area Chart

For the Area chart, RQ2.1 can be answered by looking at the Tukey-test tables in Figure 7.9.

This table shows that there is a signiﬁcant diﬀerence in render time for each library as the

number of charts increases. However, by looking at Figure 6.5 the render time for the D3 library

is reduced when rendering 11 charts instead of a single chart. But the render time increases

again when rendering 22 charts.

Figure 7.9: Combined Tukey-test results within each library for Area chart

By looking at Figure 7.10 RQ2.2 can be answered. In this case, there is a signiﬁcant diﬀerence

between all of the libraries except for the D3 library and the CanvasJS library when rendering a

single chart, the ECharts library, and the Plotly library when rendering a single chart, and the

ECharts library and the CanvasJS library when rendering 22 charts.

Figure 7.10: Combined Tukey-test results between every library for Area chart

7.2.3 Scatter Plot

For the Scatter plot, RQ2.1 can be answered by looking at the Tukey-test tables in Figure 7.11.

These tables show that there is a signiﬁcant diﬀerence in render time for all the libraries except

for the D3 library. This library seems to function in the same way even though the number of

charts rendered increases. The render time even decreases when rendering 22 charts compared

to both a single chart and 11 charts. This means that the D3 library performs better in terms of

render time when dividing the dataset into several charts, and the scalability in terms of render

time when increasing the number of charts seems very good for the D3 library. The same can

be said about the EChart library; it performs better in terms of render time when rendering 11

charts compared to when rendering a single chart or 22 charts.

Figure 7.11: Combined Tukey-test results within each library for Scatter plot

RQ2.2 can be answered by looking at the Tukey-test tables in Figure 7.12. These tables show

that there is a signiﬁcant diﬀerence in render time between all libraries except for between the

D3 library and the CanvasJS library when rendering 11 charts and between the ECharts library

and Highcharts library when rendering 22 charts. Figure 6.6 shows that the CanvasJS library

has the lowest render time when only rendering a single chart, and the D3 library performs best

in render time when rendering 22 charts.

Figure 7.12: Combined Tukey-test results between every library for Scatter plot

7.3 Scalability of memory usage

For the third experiment, the memory was measured based on the number of data points in a

single chart. This experiment helps answer the following research questions:

RQ3.1 Is there a signiﬁcant diﬀerence in allocated working memory depending on the number of

data points?

RQ3.2 Is there a signiﬁcant diﬀerence in allocated working memory depending on the implemented

library?

7.3.1 Line Chart

RQ3.1 can be partially answered by looking at the memory footprint of each library depending

on the diﬀerent dataset sizes. For the line chart tests, there were two cases in which the memory

footprint was greater when rendering a smaller dataset. This behavior was seen in the ECharts

and HighCharts libraries. In both cases, the memory footprint in category 1 was greater than in

category 2. This has a likely explanation that was ﬁrst mentioned in chapter 7.1.1. That being

that these two libraries rendered each individual data point when working with a small enough

dataset, which was the case for 704 data points. In all other cases, the memory footprint grew

when the data points increased. For which cases this increase was signiﬁcant is presented in 7.13.

Figure 7.13: Combined Tukey-test results within each library for Line chart

As can be seen in ﬁgure 7.13 only the Chartist library saw a signiﬁcant increase in every category.

For the other libraries, the diﬀerence in memory footprint between categories 1, 2, and 3 was not

always signiﬁcant. However, the memory footprint of category 4, which uses the complete data

set, is always signiﬁcantly higher than in all other categories. This indicates that the memory

footprint of the libraries will increase when the dataset grows bigger.

Figure 7.14: Combined Tukey-test results between every library for Line chart

RQ3.2 can be answered by looking at ﬁgure 7.14. The Tukey-tests show that there are cases in

categories 1, 2, and 3 where the diﬀerences in memory footprint is not signiﬁcant. However, the

majority of diﬀerences prove to be signiﬁcant, indicating that there are diﬀerences between the

library in memory footprint.

7.3.2 Area Chart

Taking a look at the area chart results with RQ3.1 in mind. As for the line chart, there were

two cases in which the memory footprint was greater when rendering a smaller dataset. One of

these cases was again ECharts, the diﬀerence here being that category 1 had a greater memory

footprint than both category 2 and 3. The other case was Plotly which had a greater memory

footprint in category 1 than category 2. The signiﬁcance of all the diﬀerences is presented in

ﬁgure 7.15.

Figure 7.15: Combined Tukey-test results within each library for Area chart

Looking at ﬁgure 7.15, a similar pattern to what ﬁgure 7.13 shows can be observed. Chartist is

the only library that saw a signiﬁcant increase in every single category. All other libraries had

cases where no signiﬁcant diﬀerences could be observed. All these cases occur between categories

1, 2, and 3. Once again, the memory footprint in category 4 is signiﬁcantly greater than all other

categories for all libraries. Thus, the area chart results also indicate that the memory footprint

will increase together with the dataset size.

Figure 7.16: Combined Tukey-test results between every library for Area chart

Taking a look at ﬁgure 7.15 further assists in assessing RQ3.2. The Tukey-tests for the area

chart results further indicate that there are signiﬁcant diﬀerences between the libraries.

7.3.3 Scatter Plot

Continuing the investigation of RQ3.1 with the scatter plot results further reinforces previous

indications that the memory footprint increases when the dataset size grows. In all cases when

rendering a scatter plot, the memory footprint increased whenever the dataset size was increased.

The signiﬁcance of these increases is presented in ﬁgure 7.15.

Figure 7.17: Combined Tukey-test results within each library for Scatter plot

Figure 7.17 shows that in all cases besides one, the increase in memory footprint is scientiﬁcally

signiﬁcant. The only case in which there was no signiﬁcant increase was between category 1 and

2 for the D3 library.

Figure 7.18: Combined Tukey-test results between every library for Scatter plot

Looking at the signiﬁcant diﬀerences in memory usage between libraries in ﬁgure 7.18, previous

indicators about the answer to RQ3.2 are once again reinforced. The majority of Tukey-tests

show a signiﬁcant diﬀerence between the libraries.

7.4 Conclusions

To show that each hypothesis for this study holds, the corresponding null hypothesis has to be

rejected. Each experiment has two hypotheses and corresponding null hypotheses.

Experiment 1

In the majority of all cases, there is a signiﬁcant diﬀerence in terms of render time when the num-

ber of data points increase. The results pertaining to scatter plot shows a signiﬁcant diﬀerence in

all cases, while the other two chart types saw two cases each where no signiﬁcance was present.

However, the fact that all libraries saw a signiﬁcant increase in render time when rendering all

15906 data points strongly indicates that the render time does increase with data points. The

null hypothesis for H1.1 will be rejected as the results indicate that given hypothesis holds.

Only a few instances were found where there is no signiﬁcant diﬀerence in render time between

the libraries and chart types. Therefore the null hypothesis for H1.2 can be rejected.

Experiment 2

In all cases but one, there is a signiﬁcant diﬀerence in terms of render time when the number

of charts increase. The one case that shows no signiﬁcant diﬀerence is the D3 library when

rendering scatter plots. These results highly indicate that the render time signiﬁcantly diﬀers

when the number of chart on a web page is increased, and the null hypothesis for H2.1 will

be rejected. However, the deviation from these ﬁndings by D3’s scatter plots is interesting. As

it shows that the number of charts does not necessarily have to signiﬁcantly aﬀect the render time.

In terms of signiﬁcant diﬀerences between the libraries, there are only a few cases where there

is no signiﬁcant diﬀerence in render time. This means that the null hypotheses for H2.2 can be

rejected.

Experiment 3

The majority of Tukey-tests shows that there are signiﬁcant diﬀerences in memory usage de-

pending on the dataset size. Therefore the null hypothesis for H3.1 can be rejected.

As for the comparison between the libraries, the majority of Tukey-tests shows that there is

signiﬁcant diﬀerences between the library. The null hypothesis for H3.2 can therefore be rejected.

8|Discussion

8.1 Ethics

The research conducted does not rely on the involvement of humans subjects. Therefore a col-

lection of data which stores personal information and opinions of human subjects has not been

performed. We believe this step alone contributes to a minimized risk of handling data that

could be interpreted as unethical. Also, regarding the dataset which is used throughout the

experiments - the dataset contains information about the number of conﬁrmed cases of Covid-19

each day since the beginning of 2020 in Sweden. However, there is no personal data for every

individual case stored within the dataset, contributing to fully anonymized data.

The experiment has been done as objectively as possible, and no data has been tempered with.

All the results that was gathered has been saved to be able to analyze it again.

When other researchers works have been used, they have been referenced to give the correct

people credit for their work.

This research is not sponsored by any organization or other private entity. The authors does

not have any aﬃliation with any of the libraries and the research has not been performed on the

behalf of anyone involved with the libraries.

To make it easier to replicate this experiment all the code used was uploaded to GitHub (GitHub

Examensarbete 2022).

8.2 Threats to Validity

8.2.1 Internal Validity

When performing the experiments, the computer was only running the necessary applications.

However, there is a possibility that some background processes were running during the experi-

ment. This may have caused some inconsistent results from the measurements. To work around

this, more tests could have been performed. However, each experiment was done in one sitting

and therefore each individual test should have been run under the same circumstances. Also,

looking at the results from the experiments, the variation in the measurements does not seem to

be majorly aﬀected by this.

To exclude any caching done by the browser, the browser was run in incognito mode the entire

time during the experiments. The browser was also closed down and reopened after ﬁve minutes

to make sure that each test ran under similar conditions.

The results, analysis, and conclusions were made on ’only’ ﬁve measurements in each case, and

an average was then drawn from this. This average will not be as accurate as if an average

was taken from a larger set of measurements. Which could also lead to the results not being as

accurate as they could have been.

The way the measurements for memory usage are made could be inaccurate. This is because

the measurements are taken manually and the built-in task manager in Google Chrome may not

update the memory footprint when the actual spike of memory usage happens. However, since

all of the measurements were done in the exact same way in all cases where memory usage was

measured, the results should give an indication of how the libraries perform compared to each

other.

The same argument can be used against the measurement of render time. The render time was

measured by checking if the library was ﬁnished rendering on an interval of 10 milliseconds. The

results may therefore be slightly inaccurate with an error margin of 10 milliseconds. However,

in most cases, an error margin of 10 milliseconds would not alter the results in a meaningful

manner and all measurements were done in the exact same way for each case. Therefore, the

results should still give a good indication of how the libraries perform compared to each other.

8.2.2 External Validity

This study was conducted on only one browser, hence the results only apply to this speciﬁc

browser. If the experiments were made on a diﬀerent browser, the results could be diﬀerent

compared to the results in this study. Therefore these results only applies to the browser used

in this study. To work around this, studies on diﬀerent browsers has to be done.

The results from this study was taken from a controlled environment where only the chart was

rendered on the webpage. The actual numbers from the measurements may not be representative

in a real application. However, the results should give an indication of how the libraries performed

compared to each other.

8.3 Value to Society

This study has been done to evaluate where the selected JavaScript libraries performs well, e.g.,

if the library performs better when a smaller dataset size is used, or if the library performs

better when rendering several charts. Getting better knowledge about this will make it easier

for developers, that does not want to or have the time to create their own charting methods, or

the time to compare diﬀerent charting libraries, to choose an already existing charting library

that performs well in the area that the developers wants to use it.

When visualization of charts works well on a website, the users will most likely be satisﬁed as

well. Because this will make it easier for the users to interpret and understand the data. If

the libraries that renders the charts are eﬃcient and easy for developers to use, this could lead

to more visualizations being done on the internet, which could improve the understanding of

statistical data.

8.4 Future Work

To continue this work, an even bigger dataset could be used, to see how well the libraries perform

with more data points, or to give a better look on the scalability of the libraries. A diﬀerent type

of dataset could also be used to be able to see how well the libraries performs with diﬀerent chart

types, since this study only uses a time-series dataset, chart types like: pie chart, bar charts, etc.

would make less sense to evaluate.

To further build on this study, more libraries could be included. This would help in terms of

getting an even broader view of where JavaScript charting libraries performs well.

By measuring other performance measurements, e.g., lines of code and response time, this re-

search could be expanded even further.

The time it takes to pre-process the data is never considered in this work. This could also be

an interesting perspective to look at how well diﬀerent JavaScript libraries both fetch the data

as well as how they perform while pre-processing the data to an acceptable format for the library.

8.5 Conclusions

This section will discuss and compare the performance of all libraries presented in the results

and analysis sections. This section will cover a broader analysis including all three experiments.

The overall best performing library in this study is the D3 library. In experiment 1 and 2, the D3

library is a clear winner in terms of the lowest render time. In experiment 1 the library had the

lowest render time in 11 out of 12 possible chart type and category combinations. In experiment 2

the library had the lowest render time in 7 out of 9 possible chart type and category combinations.

The library has the lowest render time for all line- and area-chart categories in both experiments.

Meaning that it is only beaten in some of the scatter plot categories. As for experiment 3, the

D3 library either had the second or third lowest memory footprint in all chart type and category

combinations, thus performing better than average. These results show that the D3 library is

the overall best performing library and therefore recommended in most use cases similar to what

has been researched here. The one use case where it can be challenged is for scatter plots, where

the CanvasJS library shows comparatively good results overall. Another thing that stands out

about the D3 library is the results in experiment 2, where the library saw the smallest diﬀerence

out of any library between the categories. Most interesting are the results for the scatter plot

where the library did not show any signiﬁcant diﬀerence between any categories. This is unique

as this pattern was not present anywhere else in the results. For the other two chart types the

library did lean towards a greater render time when rendering more charts but the diﬀerences

were still small in comparison the other libraries. These results together with the D3 library’s

already stellar render times leads to it being recommended for use cases where multiple charts

are to be rendered. This is line with the ﬁndings of Lee et al. (2014), which concluded that

the D3 library had the fastest render time out of their selected libraries. The libraries in their

study were not the same as the ones included in this study, exception being D3. However, this in-

dicates that D3 has a low render time in the context of popular JavaScript visualization libraries.

In experiment 3, the CanvasJS library had the lowest memory footprint by quite a margin. It

performed the best in all categories for all chart types, performing especially well in comparison

to the other libraries in the categories with more data points. This is clearly shown in category 4

for the scatter plot where CanvasJS had less than half the memory footprint of any other library.

The CanvasJS library is therefore recommended if a developer is looking for a memory eﬃcient

alternative.

As for the overall worst performing library, being the Chartist library. Chartist performed well

whenever the data set was small, as was the case in category 1 in both experiment 1 and 3.

It performed better than average in these cases compared to the other libraries but started to

struggle when the data size grew. Because of this behavior it performed by far the worst in

experiment 2 as the complete data set with the maximum dataset size was used in all test cases.

Comparatively with the other libraries though, the scatter plot results of the Chartist library

were not as bad. Chartist performed very similar for all chart types in all experiments, whereas

the other libraries all performed worse when rendering scatter plots rather than line and area

charts. The Chartist library therefore saw a better than average performance for scatter plots in

experiment 1 and 3. It still performed comparatively bad in experiment 2 most likely due to the

big dataset size. The only use case where this library would be recommended is when rendering

smaller datasets, preferably not much bigger than the smallest one in this study, being 704 data

points.

Then there are the three libraries that neither excelled nor underperformed in any of the exper-

iments. ECharts, Highcharts, and Plotly all performed in a similar vein. None of them were a

front runner in any of the experiments and more often then not kept the third, fourth, and ﬁfth

spots in terms of performance. There are only two cases where one of these libraries performed

better than third place. Both these cases can be observed in experiment 2 where ECharts has the

second-lowest render time when rendering 22 line- and area-charts. What is more common is one

of these libraries having the worst performance, which is the case in multiple categories. These

can mostly be observed in the categories where the Chartist library performed well: rendering

the small dataset and scatter plots. None of the ECharts, HighCharts, and Plotly libraries are

therefore recommended in any usage case. Interestingly, the ﬁndings of Carlström (2018) showed

Plotly had an average performance. However, she did not include any other libraries that were

in common with this study. This can therefore only be seen as an indication of Plotly’s middle

of the pack performance in the context of popular JavaScript libraries, not a conclusion.

All recommendations made in this section are exclusively derived from the performance metrics

presented in this study. Other performance metrics and more subjective aspects such as visual

design have not been considered for these recommendations.

References

Ali, S. M., Gupta, N., Nayak, G. K. & Lenka, R. K. (2016), ‘Big data visualization: Tools and

challenges’, pp. 656–660.

Aparicio, M. & Costa, C. J. (2015), ‘Data visualization’, Communication design quarterly review

3(1), 7–11.

Bonald, T., de Lara, N., Lutz, Q. & Charpentier, B. (2020), ‘Scikit-network: Graph analysis in

python.’, J. Mach. Learn. Res. 21(185), 1–6.

BSD-3-Clause (2022), ‘D3 license’, https://opensource.org/licenses/BSD-3-Clause. [On-

line; accessed 10-March-2022].

CanvasJS (2022), ‘Canvasjs library’, https://canvasjs.com/. [Online; accessed 10-March-

2022].

Carlström, A. (2018), ‘A comparative study between diﬀerent javascript libraries for visualiza-

tion: Performance measurements of javascript libraries for statistical graphs and diagrams (Dis-

sertation)’, Retrieved from: http://urn.kb.se/resolve?urn=urn:nbn:se:his:diva-15491.

Chartist (2022), ‘Chartist library’, https://gionkunz.github.io/chartist-js/. [Online; ac-

cessed 10-March-2022].

Codersera (2020), ‘Codersera’, https://codersera.com/blog/

javascript-data-visualization-libraries/. [Online; accessed 31-January-2022].

D3.js (2022), ‘D3 library’, https://d3js.org/. [Online; accessed 10-March-2022].

Duomly (2022), ‘Duomly’, https://www.blog.duomly.com/

javascript-data-visualization-libraries/. [Online; accessed 31-January-2022].

ECharts (2022), ‘Echarts’, https://echarts.apache.org/en/index.html. [Online; accessed

10-March-2022].

Folkhälsomyndigheten (2022), ‘Covid-19 dataset’, https://www.folkhalsomyndigheten.

se/smittskydd-beredskap/utbrott/aktuella-utbrott/covid-19/

statistik-och-analyser/bekraftade-fall-i-sverige/. [Online; accessed 27-January-

2022].

GitHub ECharts (2022), ‘Github echarts’, https://github.com/apache/echarts. [Online; ac-

cessed 10-March-2022].

GitHub Examensarbete (2022), ‘Examensarbete’, https://github.com/a19frebo/

Examensarbete.

Highcharts (2022), ‘Highcharts library’, https://www.highcharts.com/. [Online; accessed 10-

March-2022].

Lee, S., Jo, J.-Y. & Kim, Y. (2014), ‘Performance testing of web-based data visualization’,

pp. 1648–1653.

Logrocket (2021), ‘Logrocket’, https://blog.logrocket.com/

top-javascript-data-visualization-libraries-2021/. [Online; accessed 31-January-

2022].

MDN CSS (2022), ‘Css: Cascading style sheets’, https://developer.mozilla.org/en-US/

docs/Web/CSS. [Online; accessed 7-April-2022].

MDN HTML (2022), ‘Html: Hypertext markup language’, https://developer.mozilla.org/

en-US/docs/Web/HTML. [Online; accessed 7-April-2022].

MDN Javascript (2022), ‘Javascript basics’, https://developer.mozilla.org/en-US/docs/

Learn/Getting_started_with_the_web/JavaScript_basics. [Online; accessed 7-April-

2022].

Monterail (2021), ‘Monterail’, https://www.monterail.com/blog/

javascript-libraries-data-visualization. [Online; accessed 31-January-2022].

Nah, F. F.-H. (2004), ‘A study on tolerable waiting time: how long are web users willing to

wait?’, Behaviour & Information Technology 23(3), 153–163.

Netek, R., Brus, J. & Tomecka, O. (2019), ‘Performance testing on marker clustering and

heatmap visualization techniques: A comparative study on javascript mapping libraries’, IS-

PRS International Journal of Geo-Information 8(8).

Openbase (2022), ‘Openbase’, https://openbase.com/categories/js/

best-javascript-data-visualization-libraries. [Online; accessed 31-January-2022].

Persson, J. (2021), ‘Scalability of javascript libraries for data visualization (Dissertation)’, Re-

trieved from: http://urn.kb.se/resolve?urn=urn:nbn:se:his:diva-19994.

Plotly (2022), ‘Plotly library’, https://plotly.com/javascript/. [Online; accessed 10-March-

2022].

Shahzad, F., Sheltami, T. R., Shakshuki, E. M. & Shaikh, O. (2016), ‘A review of latest web

tools and libraries for state-of-the-art visualization’, Procedia Computer Science 98, 100–106.

Statistiska Centralbyrån (2022), ‘Statistikguiden’, https://www.scb.se/dokumentation/

statistikguiden/trender-och-analyser/diagram/. [Online; accessed 7-April-2022].

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B. & Wesslén, A. (2012), Experi-

mentation in software engineering, Springer Science & Business Media.

A|Appendix - Scalability on dat-

apoints for Line chart

III

B|Appendix - Scalability on dat-

apoints for Area chart

VII

VIII

C|Appendix - Scalability on dat-

apoints for Scatter plot

XII

D|Appendix - Scalability on Num-

ber of Charts for Line chart

XIII

XIV

XVI

E|Appendix - Scalability on Num-

ber of Charts for Area chart

XVII

XVIII

XIX

F|Appendix - Scalability on Num-

ber of Charts for Scatter plot

XXI

XXII

XXIII

XXIV

G|Appendix - Scalability on Mem-

ory usage for Line chart

XXV

XXVI

XXVII

XXVIII

H|Appendix - Scalability on Mem-

ory usage for Area chart

XXIX

XXX

XXXI

XXXII

I|Appendix - Scalability on Mem-

ory usage for Scatter plot

XXXIII

XXXIV

XXXV

XXXVI

0 views·87 pages

A performance investigation into JavaScript visualization libraries with the focus on render time and memory usage PDF Free Download

A performance investigation into JavaScript visualization libraries with the focus on render time and memory usage PDF free Download. Think more deeply and widely.

Uploaded by im_alyssa on 4/18/2026

/87

100%