Technology stack

Build a robust and scalable data engineering stack

Our multi-layer data engineering tech stack helps teams manage and analyze data effectively.

Build a real-time engineering data platforms based on tried-and-tested data warehouses, data lakes, data integration tools, data visualization, BI, and governance software.

Programming languages and ML & AI frameworks

Python

Programming language that lets you work more quickly and integrate your systems more effectively

Javascript

Lightweight interpreted programming language with first-class functions

Django

High-level Python web framework

FastAPI

Fast, web framework for building APIs with Python

Flask

Lightweight WSGI web application framework

Milvus

The high-performance vector database

sktime

A unified framework for machine learning with time series

LightGBM

A gradient boosting framework based on tree-based learning algorithms

Libraries

PyTorch

A ML library based on the Torch library, used for applications such as computer vision and natural language processing

Keras

An open-source library that provides a Python interface for artificial neural networks

Prophet

A procedure for forecasting time series data based on an additive model

The Microsoft Cognitive Toolkit

A unified deep-learning toolkit that describes neural networks as a series of computational steps via a directed graph

FastAI

A deep learning library which provides high-level components

Tensorflow

A software library for machine learning and artificial intelligence

scikit-learn

Free and open-source machine learning library for the Python programming language

XGBoost

A scalable, distributed gradient-boosted decision tree (GBDT) machine learning library

Sentence Transformers

The Python module for accessing, using, and training state-of-the-art text and image embedding models

OpenCV

The world's biggest computer vision library

BigDL

A distributed deep learning library for Apache Spark

Horovod

A distributed deep learning training framework for PyTorch, TensorFlow, Keras and Apache MXNet

Seaborn

A Python data visualization library based on matplotlib

Pandas

A fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language

Tableau

A Python library for the Tableau Server REST API

Transform your data with proven tools and expertise

Free consultation

Natural language processing

NLTK :: Natural Language Toolkit

A suite of open source Python modules, data sets, and tutorials supporting research and development

SpaCy

A free open-source library for Natural Language Processing in Python

Gensim

A Python library for topic modelling, document indexing and similarity retrieval with large corpora

fastText

Library for efficient text classification and representation learning

Computer vision

Detectron2

A Facebook AI Research's next generation library that provides state-of-the-art detection and segmentation algorithms

Torchvision

A package of popular datasets, model architectures, and common image transformations for computer vision

Mmdetection

An open source object detection toolbox based on PyTorch

Pillow

A free and open-source additional library for the Python programming language that adds support for opening, manipulating, and saving many different image file formats

Albumentations

A comprehensive, high-performance framework for augmenting images to improve machine learning models

Data preprocessing and ETL

PySpark

Enables you to perform real-time, large-scale data processing in a distributed environment using Python

Polars

An open-source library for data manipulation, known for being one of the fastest data processing solutions on a single machine

Dask

A flexible parallel computing library for analytics

Data analysis & visualization

NumPy

A Python library used for working with arrays. It also has functions for working in domain of linear algebra, fourier transform, and matrices

Plotly

For an easy transition from Python notebooks to AI-powered production data apps

Bokeh

An interactive visualization library for modern web browsers

Altair

A declarative visualization library for Python

Data storage

Get expert guidance on choosing between data warehouse, data lake, or a mix of the two for data storage

Open-source solutions

Apache HDFS

Distributed file system for scalable storage and processing of large datasets across clusters

Apache Druid

Real-time analytics database for fast, scalable querying and ingesting of large event streams

ClickHouse

Columnar database management system optimized for high-performance real-time analytics on large datasets

Ceph

Distributed storage system providing scalable object, block, and file storage for large data environments

MinIO

High-performance object storage system compatible with the S3 API for cloud-native environments

Paid solutions

Amazon S3

Scalable object storage service for secure data storage, retrieval, and backup in the cloud

Azure Data Lake Storage

Secure cloud storage service optimized for big data analytics and processing

Google Cloud Storage

Scalable, secure object storage solution for unstructured data, with built-in analytics and backup

IBM Cloud Object Storage

High-security scalable cloud object storage for storing and managing unstructured data

Snowflake

Cloud-based data platform for scalable data warehousing, analytics, and secure data sharing

Databricks

Unified data analytics platform for big data processing, machine learning, and collaborative data science

Google BigQuery

Fully managed, serverless data warehouse for fast, scalable analytics on large datasets

Azure Blob Storage

Scalable object storage service for unstructured data, optimized for cloud applications and analytics

Data ingestion

Scalable and cost-effective data ingestion tools for batch processing and data streaming.

Open-source solutions

Airbyte

Syncing data between APIs, databases, and warehouses

Singer

Extracting, transforming, and loading data

Logstash

Collecting, transforming, and forwarding data in real time.

Fluentd

Unifying and processing logs and event data in real-time

Apache Kafka

Real-time data pipelines and applications

Redpanda

High-performance, low-latency real-time data processing

Paid solutions

IBM InfoSphere DataStage

Designing, developing, and running data integration workflows

Oracle GoldenGate

Transactional data management

SAP Data Services

Data integration, transformation, and cleansing

Google Cloud Data Fusion

Building and managing scalable data integration pipelines

Azure Data Factory

Creating, scheduling, and orchestrating ETL workflows at scale

AWS Glue

Discovering, preparing, and integrating data for analytics and ML

Azure Event Hubs

Real-time event ingestion and processing

Google Pub/Sub

Real-time messaging service

AWS Kinesis Data Streams

Collecting, processing, and analyzing streaming data

Data processing and transformation

Build a tech stack for processing raw data and transforming it to fit business logic

Open-source solutions

Apache Flink

Stream processing framework for real-time, scalable, and distributed data processing and analytics

Apache Spark

Unified analytics engine for large-scale data processing, featuring batch and real-time streaming capabilities

TensorFlow

Platform for machine learning and deep learning, used for building and deploying AI models

Dbt

Data transformation tool enabling analytics engineers to transform, test, and document data in SQL

Paid solutions

Azure Data Factory

Cloud-based data integration service for orchestrating and automating data movement and transformation at scale