Heimdall

Private project

Heimdall

Streaming bridge between message brokers and data stores, with built-in parsing, filtering, compression and per-flow Docker deployment.

About

Heimdall consumes from a source and produces to a target. Each source × target pair is a flow, packaged as its own Docker image.

Private project: the source is not public.

Techstack

language: Python
infra: KafkaRabbitMQDocker
database: PostgreSQLElasticsearch
framework: module-typica

Flow

Kafka / RabbitMQ → Parse · filter · compress(consume)
Parse · filter · compress → Kafka · RabbitMQ · Postgres · Elasticsearch · files(produce)

Flow

2 steps