Read about our recent Projects.

Project List

Investment Alert Platform with Market Alerts
Pet Food Recommender

Investment Alert Platform with Market Alerts

Michael Schmuklermann | CEO Data Pioneers
Published on 01.06.2024 (5 min. read)
Business Case
Creating a platform for stock market alerts posed an exciting challenge. The core idea was to provide users with a comprehensive tool to help make informed decisions on buying and selling assets. Users can log into both a website and an app, where they receive various stock signals. These signals, derived from multiple data sources, are designed to assist users in making timely decisions. Additionally, users have the ability to create watchlists of their preferred assets and receive real-time alerts, ensuring they never miss an important market movement.
Infrastructure
To build this platform, a robust infrastructure was essential. Here's a breakdown of the technologies and services we utilized:
‍ETL: We chose Apache Airflow for orchestrating our ETL workflows, Kubernetes for container orchestration, and Docker for containerization.
‍Data: PostgreSQL served as our reliable and scalable database solution.
‍API: We developed the API using Python's FastAPI framework, known for its speed and ease of use.
‍Hosting: Our hosting solutions included Google Cloud Services, leveraging Cloud Run, Cloud SQL, Composer, and GKE for various aspects of the platform.
‍Deployment: Deployment was managed through GitLab CI/CD pipelines, ensuring continuous integration and delivery. We also employed Pytest for automated testing, ensuring our codebase remained robust and error-free.
ETL
Our ETL process was designed to handle more than 15 different data sources. These included various APIs providing stock data, as well as web crawling different stock platforms to gather stock signals. The data we collected encompassed a range of metadata and other essential information, which we then extracted, cleaned, and transformed within our ETL workflows. The results were stored in our PostgreSQL database, providing a centralized repository for all the processed data.
Asset Matching Algorithm
One of the biggest challenges we faced was matching asset data from different data providers. The goal was to assign and consolidate these assets into a unique list, aggregating all relevant data across our data sources. This process was crucial in ensuring that users received accurate and comprehensive information, irrespective of the data provider. Our asset matcher had to be both robust and intelligent, capable of handling inconsistencies and variations in data formats and naming conventions.
Conclusion
Building a platform for stock market alerts required a combination of cutting-edge technologies and a good business understanding. From ETL processes to real-time data handling and deployment, every aspect had to be carefully engineered to provide users with reliable and timely information. The result is a powerful tool that helps users navigate the complexities of the stock market with confidence.

Pet Food Recommender

Michael Schmuklermann | CEO Data Pioneers
Published on 25.05.2024 (5 min. read)
Business Case
Developing a platform tailored for dog owners who seek the best food options based on their pets' lifestyles and food sensitivity profiles has been an exciting endeavor. Our solution provides a comprehensive food recommender system with direct links to purchase products. The recommendations are generated through two main methods:
‍Questionnaire: Users answer questions about their dogs, covering aspects like age, activity level, and dietary preferences (e.g., gluten-free).
‍PDF Analysis: Users can upload laboratory analysis reports in PDF format, which our system parses to extract relevant information.Using the collected data, we automatically calculate the most suitable food products for each dog, allowing users to make immediate purchases based on these recommendations.
Infrastructure
To support this platform, we employed a robust infrastructure, including:
‍ETL: We utilized APIs and Apache Airflow to manage our ETL workflows efficiently.
‍Data: PostgreSQL serves as our database solution, providing reliable and scalable data management.
‍Hosting: Our hosting needs are met by Google Cloud Platform, leveraging services such as Cloud Run, Composer, and Cloud SQL.
ETL
Our ETL process is designed to handle various types of data essential for the platform:
‍Food Product Data: We collect information such as the name, brand, country, and ingredients of food products. This data is then normalized and integrated into our data architecture.
‍User Q&A Data: User responses to the questionnaire are stored in a normalized data structure, ensuring consistency and ease of access.
‍PDF Parser: We developed a program to parse information from uploaded PDFs, extracting relevant data about the user's dog and saving it in our database.
Food Product Recommender
The recommender system is at the heart of our platform. It aggregates all user information and provides tailored food recommendations with direct links for purchase.
‍Data Normalization: Ensuring all data is normalized is critical for the recommender to function accurately. This process involves standardizing data formats and handling inconsistencies.
‍Ingredient Consideration: The recommender must consider specific ingredients to avoid, such as gluten or corn. A significant challenge is understanding and managing the relationships between ingredients. For example, if a user wants to avoid fish, the system must also exclude subcategories like salmon.
Conclusion
Building this food recommendation platform for dogs required meticulous planning and the integration of advanced technologies. From collecting and normalizing data to developing a sophisticated recommender system, every step was crucial in delivering a seamless and personalized experience for our users. The result is a powerful tool that helps dog owners find the best food options for their pets, ensuring their health and well-being.