The Meet Group is a leading provider of interactive online dating solutions. After several acquisitions, The Meet Group sought a solution to integrate its data pipelines and central data collection to drive better real-time aggregate insights and analysis. With help from Upsolver, The Meet Group developed a powerful and scalable data lake that enables the company to save time and innovate faster to meet business demands.
Nothing can replace the powerful feelings we get when we genuinely connect with another human being. Sometimes, the challenge lies in how we get there.
The Meet Group’s mission is to help safely foster human connections using the power of technology. The Meet Group’s mobile apps are comprised of five brands and offer different experiences and options for connecting individuals. Its applications are used by millions across the globe daily and are available in multiple languages. Given the scope of The Meet Group’s reach, the company generates hundreds of millions of events from its software daily. The team then seeks to use the collected data to understand user behavior at an aggregate level, to drive better short- and long-term user experiences and application development.
“We have many daily high-velocity events we categorize into thousands of streams and collect into multiple data silos for analytics and microservices coordination,” says Bob Lunney, Lead Data Architect for The Meet Group.
The Meet Group grew quickly through acquisitions. With each acquisition, Lunney and the team were tasked with managing different data pipelines initially created to support a separate brand now managed under The Meet Group umbrella. “We have five different applications we run that all run off individual backends,” says Lunney.
The data team recognized that to drive insights for the company at an aggregate level and provide a more global perspective, they needed to consolidate information coming from each brand. Given the streaming nature of the data, The Meet Group sought to find a solution to integrate its data pipelines and central data collection to drive better real-time aggregate insights and analysis. Having already decided to focus all new development on Amazon Web Services (AWS), Lunney and the team turned to their AWS contacts for guidance.
Upsolver strives to make traditionally difficult data ingestion and ETL processing easier for users by removing the need to write code. The company’s mission is to empower users to focus on driving new business value with data, rather than managing and operating different data pipelines.
After becoming familiar with Upsolver’s solution, The Meet Group found it easy to get up-and-running with Upsolver quickly.
The Meet Group leverages Upsolver as a real-time collection and transformation engine that connects data producers such as Apache Kafka, Amazon Kinesis, and operational databases to analysis tools such as Amazon Athena, Amazon Redshift, and Amazon Elasticsearch Service. By choosing a data lake on AWS, The Meet Group can leverage the best tool for each analytics and data science use case, using Upsolver for scaling and optimizing the data pipelines automatically to meet their business demands.
All of The Meet Group’s data is collected to a centralized Kafka datahub. Upsolver ingests this data to Amazon Simple Storage Service (Amazon S3) and extracts a strongly typed schema-on-read which is updated in real time and presented in Upsolver’s visual user interface. In turn, The Meet Group users create data transformation jobs by mapping the schema-on-read to tables in various analytics tools. The syntax for these jobs is SQL-based and each job can mesh multiple data sources into a single table. Once the jobs are run by the user, Upsolver continuously updates the target tables and data becomes queryable in Athena. The data Upsolver creates for Athena is optimized for faster queries by using columnar formats such as Apache Parquet, compression, partitioning, and compaction of small files. The metadata for Athena tables is maintained by Upsolver in the AWS Glue Catalog.
Using Upsolver and AWS has helped Lunney and the team focus their efforts on data analysis, rather than pipeline development and operations. “Frankly, to do what Upsolver does for us would have taken our team months and months of coding—and we would not have come up with a tool as optimized as Upsolver,” explains Lunney. “Replacing our data pipelines with Upsolver dramatically reduced the operational and developmental burden on our team.”
By using outputs in Upsolver, The Meet Group has been able to send data to other affiliates and across suppliers very quickly and easily. “Trying to set that up on our own, particularly for streaming data, would have taken us quite a long time,” says Lunney. The ease with which Upsolver enables The Meet Group to set up outputs in its platform also helps the company take better advantage of AWS services for data management and analysis.
The intuitive nature of Upsolver’s tool has been invaluable for The Meet Group.
Originally published on the AWS website.