A very warm welcome to the February feature summary. It has been an active start to the year here at Upsolver, planning and delivering our Chill Data Summit in New York City. If you missed out, stay tuned because all sessions were filmed and will be available very soon. 

Live from NYC at the summit, our CEO, Ori Rafael, and our CTO, Yoni Eini, announced Upsolver’s support for Apache Iceberg – expect a big feature summary in March! – but in the meantime, check out the news release for more information. Needless to say, we keep our engineering team busy, so let’s dive into what they have been up to of late.

This update includes the following releases:

• 2024.01.02-13.40
• 2024.01.09-14.59
• 2024.01.16-08.45

Contents

• Jobs
  • Replication Jobs Now Support AWS Glue Data Catalog as a Target
  • Fixed New Apache Kafka Ingestion/Data Sources Stalling
  • Resolved Duplicate Ingestion of Same File from Amazon S3

• SQL Queries
  • Default Aggregation Window is Now Set to Job Interval

• Enhancements
  • Performance Improvement in Transformations Jobs / Materialized Views

Jobs

Replication Jobs Now Support AWS Glue Data Catalog as a Target

We now support AWS Glue Data Catalog as a target for replication jobs. Upsolver will create a table and maintain schema evolution in the target for every replicated table 

Learn how to create a connection to your AWS Glue Data Catalog.

Fixed New Apache Kafka Ingestion/Data Source Stalling 

The issue whereby a new Apache Kafka ingestion/data source was stalling when reading from the start in some situations.

Discover how to create a connection to your Apache Kafka source.

Resolved Duplicate Ingestion of Same File from Amazon S3

We came across an unusual bug whereby an Amazon S3 data source is used, but has a date pattern that does not follow lexicographical order. This sometimes resulted in duplicated ingestion of the same file. This is now fixed but please contact our technical support team should you experience this issue. 

What is the lexicographical order? Lexicographical order, also known as alphabetical order, is the arrangement of words based on the order of the letters in the alphabet. In lexicographical order, words are sorted from the beginning of the alphabet (A or a) to the end (Z or z), taking into consideration the sequence of letters. This order is commonly used for organizing dictionaries, directories, and lists to make it easy to locate and reference words or items. Each letter is assigned a specific position in the alphabet, and words are sorted based on the order of their corresponding letters. If there is a tie in the initial letter, the second letter is considered, and so on, until the complete order is determined.

Check out the documentation and learn how to create a connection to Amazon S3.

SQL Queries

Default Aggregation Window is Now Set to Job Interval

For aggregated jobs (transformation jobs using GROUP BY) with an unspecified time window in time filters, e.g. WHERE TIME_FILTER() or WHERE $event_time BETWEEN RUN_START_TIME() AND RUN_END_TIME(), the default aggregation window is now set to the job’s interval, replacing the previous infinite window default.

Learn how to create a transformation job with Upsolver.

Enhancements

Performance Improvement in Transformations Jobs / Materialized Views 

We have now fixed an issue that was affecting the performance of transformation jobs and materialized views. You should experience improved performance in your jobs, however, please get in touch with our support team if you have any problems.

Discover how you can create a transformation job or create a materialized view.

That’s it for February’s update. I’ll be back in March with the latest features and enhancements that will help you build and maintain your pipelines, and raise your data quality. In the meantime, if you have any questions, please reach out to our friendly support team, who will be happy to help. 

If you’re new to Upsolver, why not start your free 14-day trial, or schedule your no-obligation demo with one of our in-house solutions architects who will be happy to show you around.