Cloud leader AWS shifts database focus to DataZone and Zero-ETL

To further strengthen our commitment to providing industry-leading coverage of computer technology, VentureBeat is pleased to welcome Andrew Brust and Tony Baer as regular contributors. Look for their articles in Data Pipeline.

Cloud leader AWS’s impact on IT trends takes many forms, but none has been more impactful than its stable database services.

At most annual re:Invent conferences, AWS has rolled out a shiny new database that confirms the company’s presence among cloud-based databases. These were sometimes open source and often custom built.

But this year was different. At AWS re:Invent 2022, the company turned its sights on making its existing range of cloud computing tools more palatable to enterprise IT. This means that data integration and data management are now coming to attention.

To that end, the company released Amazon DataZone data management services to catalog and manage data stored in the AWS cloud and on-premises. DataZone can also support third-party sources via APIs, AWS said, citing partners DataBricks, Snowflake and Tableau in this regard.

Event

Intelligent Security Summit On-Demand

Learn the critical role of AI and ML in cybersecurity and industry-specific case studies. Watch sessions on demand today.

Look here

The timing is right. Businesses are finding that the number of different data sources they need to combine is growing dramatically. Management and control of dispersed data holdings becomes burdensome.

Combine data feeds

Now as ever, cost efficiency is driving IT to the cloud, AWS CEO Adam Selipsky told re:Invent attendees. For AWS today, cost-effective data engines begin with Aurora, AWS’s version of open source PostgreSQL, and Redshift, the columnar MPP data warehouse that improved the economics of data analytics with its introduction in 2012.

The database procession that brought Aurora and Redshift also included RDS, Neptune, DynamoDB, DocumentDB, Elastic Cache, TimeStream and Quantum Ledger DB, some of which are stirring controversy as startups grapple with cloud giant AWS’s aggressive approach to open source licensing. .

Selipsky didn’t come to re:Invent to present a new database – although there were updates to several existing engines. Instead, he promoted the idea of ​​tying the existing portfolio together more effectively.

“Having all these tools to store and analyze data reveals the next challenge that people face … you have to be able to combine information across these different methods of data exploration to see the whole picture and really get insights,” he said.

Give them ETL

In his re:Invent address, Selipsky took aim at the integration challenges surrounding Extract Transform Load (ETL), the long-standing hi-tech backwater that innovators have recently been revisiting.

He announced new integrations that are said to eliminate the need for ETL between Amazon Aurora and Amazon Redshift services, and between Spark and Redshift.

Selipsky’s goal here is clear eyes. With low code/no code on the rise, it may be time to dial up “Zero ETL.” It’s a stage in data processing, involving a lot of repetitive custom scripting, that is necessary and generally swamped when digital transformation is the company’s ultimate goal.

The tedious work of ETL data preparation can stand in the way of progress. To show IT’s frustration with the process, Selipsky read an excerpt from a letter from a customer who described ETL as a “thankless, unsustainable black hole.” The new Aurora and Redshift capabilities help customers move toward a zero-ETL future on AWS, he said.

Echo of tableau

While perhaps overshadowed by machine learning and other announcements, the focus on larger data management issues at re:Invent 2022 suggests new maturity in AWS’s approach to IT’s data needs.

There is also the implication here that Adam Selipsky is setting a new course for the AWS cloud. Given his years at the helm of business intelligence provider Tableau, this is not entirely unexpected.

Under his watch, Tableau distinguished itself for innovation in visual data presentation and established itself as an expert in ease of use and drag-and-drop integration support for both structured and unstructured datasets.

AWS’s DataZone and Zero-ETL fit nicely into a similar picture of the evolution of cloud data. Future moves will be closely watched to see if AWS moves more up-stack in the data architecture.

VentureBeat’s mission will be a digital town square for technical decision makers to gain knowledge about transformative business technology and transactions. Discover our orientations.

Leave a Reply

Your email address will not be published. Required fields are marked *