InfoQ Homepage Big Data Content on InfoQ
-
Adopting Artificial Intelligence: Things Leaders Need to Know
Artificial intelligence (AI) can help companies identify new opportunities and products, and stay ahead of the competition. Senior software managers should understand the basics of how this new technology works, why agility is important in developing AI products, and how to hire or train people for new roles.
-
AWS Introduces Athena Provisioned Capacity
AWS recently announced a new feature Provisioned Capacity for Athena, that allows users to run SQL queries on fully-managed compute capacity for a fixed price and no long-term commitments.
-
AWS Data on EKS Provides Opinionated Data Workload Blueprints
AWS has released Data on EKS (DoEKS), an open-source project providing templates, guidance, and best practices for deploying data workloads on Amazon Elastic Kubernetes Service (EKS). While the main focus is on running Apache Spark on Amazon EKS, blueprints also exist for other data workloads such as Ray, Apache Airflow, Argo Workflows, and Kubeflow.
-
Apache Linkis Graduated to Apache Top-Level Project
Apache Linkis is a computation middleware that acts as a layer between upper-level applications and underlying engines, such as Apache Spark, Apache Hive and Apache Flink. It started as an Apache Incubator project in 2021 and graduated to a Top Level Project in January 2023.
-
Amazon Athena Now Supports Apache Spark Engine
Amazon Athena now supports the open-source distributed processing system Apache Spark to run fast analytics workloads. Data analysts and engineers can use Jupyter Notebook in Athena to perform data processing and programmatically interact with Spark applications.
-
Apache Druid 25.0 Delivers Multi-Stage Query Engine and Kubernetes Task Management
Apache Druid is a high-performance real-time datastore and its latest release, version 25.0, provides many improvements and enhancements. The main new features are: the multi-stage query (MSQ) task engine used for SQL-based ingestion is now production ready, and Kubernetes can be used to launch and manage tasks eliminating the need for middle managers...
-
How Twitter Automated Data Quality Check Process
Twitter engineering has recently shared a blog post on how they architected and developed a quality automation platform. Twitter digests and creates thousands of data sets for different data products and applications. The next natural step is to make sure of the quality of the data by adding automation on top of it. In this news post, we explore this architecture in more detail.
-
AWS Announces Clean Rooms for Secure Collaboration with Analytics Data
During the recent re:Invent conference, AWS announced the preview of Clean Rooms for analytics data. The new service provides safe environments where multiple customers can securely share and analyze data with control of how the data is used, reducing the risk of sharing personal data.
-
Uber Reduces Logging Costs by 169x Using Compressed Log Processor (CLP)
Uber recently published how it dramatically reduced its logging costs using Compressed Log Processor (CLP). CLP is a tool capable of losslessly compressing text logs and searching them without decompression. It achieved a 169x compression ratio on Uber's log data, saving storage, memory, and disk/network bandwidth.
-
Uber Freight Near-Real-Time Analytics Architecture
Uber Freight is the Uber platform dedicated to connecting shippers with carriers. Providing reliable service to shippers is crucial for Uber Freight. This is why the Carrier Scorecard was developed, with several metrics including on-time pickup/delivery, tracking automation, and late cancellations.
-
Unraveling Techno-Solutionism: How I Fell out of Love with “Ethical” Machine Learning
At the recent QCon San Francisco conference, Katherine Jarmul gave a talk on unravelling techno-solutionism, in which she explored the inherent bias in AI training datasets, the bias that assumes there will be a technical solution to almost any problem and that those technical solutions will be beneficial for mankind. She posed questions for technologists to consider when building products.
-
Snap Way to Design Ads Ranking Service Using Deep Learning
Snap engineering has recently published a blog post on how they designed their ads ranking and targeting service using deep learning. Showing ads to the users is the mainstream of social network platform monetization. Snap ad ranking system is designed to target the right user at the right time. Snap is providing an excellent user experience while preserving user privacy and security.
-
Open-Source Constellation K8 Engine Aims to Bring Confidential Computing to Kubernetes
Constellation is a Kubernetes engine that shields Kubernetes clusters from the rest of the cloud infrastructure using confidential computing and confidential VMs. This creates a confidential context that ensures data is always encrypted, both at rest and in memory.
-
Azure Data Explorer Supports Native Ingestion from Amazon S3
Microsoft recently announced the ability to natively ingest data from Amazon S3 into Azure Data Explorer (ADX). The new feature simplifies multi-cloud data analytics deployments, bringing data from Amazon S3 to Azure, without relying on custom ETL pipelines.
-
Microsoft Releases SynapseML 0.1.0 with .NET and Cognitive Services Support
Microsoft announced the first .NET-compatible version of SynapseML, a new machine learning (ML) library for Apache Spark distributed processing platform. Version 0.1.0 of the SynapseML library adds support for .NET bindings, allowing .NET developers to write ML pipelines in their preferred language.
Sponsored Content
7 Essentials When Selecting a NoSQL Database-as-a-Service (DBaaS)
This paper outlines 7 key considerations that help teams tap the many benefits a DBaaS has to offer — without falling into some of the common traps that impact agility, productivity, costs, and growth. Download Now.
SQL to NoSQL: Architecture Differences and Considerations for Migration
Learn the tradeoffs between flexibility, scale and cost, plus what to consider for successful SQL to NoSQL migrations. Download Now.
7 Reasons Not to Put an External Cache in Front of Your Database
Read this white paper to learn common approaches to caching data, 7 specific reasons why external caching can be a bad choice. real-world examples of successfully eliminating external cache by companies such as Comcast, and more. Download Now.