Streaming Join State Growth Simulation Dataset

#State Prediction #State Size Estimation #Memory Pressure Prediction #Output Rate Estimation #Stream Processing System Design #Streaming Join Operator Tuning #Memory Resource Planning #Stream Processing Performance Analysis

4460
0.3GB
JSON
CATL
MOBIUSI INC

Updated:2026-11-04

AI Analysis & Value Prop

This dataset simulates streaming join behavior using parameters such as the arrival rates of two input streams, join-key cardinality, time-window length, out-of-order degree, match rate, and state cleanup policy. Each record combines scenario inputs with estimates of state records, pending unmatched records, memory usage, memory pressure, cleanup rate, and output rate. It supports state modeling, memory resource assessment, operator tuning, and stream processing system design.

Dataset Insights

Sample Examples

{}

Technical Specifications

Field	Type	Description
file_name	string	File name
match_rate	number	Proportion of records from the two streams that successfully match within the join window, from 0 to 1.
key_cardinality	integer	Number of distinct keys participating in the join during the simulation period.
total_arrival_rate	number	Combined record arrival rate of Stream A and Stream B, in records per second.
out_of_order_degree	number	Degree of input record disorder, from 0 to 1; higher values indicate more pronounced disorder.
time_window_seconds	number	Length of the time window used to retain and match records in the streaming join, in seconds.
state_cleanup_policy	string	Policy used by the join operator to remove expired state or state for records whose matching is complete.
estimated_output_rate	number	Estimated rate of matched output records produced by the join operator, in records per second.
memory_pressure_ratio	number	State memory usage relative to available memory capacity or a configured memory budget.
stream_a_arrival_rate	number	Number of records arriving from Stream A per second.
stream_b_arrival_rate	number	Number of records arriving from Stream B per second.
estimated_memory_usage_mb	number	Estimated memory used by the join operator based on the size of its state, in megabytes.
simulation_duration_seconds	number	Duration of the streaming join simulation, in seconds.
estimated_state_size_records	number	Estimated number of state records that the join operator must maintain during the simulation.
estimated_cleanup_release_rate	number	Estimated number of expired or fully matched records released by the state cleanup policy per second.
effective_state_retention_seconds	number	Estimated effective state retention duration after accounting for the time window, out-of-order degree, and cleanup policy, in seconds.
estimated_pending_unmatched_records	number	Estimated number of records that have not yet matched under the current window and match rate.

Compliance Statement

Authorization Type	Proprietary - Commercial AI Training License (No Redistribution)
Commercial Use	Requires exclusive subscription or authorization contract (monthly or per-invocation charging)
Privacy and Anonymization	No PII, no real company names, simulated scenarios follow industry standards
Compliance System	Compliant with China's Data Security Law / EU GDPR / supports enterprise data access logs

Frequently Asked Questions

What does the dataset contain?: It describes simulated streaming join scenarios through input parameters and estimated outcomes. Inputs include two-stream arrival rates, key cardinality, window length, out-of-order degree, match rate, and cleanup policy. Estimated metrics include state records, pending unmatched records, memory usage, cleanup rate, and output rate.

Are these measurements from production systems?: The data is intended for simulation learning and scientific modeling. Its fields represent simulated scenario inputs and estimates, so they should not be assumed to be measurements from production systems. Validate or calibrate the data against actual workloads before using it to guide deployment decisions.

Which modeling and evaluation tasks does it support?: It can support state prediction, state size estimation, memory pressure prediction, and output rate estimation for stateful streaming joins. It can also help compare operator behavior under different windows, workloads, and cleanup policies.

How can I access the data, and what are the licensing terms?: Refer to the dataset product page for available download methods, formats, and license terms. Confirm the applicable permissions and restrictions before using the data in commercial projects or redistributing it.

Can't find the data you need?

Post a request and let data providers reach out to you.

Get this Dataset

Verified for Enterprise Use

Cite this Work

@dataset{Mobiusi2026,
  title={Streaming Join State Growth Simulation Dataset},
  author={MOBIUSI INC},
  year={2026},
  url={https://www.mobiusi.com/datasets/7a7fb01b51f7b60be4a1270b7f95956d},
  urldate={2026-10-27},
  keywords={streaming join simulation data,stream processing state prediction,join state growth,stream operator memory estimation,two-stream join modeling,state cleanup policy,stream processing performance evaluation,pending unmatched record prediction},
  version={1.0}
}

Using this in research? Please cite us.

placeholder

Scene

Domain

Modality

Task