performance-profiling

Goal

Provide tools to analyze simulation performance, identify bottlenecks, and recommend optimization strategies for computational materials science simulations.

Requirements

• Python 3.10+
• No external dependencies (uses Python standard library only)
• Works on Linux, macOS, and Windows

Inputs to Gather

Before running profiling scripts, collect from the user:

Input	Description	Example
Simulation log	Log file with timing information	simulation.log
Scaling data	JSON with multi-run performance data	scaling_data.json
Simulation parameters	JSON with mesh, fields, solver config	params.json
Available memory	System memory in GB (optional)	16.0

Decision Guidance

When to Use Each Script

Need to identify slow phases?
├── YES → Use timing_analyzer.py
│         └── Parse simulation logs for timing data
│
Need to understand parallel performance?
├── YES → Use scaling_analyzer.py
│         └── Analyze strong or weak scaling efficiency
│
Need to estimate memory requirements?
├── YES → Use memory_profiler.py
│         └── Estimate memory from problem parameters
│
Need optimization recommendations?
└── YES → Use bottleneck_detector.py
          └── Combine analyses and get actionable advice

Choosing Analysis Thresholds

Metric	Good	Acceptable	Poor
Phase dominance	<30%	30-50%	>50%
Parallel efficiency	>0.80	0.70-0.80	<0.70
Memory usage	<60%	60-80%	>80%

Script Outputs (JSON Fields)

All scripts wrap their payload in a top-level object with two keys: inputs and results. The fields below live under results.

Script	Key Outputs (under results)
timing_analyzer.py	results.phases, results.slowest_phase, results.total_time
scaling_analyzer.py	results.results, results.efficiency_threshold_processors, results.average_efficiency, results.baseline
memory_profiler.py	results.total_memory_gb, results.per_process_gb, results.field_memory_gb, results.solver_workspace_gb, results.matrix_storage_gb, results.warnings
bottleneck_detector.py	results.bottlenecks, results.recommendations

Workflow

Complete Profiling Workflow

1. Analyze timing from simulation logs
2. Analyze scaling from multi-run data (if available)
3. Profile memory from simulation parameters
4. Detect bottlenecks and get recommendations
5. Implement optimizations based on recommendations
6. Re-profile to verify improvements

Quick Profiling (Timing Only)

1. Run timing analyzer on simulation log
2. Identify dominant phases (>50% of runtime)
3. Apply targeted optimizations to dominant phases

CLI Examples

Timing Analysis

# Basic timing analysis
python3 scripts/timing_analyzer.py \
    --log simulation.log \
    --json

# Custom timing pattern
python3 scripts/timing_analyzer.py \
    --log simulation.log \
    --pattern 'Step\s+(\w+)\s+took\s+([\d.]+)s' \
    --json

Scaling Analysis

# Strong scaling (fixed problem size)
python3 scripts/scaling_analyzer.py \
    --data scaling_data.json \
    --type strong \
    --json

# Weak scaling (constant work per processor)
python3 scripts/scaling_analyzer.py \
    --data scaling_data.json \
    --type weak \
    --json

Memory Profiling

# Estimate memory requirements
python3 scripts/memory_profiler.py \
    --params simulation_params.json \
    --available-gb 16.0 \
    --json

Bottleneck Detection

# Detect bottlenecks from timing only
python3 scripts/bottleneck_detector.py \
    --timing timing_results.json \
    --json

# Comprehensive analysis with all inputs
python3 scripts/bottleneck_detector.py \
    --timing timing_results.json \
    --scaling scaling_results.json \
    --memory memory_results.json \
    --json

Conversational Workflow Example

User: My simulation is taking too long. Can you help me identify what's slow?

Agent workflow:

1. Ask for simulation log file
2. Run timing analyzer:

   python3 scripts/timing_analyzer.py --log simulation.log --json
   

3. Interpret results (the detector flags solver/assembly phases above 50% and I/O phases above 30%; >70% is high severity):
   • If solver dominates (>50%, high above 70%): Recommend preconditioner tuning
   • If assembly dominates (>50%): Recommend caching or vectorization
   • If I/O dominates (>30%): Recommend reducing output frequency
4. If user has multi-run data, analyze scaling:

   python3 scripts/scaling_analyzer.py --data scaling.json --type strong --json
   

5. Generate comprehensive recommendations:

   python3 scripts/bottleneck_detector.py --timing timing.json --scaling scaling.json --json
   

Interpretation Guidance

Timing Analysis

The detector applies per-type dominance thresholds: solver/assembly/general phases are flagged above 50% of runtime; I/O phases above 30%. Any flagged phase above 70% is reported as high severity.

Scenario	Meaning	Action
Solver >50% (high >70%)	Solver-dominated	Tune preconditioner, check tolerance
Assembly >50%	Assembly-dominated	Cache matrices, vectorize, parallelize
I/O >30%	I/O-dominated	Reduce frequency, use parallel I/O
Balanced (below thresholds)	Well-balanced	Look for algorithmic improvements

Scaling Analysis

Efficiency	Meaning	Action
>0.80	Excellent scaling	Continue scaling up
0.70-0.80	Good scaling	Monitor at larger scales
0.50-0.70	Poor scaling	Investigate communication/load balance
<0.50	Very poor scaling	Reduce processor count or redesign

Memory Profile

Usage	Meaning	Action
<60% available	Safe	No action needed
60-80% available	Moderate	Monitor, consider optimization
>80% available	High	Reduce resolution or increase processors
>100% available	Exceeds capacity	Must reduce problem size

The estimate follows the three-term formula Total = Field + Solver Workspace + Matrix Storage (see references/profiling_guide.md). Matrix storage and solver workspace depend on solver.type:

• iterative (default): sparse matrix (default 7-point stencil, override via solver.stencil_nnz) plus workspace vectors.
• direct: sparse matrix scaled by a conservative fill-in factor (solver.fillin_factor, default 10) to reflect factorization fill-in — a direct solver estimates far more memory than an iterative one for the same mesh.
• matrix-free: no assembled matrix; workspace vectors only.

The estimate is intentionally conservative so a "will it fit in RAM?" decision does not silently under-estimate.

Error Handling

Error	Cause	Resolution
Log file not found	Invalid path	Verify log file path
No timing data found	Pattern mismatch	Provide custom pattern with --pattern
At least 2 runs required	Insufficient data	Provide more scaling runs
Missing required parameters	Incomplete params	Add mesh and fields to params file

Optimization Strategies by Bottleneck Type

Solver Bottlenecks

• Use algebraic multigrid (AMG) preconditioner
• Tighten solver tolerance if over-solving
• Consider direct solver for small problems
• Profile matrix assembly vs solve time

Assembly Bottlenecks

• Cache element matrices if geometry is static
• Use vectorized assembly routines
• Consider matrix-free methods
• Parallelize assembly with coloring

I/O Bottlenecks

• Reduce output frequency
• Use parallel I/O (HDF5, MPI-IO)
• Write to fast scratch storage
• Compress output data

Scaling Bottlenecks

• Investigate communication overhead
• Check for load imbalance
• Reduce synchronization points
• Use asynchronous communication
• Consider hybrid MPI+OpenMP

Memory Bottlenecks

• Reduce mesh resolution
• Use iterative solver (lower memory than direct)
• Enable out-of-core computation
• Increase number of processors
• Use single precision where appropriate

Verification checklist

Before trusting a profiling result or acting on a recommendation, record the concrete evidence below:

• Confirmed timing_analyzer.py actually matched entries — results.phases is non-empty and results.total_time > 0; if a custom --pattern was used and results.message/suggested_patterns appeared, the pattern was fixed and re-run (an empty phases list silently looks like a fast simulation).
• Cross-checked that the sum of phases[].percentage is ~100% and that named phases cover the wall-clock time — unaccounted-for time means missing log lines, not a balanced run.
• For scaling claims, used >=2 runs spanning a real processor range and recorded results.average_efficiency and results.efficiency_threshold_processors from scaling_analyzer.py; verified the --type (strong vs weak) matches how the runs were generated (fixed total size vs fixed work-per-rank).
• Recorded the memory breakdown from memory_profiler.py (field_memory_gb, solver_workspace_gb, matrix_storage_gb, total_memory_gb) and confirmed solver.type (iterative / direct / matrix-free) matches the real solver — a direct solve carries the ~10x fill-in factor and a wrong type makes the "fits in RAM?" answer unsafe.
• Checked results.warnings and compared total_memory_gb (and per_process_gb) against the actual --available-gb; treated >80% as the documented "high" band, not a pass.
• For each bottleneck_detector.py recommendation, confirmed the driving bottleneck (its category, value, and threshold) is consistent with the timing/scaling/memory inputs that were actually supplied — recommendations only reflect the JSON files passed via --timing/--scaling/--memory.
• After implementing an optimization, re-ran the relevant analyzer and recorded the before/after value to confirm the bottleneck actually moved (re-profile step of the workflow).

Common pitfalls & rationalizations

Tempting shortcut	Why it's wrong / what to do
"timing_analyzer.py returned no bottlenecks, so the run is balanced."	An empty/low result is often a pattern mismatch — phases may be empty or partial. Verify total_time matches wall-clock and that phase percentages sum to ~100% before concluding "balanced".
"Two runs scaled fine, so it scales."	Two points only give an average efficiency; they cannot reveal where efficiency falls off. Add more processor counts and check efficiency_threshold_processors, and confirm you used the correct --type (strong vs weak).
"Iterative vs direct is just a flag; memory is about the same."	memory_profiler.py applies a conservative ~10x fill-in factor for direct and stores no matrix for matrix-free. Setting the wrong solver.type can under-estimate RAM by an order of magnitude — set it to the real solver.
"It fits in --available-gb total, so we're fine."	The relevant number for an MPI run is per_process_gb against per-node/per-rank RAM, and >80% of total already triggers a warning. Check the per-process figure and the warnings list, not just the total.
"I/O is under 50%, so I/O isn't the bottleneck."	I/O is flagged at the lower 30% threshold, not 50%. A 30-50% I/O phase is a real bottleneck the detector reports — reduce output frequency or use parallel I/O.
"The recommendation says tune the preconditioner, so the solver is the problem."	Recommendations are only as complete as the JSON you passed in. If --scaling/--memory were omitted, those bottlenecks are simply invisible — feed all available analyses before trusting the priority ranking.

Security

Input Validation

• User-supplied --pattern regex values are validated for length (500 chars max) and rejected if they contain constructs prone to catastrophic backtracking (ReDoS)
• Scaling data entries are validated for finite time values, integer processor counts, and bounded run count (10,000 max)
• available_gb is validated as a positive finite number; mesh dimensions and field parameters are validated as positive integers
• --type (scaling type) is validated against a fixed allowlist (strong, weak)
• All loaded JSON files must have an object (dict) as root element

File Access

• timing_analyzer.py reads a single log file specified by --log; log files are capped at 500 MB and rejected before parsing
• scaling_analyzer.py, memory_profiler.py, and bottleneck_detector.py read JSON files capped at 100 MB
• Phase names extracted from log files are truncated to 200 characters and stripped of control characters to prevent prompt-injection payloads from propagating into agent context
• No scripts write to the filesystem; all output goes to stdout

Tool Restrictions

• Read: Used to inspect script source, references, simulation logs, and result files
• Write: Used to save profiling reports or optimization recommendations; writes are scoped to the user's working directory
• Grep/Glob: Used to locate log files, result files, and search references
• The skill's allowed-tools excludes Bash to prevent the agent from executing arbitrary commands when processing untrusted simulation logs or result files

Safety Measures

• No eval(), exec(), or dynamic code generation
• All subprocess calls use explicit argument lists (no shell=True)
• Reduced tool surface (no Bash) limits the agent to read/write operations only
• Phase names and diagnostic strings are sanitized before inclusion in output to prevent injection

Limitations

• Log parsing: Depends on pattern matching; may miss unusual formats
• Scaling analysis: Requires at least 2 runs for meaningful results
• Memory estimation: Approximate; actual usage may vary
• Recommendations: General guidance; may need domain-specific tuning

References

• references/profiling_guide.md - Profiling concepts and interpretation
• references/optimization_strategies.md - Detailed optimization approaches

Version History

See CHANGELOG.md for the authoritative, dated release history.