ICP - Inductively Coupled Plasma

Inductively Coupled Plasma Optical Emission Spectroscopy (ICP-OES) is used to measure the elemental composition of solid materials. It provides quantitative elemental analysis expressed in ppm or weight percent.

Method Overview

Property	Value
Full Name	Inductively Coupled Plasma Optical Emission Spectroscopy (ICP-OES)
Purpose	Measures elemental composition of solid materials
Output Units	ppm or % (mass/mass)
SharePoint Location	Analytical Data > ICP
File Format	CSV (raw), XLSX (compiled)

Elements Measured

Element	Symbol	Description
Calcium	Ca	Major element in calcium silicates
Magnesium	Mg	Major element in magnesium silicates
Aluminum	Al	Aluminum content
Iron	Fe	Iron content
Sodium	Na	Sodium content
Potassium	K	Potassium content
Silicon	Si	Silicon content
Titanium	Ti	Titanium content
Manganese	Mn	Manganese content
Phosphorus	P	Phosphorus content
Boron	B	Boron content
Chromium	Cr	Chromium content
Copper	Cu	Copper content
Gallium	Ga	Gallium content
Nickel	Ni	Nickel content
Vanadium	V	Vanadium content
Zinc	Zn	Zinc content

Detection Methods

Method	Description
Axial	Viewing plasma horizontally - higher sensitivity
Radial	Viewing plasma from the side - better for high concentrations

Data Pipeline

SharePoint Sources

Type	Site	Folders	Files
Raw CSV	Analytical Data	ICP, ICP DATA, CSV Files	.csv
Compiled	Analytical Data	ICP, ICP DATA, Compiled Files	_compiled.xlsx
XLS	Analytical Data	ICP, ICP DATA, XLS Files	.xls

Dagster Assets

The ICP data pipeline consists of the following assets in apps/datasmart/src/datasmart/assets/analytical/icp.py:

icp_raw_incremental (sharepoint_multi_asset)
├── icp_samples (analytical.icp_samples)
└── icp_raw_data (analytical.icp_raw_data)

icp_compiled_incremental (sharepoint_multi_asset)
├── icp_compiled_metadata (backend.icp_compiled_metadata)
└── icp_compiled_data (backend.icp_compiled_data)
          ↓
    icp_compiled (analytical.icp_compiled)

Asset Descriptions

Asset	Schema	Description
icp_samples	analytical	Sample metadata from raw CSV files
icp_raw_data	analytical	Raw measurement values with wavelength and detection method
icp_compiled_metadata	backend	Metadata from compiled Excel files
icp_compiled_data	backend	Melted element values (long format)
icp_compiled	analytical	Wide format, 1 row per sample with element columns

Database Tables

analytical.icp_samples

Sample metadata extracted from raw CSV files.

Column	Type	Description
file_id	string	SharePoint file identifier
sample_number	integer	Row number within file
total_dilution_factor	float	Dilution applied to sample
original_sample_id	string	Raw sample ID from file
sample_id	string	Normalized sample ID
exp_id	string	Experiment-level ID
time_created	datetime	File creation time
sp_site	string	SharePoint site name
file_loc	string	File path in SharePoint

analytical.icp_raw_data

Raw measurement data with element, wavelength, and detection method.

Column	Type	Description
file_id	string	SharePoint file identifier
sample_number	integer	Row number within file
original_sample_id	string	Raw sample ID from file
type	string	Measurement type
export_group	string	Export grouping
element	string	Element symbol (Ca, Mg, etc.)
wavelength	float	Emission wavelength in nm
value	float	Measured concentration
unit	string	ppm or %
detection_method	string	Axial or Radial

analytical.icp_compiled

Wide format table with one row per sample.

Column	Type	Description
sample_id	string	Normalized sample ID
dilution	integer	Dilution factor extracted from sample name
original_sample_id	string	Raw sample ID
Ca	float	Calcium concentration
Mg	float	Magnesium concentration
Al	float	Aluminum concentration
Fe	float	Iron concentration
Na	float	Sodium concentration
K	float	Potassium concentration
Si	float	Silicon concentration
Ti	float	Titanium concentration
Mn	float	Manganese concentration
P	float	Phosphorus concentration
B	float	Boron concentration
Cr	float	Chromium concentration
Cu	float	Copper concentration
Ga	float	Gallium concentration
Ni	float	Nickel concentration
V	float	Vanadium concentration
Zn	float	Zinc concentration
TotalDF	float	Total dilution factor
file_id	string	SharePoint file identifier
file_loc	string	SharePoint file path
time_created	datetime	File creation time

File Format

Raw CSV Files

Raw ICP CSV files have a multi-header structure with 4 header rows:

1. Type - Measurement type
2. Export Group - Grouping information
3. Analyte - Element and wavelength (e.g., "Ca 317.933 (Axial)")
4. Unit - ppm or %

The analyte header encodes three pieces of information:

• Element symbol (e.g., Ca)
• Emission wavelength in nm (e.g., 317.933)
• Detection method (Axial or Radial)

Compiled Excel Files

Compiled files (*_compiled.xlsx) contain averaged results with:

• calculations sheet (ALP method) or Sheet1 (typical methods)
• Columns for each element
• Sample name in first column

File Filtering

Files are skipped if they match these patterns:

Path Exclusions

• Contains archive
• Contains calibration curves

Filename Exclusions

• Contains template
• Contains compiled (for raw processing)
• Contains old
• Contains development
• Contains test
• Contains both dont and use

Dilution Factor Handling

Dilution factors appear in two places:

Sample Name Suffix

P800-001-1-QBND-FM-ICP10  ->  dilution = 10

The -ICP[number] suffix is extracted and removed from the sample ID.

Total Dilution Factor Row

A special row in the CSV contains the Total Dilution Factor, extracted as the total_dilution_factor column.

Leach Sample ID Normalization

Special patterns in leaching experiment sample IDs are normalized:

Original Pattern	Normalized
sample FR1 Wash 2	sample-FR1W2
sample FR1 PLS	sample-FR1PLS
sample A1	sample-A1
sample Wash1 or sample Wash 1	sample-W1

Usage Examples

Query Compiled Data

from shared.db.sql import SQL

# Get all compiled ICP data
icp = SQL.read("SELECT * FROM analytical.icp_compiled")

# Filter by project team
icp = SQL.read("""
    SELECT * FROM analytical.icp_compiled
    WHERE sample_id LIKE 'P800%'
""")

Query Raw Data with Wavelength Info

raw_data = SQL.read("""
    SELECT
        s.sample_id, s.exp_id, s.total_dilution_factor,
        r.element, r.wavelength, r.value, r.detection_method
    FROM analytical.icp_samples s
    JOIN analytical.icp_raw_data r
        ON s.file_id = r.file_id AND s.sample_number = r.sample_number
    WHERE s.sample_id IS NOT NULL
""")

Compare Detection Methods

# Compare Axial vs Radial measurements for calcium
detection_comparison = SQL.read("""
    SELECT
        s.sample_id, r.element, r.wavelength, r.detection_method, r.value
    FROM analytical.icp_samples s
    JOIN analytical.icp_raw_data r
        ON s.file_id = r.file_id AND s.sample_number = r.sample_number
    WHERE r.element = 'Ca'
    ORDER BY s.sample_id, r.detection_method
""")

Join with XRF Data

combined = SQL.read("""
    SELECT
        i.sample_id,
        i.Ca as icp_Ca, i.Mg as icp_Mg, i.Si as icp_Si,
        x.CaO, x.MgO, x.SiO2
    FROM analytical.icp_compiled i
    LEFT JOIN analytical.xrf_simplified x
        ON i.sample_id = x.sample_id
    WHERE i.sample_id IS NOT NULL
""")

Sample ID Processing

The pipeline uses the SampleID class to normalize sample identifiers:

from data_infrastructure.sample_id import SampleID

# Normalize sample IDs
df = SampleID.update_sample_id(
    df,
    is_material=True,
    project_team_map=project_team_map,
    crosswalk=sample_id_crosswalk,
    extended=True,
)

# Add experiment ID
df = SampleID.construct_exp_id(df)

Related Assets

Asset	Description
sample_id_crosswalk	Maps messy sample IDs to canonical format
leach_crosswalk	Special mappings for leaching experiment IDs
project_team_map	Maps old process area codes to new project team codes

Troubleshooting

Missing Samples

If a sample is missing from icp_compiled:

1. Check if it exists in raw tables (icp_samples, icp_raw_data)
2. Verify the source file exists in SharePoint
3. Check if the file was filtered out by naming patterns

Sample ID Normalization Issues

If sample IDs aren't normalizing correctly:

1. Check the original_sample_id column for the raw value
2. Review the sample_id_crosswalk table for applicable mappings
3. For leach samples, check the leach_crosswalk table

Dilution Factor Verification

# Check dilution factors for a sample
SQL.read("""
    SELECT original_sample_id, dilution, sample_id
    FROM analytical.icp_compiled
    WHERE sample_id = 'P800-001-1-QBND-FM'
""")