Model Data Sources & Intelligence Feeds

Open Intelligence Suite $0 Cost • Open Access Multi-Modal Feature Space

The Midgley Forecasting Engine bridges quantitative financial futures with physical supply-chain telemetry, atmospheric hazard sensors, river navigation gages, state motor fuel tax portals, crowdsourced pump benchmarks, and real-time financial news. Explore complete technical specifications, update cadences, endpoint schemas, and caching policies across all 28 monitored feeds.

View Math Guide Citations Portal Live Telemetry & Quotas

Ingestion Feeds

$0.00

Paid API Cost

3-Tier

Failover Resiliency

< 15 min

Fast-Path Polling

Quantitative Commodity Futures & Energy Benchmarks

Exchange-traded financial benchmarks, government energy inventory ledgers, and biofuel credit rack series.

7 Feeds

NYMEX RBOB Gasoline

Ticker: RB=F

Daily Close

Reformulated Blendstock for Oxygenate Blending (RBOB) delivered at New York Harbor (42,000 gal contract). Primary baseline target for national wholesale price discovery.

Variables: Open, High, Low, Close, Volume, Returns

Consumer: src/data_ingestion.py

Provider: Yahoo Finance / CME $0 Open Access

WTI Crude Oil Futures

Ticker: CL=F

Daily Close

West Texas Intermediate light sweet crude physical delivery benchmark at Cushing, Oklahoma hub ($/bbl). Used to calculate raw refining crack margins: P_RBOB − P_WTI / 42.0.

Variables: Settlement ($/bbl), Volumetric ($/gal), Returns

Consumer: src/data_ingestion.py

Provider: NYMEX / CME Group $0 Open Access

Brent Crude Futures

Ticker: BZ=F

Daily Close

North Sea sea-borne crude pricing benchmark. Governs import parity costs for East Coast (PADD 1B Delaware City) and West Coast (PADD 5 CARB) waterborne crude shipments.

Variables: Brent Settlement, Brent-WTI Spread

Consumer: src/data_ingestion.py

Provider: ICE Futures Europe $0 Open Access

NYMEX ULSD / Heating Oil

Ticker: HO=F

Daily Close

Ultra-Low Sulfur Diesel and heating oil benchmark. Critical for computing the industry-standard 3-2-1 refining crack margin: Crack₃₂₁ = (2 × RBOB + ULSD − 3 × WTI) / 3.

Variables: HO Settlement ($/gal), 3-2-1 Crack Margin

Consumer: src/data_ingestion.py

Provider: NYMEX / CME Group $0 Open Access

U.S. EIA API v2 Data

api.eia.gov/v2

Wed 10:30 ET

Official Weekly Petroleum Status Report (WPSR). Ingests regional motor gasoline inventories, crude stocks, PADD refinery capacity utilization %, and weekly retail fuel prices.

Endpoints: /petroleum/pri/gnd/, /petroleum/stoc/

Consumer: src/data_ingestion.py

Provider: U.S. Dept of Energy (EIA) Free API Key

FRED St. Louis Fed

api.stlouisfed.org

Weekly Mon

Federal Reserve Economic Data energy series. Tracks weekly national retail regular gasoline (GASREGW), diesel (GASDESW), and BLS CPI Gasoline index (CUUR0000SETB01).

Series: GASREGW, GASDESW, GASREGWCW

Consumer: src/data_ingestion.py

Provider: Federal Reserve Bank Free API Key

USDA Ethanol & RIN Credits

marsapi.ams.usda.gov

Daily • Issue #141

Spot Midwest E100 ethanol rack prices and EPA Renewable Fuel Standard (RFS) D6 Renewable Identification Number (RIN) credits. Models E10 finished unleaded cost: 0.10 × E100 + 0.90 × RBOB + RIN.

Variables: Spot E100 ($/gal), D6 RIN ($/gal)

Consumer: src/data_ingestion.py

Provider: USDA Agricultural Mkt Service $0 Open API

Alternative Physical, Upstream & Macroeconomic Telemetry

Options implied volatility skew, drilling rig lead indicators, Treasury yields, CFTC hedge fund crowding, and interstate pipeline tariffs.

7 Feeds

Cboe OVX Volatility Index

Ticker: ^OVX

Daily Close

Cboe Crude Oil Volatility Index ("VIX for Oil") measuring 30-day implied options tail-risk volatility from USO options. Spikes indicate geopolitical panic buying and downside hedging skew.

Variables: OVX Index Level, 1-Day & 5-Day ΔOVX

Consumer: src/alternative_data_feeds.py

Provider: Chicago Board Options Exchange $0 Open Access

Baker Hughes Rig Counts

bakerhughes.com

Fri 13:00 ET

Total active US drilling rigs, Permian Basin oil rigs, and gas rigs. Acts as a 3-to-6 month forward upstream supply pipeline indicator into medium-term commodity baseline models.

Variables: Total US Rigs, Oil Rigs, 1-Week ΔRigs

Consumer: src/alternative_data_feeds.py

Provider: Baker Hughes Energy Services $0 Open Feed

Enverus Daily Rig Count

enverus.com/dailyrigcount

Daily • Issues #533, #640

GPS-tracked daily active rig counts across major production basins (Permian, Gulf Coast/Eagle Ford, Williston, Anadarko, Appalachia, DJ). Extracted via Cloudflare Browser Rendering API Quick Actions with zero Firecrawl quota usage.

Variables: Permian, Gulf Coast, Total US, ΔDay, ΔMonth

Consumer: src/enverus_rig_feed.py • workers/enverus_scraper.ts

Provider: Enverus / Drillinginfo $0 Edge Browser

US Treasury & TIPS Yields

^TNX / DGS10 / DFII10

Daily • Issue #66

10-Year Treasury Yield (10Y), 10Y-2Y yield curve slope (ΔY), and 10-Year TIPS real yield. Quantifies macro demand, recession probabilities, and breakeven inflation.

Variables: 10Y Nominal, 10Y-2Y Spread, 10Y Real TIPS

Consumer: src/treasury_yield_feed.py

Provider: U.S. Treasury / St. Louis Fed $0 Open Access

CFTC Commitments of Traders

cftc.gov/dea/futures/

Fri 15:30 ET

Commodity Futures Trading Commission weekly positioning report. Extracts non-commercial speculative net length and 3-year rolling Z-score to diagnose hedge fund crowding.

Variables: Net Spec Length, Spec Z-Score, Commercial Ratio

Consumer: src/alternative_data_feeds.py

Provider: Commodity Futures Trading Comm $0 Open Access

FERC Form 6 Pipeline Tariffs

ferc.gov/industries-data/

Annual • Issue #123

Federal Energy Regulatory Commission interstate pipeline tariff filings. Calibrates physical transport cost baselines for Colonial (Lines 1 & 2), Plantation, and Explorer pipelines.

Variables: Line 1 Tariff ($/bbl), Line 2 Tariff, PADD Offset

Consumer: src/feature_engineering.py

Provider: Federal Energy Regulatory Comm $0 Open Access

Energy Sector Equities

XLE, VLO, MPC, PSX

Daily Close

Equity valuations and intra-day returns for major independent US refiners (Valero, Marathon Petroleum, Phillips 66) and energy ETF (XLE) capturing corporate margin health.

Variables: Stock Returns, Sector Relative Strength, Refining Beta

Consumer: src/energy_equities_feed.py

Provider: Yahoo Finance / NYSE $0 Open Access

U.S. BTS Freight TSI & Truck Tonnage

data.bts.gov • bw6n-ddqk

Monthly • Issue #74

U.S. Bureau of Transportation Statistics Freight Transportation Services Index and Truck Tonnage Index. Serves as a leading physical commercial freight demand proxy for diesel and motor gasoline consumption.

Variables: Freight TSI, Truck Tonnage, Petroleum Transport, MoM Δ%

Consumer: src/bts_transportation.py

Provider: U.S. Bureau of Transportation Statistics $0 Open SODA API

Real-Time Financial Media, Web Scraping & Event Intelligence

Gemini LLM news scoring, Firecrawl web-to-markdown extraction, push webhooks, executive social media, and SEC 8-K outage filings.

6 Feeds

Finlight.me REST API

api.finlight.me/v1/

Trading Hours

Ingests real-time financial energy headlines from tier-1 media (Reuters, Bloomberg, Seeking Alpha, Investing.com). Hard quota ledger enforces 150 call/month limit with offline fallback.

Output: Price Pressure (ΔP), Supply (S), Geo (G)

Consumer: src/finlight_feed.py

Provider: Finlight REST API 150 Call/Mo Cap

Browser Rendering & Firecrawl

workers.cloudflare.com • Issue #640

Edge Headless • $0 Quota

Serverless headless Chromium execution at the edge via Cloudflare Browser Rendering API (@cloudflare/puppeteer / Quick Actions). Scrapes JavaScript-rendered dynamic news, South Coast AQMD Rule 1118 refinery flare logs, and NYSERDA weekly retail fuel surveys, bypassing bot protection while preserving metered Firecrawl quotas (800 calls/mo).

Endpoints: /api/v1/news/scrape-markdown, /api/v1/energy/outages/scaqmd

Consumer: workers/regional_outage_scraper.ts, src/firecrawl_scraper.py

Provider: Cloudflare Browser Rendering / Firecrawl $0 Edge / Secondary Metered

Free Energy RSS Feeds

Google News / NYT / CNBC

15-Min Polling

Zero-cost continuous RSS polling across Google News Energy (when:1d), NYT Energy, and CNBC Commodities with 24-hour timestamp deduplication.

Filter: Regex Gate (|ΔP| ≥ 0.40 or S ≥ 0.50)

Consumer: src/intraday_event_monitor.py

Provider: RSS Syndicate Stream $0 Open RSS

Push Webhook Gateway

POST /api/v1/events/webhook

Real-Time Push

REST push ingestion endpoint accepting breaking alerts from Zapier, TradingView, and IFTTT. Secured by HMAC-SHA256 signature verification and IPASIS Tor/proxy filtering.

Security: HMAC-SHA256, IPASIS client-IP inspection

Consumer: src/api_server.py

Provider: External Webhook Senders Instant Push

Executive Social Feed

Twitter / Truth Social

Intraday

Tracks executive posts regarding OPEC pricing pressure (-1.85% single-day shock) and tariff declarations (+2.10% shock), applying a 1.42x multiplier to weekend closed-market posts.

Impact: 1.42x Monday open gap volatility weighting

Consumer: src/executive_social_feed.py

Provider: Social API Syndication $0 Open Feed

SEC EDGAR 8-K Outages

sec.gov/edgar/

Real-Time

Monitors SEC EDGAR Form 8-K corporate disclosures for unscheduled refinery outages, turnaround delays, pipeline force majeure declarations, and material refining equipment failures across Tier-1/Tier-2 operators. Uses ETag / If-Modified-Since HTTP conditional caching and operational title keyword gating.

Entities: PBF, DINO, MPC, VLO, PSX, XOM, CVX, SU, PARR, DK, KMI, EPD

Cadence: 15-min cron, HTTP 304 ETag Cache, ≤10 req/s

Consumer: src/edgar_8k_monitor.py, workers/intraday_monitor_worker.ts

Provider: U.S. Securities & Exchange Comm $0 Open Data

Atmospheric, Hydrological & Seismic Hazard Telemetry

NOAA NWS weather alerts, SPC convective outlooks, NHC hurricane cones, USGS river stages, USACE locks, and earthquake ShakeMaps.

9 Feeds

NOAA NWS Weather Alerts

api.weather.gov • t.wxs.us

15-Min • 90% Token Sav

Ingests real-time NWS active alerts across 10 regional metro county zones and 6 key refining basins. Integrates t.wxs.us lightweight endpoint saving 90-95% context tokens.

Zones: OKZ060, DEZ001, OHZ077, NCZ044, CAZ508, CAZ041

Consumer: src/noaa_weather.py

Provider: National Weather Service $0 Open API

NOAA SPC Convective Risk

spc.noaa.gov

0-Token Mapping

Deterministic 0-token SPC convective risk mapping (HIGH: 1.0, MDT: 0.8, ENH: 0.6, SLGT: 0.4, MRGL: 0.2) tracking severe tornado, hail, and straight-line wind threats to refineries.

Risk Matrix: Tornado (T), Hail (H), Wind (W)

Consumer: src/noaa_weather.py

Provider: NOAA Storm Prediction Center $0 Open Access

NOAA NHC Tropical Cones

nhc.noaa.gov/gis/

Active Season

National Hurricane Center active storm GIS track, 34kt/64kt wind radii swaths, and 5-day forecast cone feeds. Intersects spatial hazard boundaries with EIA-820 refinery geographic coordinates to calculate capacity-weighted PADD 3 refining exposure scores.

Variables: 34kt/64kt Wind Swaths, Forecast Cones, EIA-820 Refinery Exposure Score

Consumer: src/nhc_hurricane.py

Provider: National Hurricane Center $0 Open GIS

BSEE Gulf Shut-in Reports

bsee.gov

Event-Driven

Bureau of Safety and Environmental Enforcement official tropical storm offshore Gulf of Mexico production shut-in percentages and offshore platform evacuation statistics.

Variables: Shut-in Oil % (bpd), Evacuated Platforms

Consumer: src/bsee_shutins.py

Provider: Dept of the Interior (BSEE) $0 Open Access

USGS Water Data Telemetry

api.waterdata.usgs.gov

15-Min Telemetry

Real-time streamflow (cfs), gage height (ft), and water temperature across the Mississippi (Cairo, Memphis), Ohio (Cincinnati), Arkansas (Tulsa), and Delaware rivers for barge draft modeling.

Sites: 03612500 (Cairo), 07032000 (Memphis), 03255000 (Cin)

Consumer: src/usgs_water_feed.py

Provider: U.S. Geological Survey $0 Open REST

USACE Lock Performance

ndc.iwr.usace.army.mil

Daily

U.S. Army Corps of Engineers Lock Performance Monitoring System (LPMS). Tracks vessel queues, barge draft restrictions, and unscheduled lock chamber closures on the Ohio and MKARNS waterways.

Variables: Lock Stall Delay Hours, Draft Limits

Consumer: src/usace_locks.py

Provider: U.S. Army Corps of Engineers $0 Open Access

USGS Seismic Hazards

earthquake.usgs.gov

Real-Time GeoJSON

Real-time seismic events (≥ M3.0), ShakeMap Peak Ground Acceleration (PGA), and refinery fault proximity for the Hayward Fault (Chevron Richmond), Newport-Inglewood / Palos Verdes Faults (El Segundo / Torrance), and New Madrid Seismic Zone.

Variables: Magnitude (M), Depth (km), Refinery Distance (d)

Consumer: src/usgs_seismic.py

Provider: USGS Earthquake Hazards $0 Open GeoJSON

Air Quality & Flaring Index

PurpleAir • OpenAQ • AirNow

Hourly • Issue #110

Multi-feed air quality sensors monitoring fence-line particulate (PM_2.5) and SO₂ anomalies near refineries (e.g. Richmond, Martinez, El Segundo, Torrance, Carson/Wilmington, Delaware City, Commerce City) to detect unplanned unit flaring trips.

Metrics: Flaring Anomaly Z-Score, Sensor Triangulation

Consumer: src/aqi_feed.py

Provider: EPA AirNow / PurpleAir / OpenAQ $0 Open API

Pirate Weather Reanalysis

api.pirateweather.net

Reanalysis • Issue #442

Dark Sky-compatible historical reanalysis processing NOAA High-Resolution Rapid Refresh (HRRR 3km) and ERA5 data across exact refining hub coordinates for backtesting freeze-off risk and heat stress.

Variables: Freeze Duration (≤32°F), Heat Stress (≥95°F), Wind Gusts

Consumer: src/noaa_weather.py

Provider: Pirate Weather / NOAA HRRR $0 Open Access / Fallback

NASA FIRMS Satellite Telemetry

firms.modaps.eosdis.nasa.gov

VIIRS/MODIS • Issue #453

NASA Fire Information for Resource Management System (FIRMS) active fire and thermal anomaly satellite telemetry (VIIRS NOAA-20/SNPP & MODIS) monitoring refining clusters for flaring upsets and emergency unit shutdowns.

Variables: Thermal Radiance (MW), Brightness (K), Thermal Anomaly Z-Score

Consumer: src/firms_satellite_feed.py • src/knowledge_graph.py

Provider: NASA EOSDIS / FIRMS $0 Free Map Key / Cache Fallback

State Open Data, Tax Portals & Crowdsourced Retail Pump Feeds

Universal 50-State open data connectors, state motor fuel excise tax rates, UST fees, Census commuter surveys, and crowdsourced pump prices.

3 Feeds

50-State Open Data Portals

data.<state>.gov / Socrata

51 Locales • Issue #141

Connects dynamically to state open data portals (Socrata), U.S. Census State Tax API, and FTA indices across all 50 states + DC for statutory excise taxes, UST fees, and inspection fees.

Variables: State Excise ($/gal), Local UST Fee ($/gal)

Consumer: src/state_open_data.py

Provider: State Depts of Revenue / Socrata $0 Open Access

U.S. Census Demographics

api.census.gov

ACS 5-Year • Issue #141

American Community Survey (ACS) 5-year county demographics. Quantifies vehicles per household, commute durations, and median income to calibrate localized retail elasticity.

Variables: Commute Mean Mins, Vehicle Ratio, Income

Consumer: src/census_demographics.py

Provider: U.S. Census Bureau $0 Open API

Live Retail Fuel Feeds

lykmapipo CSV • Cloudflare Browser Run • GasBuddy

Daily CSV • 24h TTL

Primary ingestion via lykmapipo/US-Gas-Prices daily AAA metro and state CSVs (multi-grade zero-cost observations). Secondary fallback to headless Chromium Cloudflare Browser Run edge rendering (Issue #640), hyper-local station query via GasBuddy GraphQL, and EIA benchmark calibration.

Coverage: All 18 calibration metro hubs, state-level aggregates, and out-of-metro rural ZIP routing

Consumer: src/live_fuel_feed.py

Provider: lykmapipo (AAA) / Cloudflare Browser Run / GasBuddy $0 Open Ingestion

Continuous Academic Literature & Developer Catalog Feeds

Weekly automated research paper scans, preprint monitors, and open developer directory indexing.

3 Feeds

arXiv Research Preprints

export.arxiv.org/api/query

Weekly Sat 08:00 CT

Queries arXiv API for newly published preprints across Quantitative Finance (q-fin.PR), Econometrics (econ.EM), and Machine Learning (cs.LG).

Pipeline: Ingests abstracts into Saturday review runner

Consumer: src/arxiv_monitor.py

Provider: Cornell University arXiv $0 Open API

CORE Open-Access Literature

api.core.ac.uk/v3/

Weekly • Issue #53

Integrates CORE API v3 to scan millions of open-access university research papers and academic journal articles covering asymmetric price transmission and rockets & feathers econometrics.

Query: "gasoline price", "crack spread", "refinery margin"

Consumer: src/core_monitor.py

Provider: CORE (Jisc / Open Univ) $0 Open API

Developer Catalogs (10 Indexes)

public-apis, free-for-dev

Weekly • Gemini Scored

Monitors 10 major open-source developer indexes (public-apis, awesome-quant, free-for-dev) for new $0 tools, automatically generating GitHub feature issues for tools scored ≥ 7.0/10.

Policy: 100% zero-cost; explicitly bars paid Apify tools

Consumer: src/catalog_monitor.py

Provider: GitHub Curated Repos $0 Open Repos

Data Ingestion & Governance Ledger Matrix

Master architectural summary table detailing protocols, caching TTLs, update cadences, and rate-limiting safeguards.

Data Ingestion & Governance Ledger Matrix
Feed / Identifier	Domain Category	Provider / Endpoint	Protocol / Auth	Cache TTL	Update Cadence	Cost / Safeguard	Consumer Module
RB=F • NYMEX RBOB	Commodity Futures	CME / Yahoo Finance	HTTPS / Zero Auth	6 Hours	Daily Settlement	$0 Open Access	src/data_ingestion.py
CL=F • WTI Crude	Commodity Futures	Cushing / Yahoo Finance	HTTPS / Zero Auth	6 Hours	Daily Settlement	$0 Open Access	src/data_ingestion.py
HO=F • Heating Oil/ULSD	Commodity Futures	NYMEX / CME Group	HTTPS / Zero Auth	6 Hours	Daily Settlement	$0 Open Access	src/data_ingestion.py
^OVX • Cboe Volatility	Physical & Macro	Cboe Options Exchange	HTTPS / Zero Auth	6 Hours	Daily Close	$0 Open Access	src/alternative_data_feeds.py
BTS TSI • Freight & Trucking	Physical Demand	data.bts.gov/resource/bw6n-ddqk	Open SODA REST	7 Days	Monthly Release	$0 Open Access	src/bts_transportation.py
Finlight.me • Energy News	Financial News	api.finlight.me/v1/	Bearer Key	1 Hour	Trading Hours	150 Call/Mo Cap	src/finlight_feed.py
Browser Rendering • Scraper	Dynamic Headless / Outages	workers.cloudflare.com • api.firecrawl.dev	Bearer Token / Edge	24h D1/Disk Cache	Edge Scheduled / On-Demand	$0 Edge / Secondary Metered	workers/regional_outage_scraper.ts
NOAA NWS • Weather Alerts	Atmospheric Hazard	api.weather.gov • t.wxs.us	User-Agent Auth	15 Minutes	15-Min Polling	$0 Open API	src/noaa_weather.py
USGS Water • River Telemetry	Hydrology Telemetry	api.waterdata.usgs.gov	HTTPS / Zero Auth	15 Minutes	15-Min Polling	$0 Open REST	src/usgs_water_feed.py
50-State Portals • Fuel Taxes	Regulatory Tax	data.<state>.gov / Socrata	Open SODA REST	24 Hours	Daily Sync	$0 Open Data	src/state_open_data.py
arXiv API • Research Preprints	Academic Literature	export.arxiv.org/api/query	HTTPS / Zero Auth	7 Days	Weekly Sat 08:00 CT	$0 Open API	src/arxiv_monitor.py

Model & Architecture Documentation

Learn how these raw data feeds are transformed into 9 regularized econometric feature vectors, continuous shock memories, and ridge regression estimates.

Mathematical Architecture Academic Citations