Loading...
MoneyControl F&O Market Data Pipeline
- Published
- Oct 1, 2024
- Technologies
- PythonFastAPIWeb ScrapingGoogle Cloud StoragePandasFinancial Data
- Quick Access
- Source Code
Production-grade FastAPI service scraping 14 F&O market categories from MoneyControl, processing 10,000+ daily options records, and storing structured datasets in Google Cloud Storage with automated scheduling.
The Challenge
Financial traders and quantitative analysts require daily F&O (Futures & Options) market data across multiple categories for algorithmic trading and research. MoneyControl provides this data on mobile pages, but manual collection is impractical for systematic analysis. No official API exists, and data from 14 different market statistics pages (top gainers/losers, most active, OI changes) needs to be aggregated, cleaned, and stored in a queryable format for historical analysis and backtesting strategies.
Solution & Approach
- FastAPI Microservice Architecture: Built production REST API using FastAPI with a single `/run_agent` endpoint that orchestrates the entire scraping workflow. Service runs on Uvicorn with configurable GCS bucket (money_control_daily_data) and base prefix (fno-daily) via environment variables for deployment flexibility
- 14-Category Scraper with tabs.py: Implemented modular scraper covering all MoneyControl F&O mobile URLs - top gainers/losers, most active (shares/value), active calls/puts, OI increase/decrease, highest/lowest OI, and 4 OI-price combination patterns (inc/inc, inc/dec, dec/dec, dec/inc) for comprehensive market coverage
- Pandas HTML Parser with BeautifulSoup Backend: Used pd.read_html() with BeautifulSoup4 + html5lib (avoiding lxml native dependencies) to parse tables. Implemented smart column mapping to handle non-breaking spaces (Last\xa0Price → Last Price) and automatic selection of widest table when multiple tables exist on page
- Robust Data Cleaning Pipeline: Built _clean_columns() and _coerce_numeric() functions handling Indian number formats - removing commas from lakhs (1,23,456), extracting percentages (7.65% → 7.65), and using regex to parse mixed text-numeric values across 10 numeric columns (Last Price, Change, %Chg, High, Low, OI, OI Change, Volume, Contracts, Value)
- GCS Date-Partitioned Storage: Implemented IST-aware date partitioning (UTC + 5:30) creating folder structure gs://bucket/fno-daily/YYYY-MM-DD/*.csv with 14 CSV files per trading day. Used BytesIO for efficient in-memory CSV upload without disk I/O, enabling seamless BigQuery external table integration
- Production Reliability: Added configurable politeness delays (0.7-1.4s random uniform) to respect server resources, proper timeout tuples (10s connect, 20s read), error isolation per category (one failure doesn't stop others), and per-tab error logging with DataFrames containing error messages for monitoring
Tech Stack & Tools
Python 3.8+FastAPI 0.115.0Uvicorn (with standard extras)Pandas 2.2.2BeautifulSoup4 4.12.3html5lib 1.1Requests 2.32.3Google Cloud Storage SDK 2.18.2
Impact & Results
14 scraped
F&O Categories
10,000+ records
Daily Options
~25 seconds
Scraping Time
14 columns
Data Fields
Date-partitioned
Storage Structure
Isolated by tab
Error Handling
Project Visuals
MoneyControl web scraping architecture
F&O data processing pipeline
Google Cloud Storage organization
FastAPI service architecture
Related Projects
Flipkart Product Data Scraper & Analyzer
Web scraping pipeline built with BeautifulSoup and Pandas to extract product listings from Flipkart search results, collecting names, prices, ratings, and review counts, with automated data cleaning and CSV export for e-commerce market analysis.
PythonWeb ScrapingBeautifulSoup
V-Mart Retail Store Data Extraction
Automated web scraping pipeline extracting location data for 465 V-Mart retail stores across India. Collects store names, addresses, operating hours, geocoordinates, and contact information with BeautifulSoup-based intelligent parsing and CSV export.
PythonWeb ScrapingBeautifulSoup
B2B Search Performance Optimization
Achieved 740% growth in Google Search impressions through comprehensive SEO optimization, content restructuring, and data-driven analytics. Implemented systematic metadata improvements and Looker dashboards for continuous performance monitoring.
SEOPythonWeb Scraping