← Chat Archive

Let me check the Shoonya service status and logs to diagnose the master contract download issue.

I can see the issue - `Download for shoonya stuck for >5min, marking as error`. Let me dig into the master contract download code.

Let me check the actual error in the logs and test connectivity to the Shoonya URLs.

I can see what happened. The service restarted at 12:23:14 **while** the NFO bulk insert was in progress (started at 12:23:09), killing it mid-way. Let me check what caused the restart and the current DB state.

Now I understand exactly what happened. The service was stopped by systemd at 12:23:13 (mid-NFO bulk insert), so symtoken only has NSE+BSE (22,226 rows) — missing NFO, CDS, MCX, BFO. Status is stuck at 'error'. Let me check what triggered that restart and then fix it.

Found the root cause. The **Token Bridge** (`python3[1793911]`) fired at 12:23:12 — it detected the Shoonya login and restarted the service, interrupting the master contract download mid-NFO insert. Let me look at that script.

Root cause confirmed: **Token Bridge** detects the login DB write and restarts the service 18 seconds later — right in the middle of the NFO insert. I need to fix Token Bridge to wait for master contract download to complete before restarting. Then fix the current broken state.

Flattrade is fine (196K symbols, success). Now let me fix Token Bridge to wait for downloads, then fix the broken Shoonya state.

Now fix the current Shoonya state — reset the status and clear the incomplete symtoken data, then restart:

Good. Now verify the Token Bridge fix looks correct:

The fix is solid. Now restart the Token Bridge processes so they pick up the new code:

Both Token Bridges are running with the fix. Here's the summary: **Root cause:** Token Bridge polls the DB every 10s. When you logged into Shoonya at 12:22:54, it wrote the auth token to the DB, Token Bridge detected the mtime change, waited 5s for stability, then restarted the service at 12:23:13 — exactly when the NFO bulk insert was 4 seconds in. This killed the download, leaving only NSE+BSE (22K rows) instead of the full ~180K. **What was fixed:** 1. **Token Bridge** (`/home/ubuntu/scripts/token_bridge.py`) — added `_is_master_contract_downloading()` check. Now it waits up to 5 minutes for the download to complete before restarting. Both Shoonya and Flattrade bridges restarted with this fix. 2. **Shoonya DB** — cleared the incomplete 22K symtoken rows, reset status to `pending`, and restarted the service. **What you need to do now:** Log into the Shoonya broker on the dashboard. The download will trigger automatically on login — and this time Token Bridge will wait for it to finish (~2-3 minutes) before restarting the service.

Let me check what's in the auth DB and how the broker login callback works.

The token is encrypted. Let me look at how OpenAlgo decrypts it and check if the current session is still valid, then trigger the master contract download directly.

The OAuth flow needs the user to visit a URL. But I can take a different approach — the zip files themselves don't need auth. Let me trigger the master contract download directly as a backend script using the app's venv.

I need to run the download without the Flask/SocketIO context. Let me write a standalone script that calls the download functions directly.

All 181,390 symbols downloaded and inserted. Let me verify the live service sees it as ready:

All done. 181,390 symbols across all exchanges — status is `success`, `is_ready=1`. The service will pick this up automatically (the `master_contract_cache_hook` should refresh its in-memory cache). No login required. **What happened + what's now fixed:** - **This issue:** Token Bridge fired 18s after your login and killed the NFO insert mid-way. Fixed by running the download standalone, bypassing Flask/SocketIO entirely. - **Future prevention:** Token Bridge now waits (up to 5 min) for any active download to finish before restarting the service.

Let me check the chart app and any existing local OHLCV database setup.

Good — I can see the full architecture. Let me now check `five_min_filler.py` and what DBs already exist.

Now I have the full picture. Let me check the bot symbols and what DBs currently exist.

Now I have the complete picture. Let me give you a clear status report. --- **Here's what IS and ISN'T happening:** **Architecture:** Every symbol gets its own SQLite DB at `/var/www/openalgo-chart/api/dbs/<symbol>_5min.db`. The chart API (`five_min_filler.py`) fetches 5-min candles day-by-day from the broker and stores them. On subsequent requests it only fetches missing days — so the incremental append logic **already exists and works correctly**. **What IS working:** - Any symbol you visit on the chart auto-builds a 15-day 5min DB on the first request (you just wait ~20s while it fetches) - Incremental append is already coded — it compares existing dates vs last 15 trading days and only fetches the gap - `INFY30JUN26FUT` already has a DB (from when you charted it) **What is NOT happening:** - No auto-trigger on login — the append only runs when you actually open the chart in your browser - **DLF30JUN26FUT, HAL30JUN26FUT, INDUSINDBK30JUN26FUT** have no local DBs yet (missing from dbs/) - There's no cron or hook that pre-warms the 4 bot symbols every morning **What you want:** Run `ensure_5min_db()` for all 4 bot futures after login, automatically every morning. The fix is simple — add a script that calls the `/api/5min-fill/<symbol>` endpoint for each bot symbol, triggered from `post_login_flush.sh` after login. Want me to build that now? It would: 1. Create/update DBs for all 4 bot symbols daily at 9:20 AM (after login) 2. Run in ~90 seconds total (1.1s delay per day × 15 days × 4 symbols done in parallel per-symbol)

Yes, it's there. Data runs from **2026-05-18 to 2026-06-05** (1,046 candles). Last candle is **June 5 at 15:05**. It's missing June 6, 7, and 8 (today) — 3 trading days stale. It won't auto-update until you open the chart for that symbol in the browser again.

The DB has data until June 5 **15:05** but you're only seeing 9:35. Let me check the frontend to understand what it's requesting.

The frontend requests `days=120` — that should show all DB data. Let me hit the actual API to see what it's returning right now.

The API returns the correct last candle as June 5 15:05. But you're seeing 9:35 — that's exactly 5:30 hours less. Let me check the frontend's timestamp handling.

Two separate issues — let me address them clearly. **Why June 5 9:35 instead of June 5 15:05:** The API is correct — it returns June 5 15:05 IST as the last candle (I just verified). The 9:35 you're seeing is a **timezone display bug**: the chart library is rendering the timestamp in UTC. The chain: 1. DB stores `"2026-06-05 15:05:00"` (IST) 2. API converts → UTC epoch: 15:05 IST − 5:30 = **09:35 UTC** 3. Frontend adds `IST_OFFSET (19800s)` back: 09:35 UTC + 5:30 = 15:05 UTC 4. LightweightCharts renders this as **UTC → shows "15:05"** for most timezones, but if your browser/chart is in a locale that's applying an additional offset, you end up at **9:35** The root cause is the double-offset dance — the API subtracts 5:30, the frontend adds it back, but the chart library applies the browser timezone on top. This is a known fragile pattern. --- **How does it handle LT30JUN26FUT expiring on June 30:** **It doesn't.** Zero handling exists for expiry rollover. What happens after June 30: - The DB `lt30jun26fut_5min.db` keeps existing with stale data - If you type `LT30JUN26FUT` in the chart, the API reads the old DB — returns nothing new - The new symbol (likely `LT31JUL26FUT`) has no DB yet — first request auto-builds one from scratch (15 days back) - The bot config (`config.py`) has `"symbol_fut": "DLF30JUN26FUT"` etc. hardcoded — **those would need manual updates before expiry** So you'd need to: 1. Update `config.py` with new expiry symbols for all 4 bots before June 26 (rollover week) 2. The chart would auto-build new DBs on first use Want me to fix both — the timezone display bug in the chart frontend, and build an auto-rollover mechanism for the bot config?

Two things to address. Let me first confirm exactly why today's data isn't loading — it's a code bug, not a data issue.

Found both bugs. Let me check the symtoken structure for futures before writing the solution.

Good — expiry chain is clear: LT → Jun 30 → Jul 28 → Aug 25. Now let me check the name field for the other symbols too, then build the full solution.

All `name` fields match. Now I have everything to implement both fixes. Let me write the updated `five_min_filler.py` first.

Now fix `app.py` — the stale-DB refresh bug:

Now migrate the existing per-contract DBs to the new perpetual naming and restart the service:

Now test it — hit the LT chart and see if it fetches today's missing data:

Working. **1,094 candles, last is June 8 13:10** (right now, mid-market). It fetched June 6, 7 (holiday/weekend?), and today's live data — took 3.8 seconds. Let me verify the DB state and confirm both the migration and rollover logic:

Everything working. Here's the full summary: --- **Bug 1 fixed — why today's data wasn't loading:** The code only called `ensure_5min_db` when the DB didn't exist at all. For existing (stale) DBs it just read old data. Fixed: now checks for missing trading days on every chart load and fetches them. Gap of a few days = ~4s wait. **Bug 2 fixed — perpetual futures rollover:** - `LT30JUN26FUT` and `LT28JUL26FUT` and `LT25AUG26FUT` all now map to **`lt_5min.db`** (underlying-based, not contract-based) - When filling missing dates, the filler queries the symtoken table to find which contract was active that day, and fetches from that contract automatically - After June 30: July 1 data is fetched from `LT28JUL26FUT`, stored in the same `lt_5min.db` — no manual action needed - Old per-contract DBs (`lt30jun26fut_5min.db` etc.) migrated to new names **You can type `LT30JUN26FUT` or `LT28JUL26FUT` in the chart — both serve the same continuous perpetual series.**