Bring Your Own Price Feed

No data provider covers everything. The gaps show up in the issue tracker one holding at a time: Euronext and Italian exchange data (#605), Indian mutual funds (#553), a basket of funds quoted only on a public web page (#595), Italian government bonds, individual corporate bonds, regional ETFs that Yahoo prices in the wrong currency. Different markets, one request: the price exists, Wealthfolio just isn’t looking where it lives.

Chasing each source with its own integration means maintaining a long tail of brittle scrapers forever. The alternative is a single mechanism: let anyone point Wealthfolio at a URL that returns a price and describe how to read it. That is what this feature does. The same mechanism plugs in any market-data API that returns JSON, including the commercial ones behind an auth-header token, with no code and nothing for me to bundle.

The shape of the problem

Most securities that the big providers miss still have a price somewhere on the public internet. Sometimes it is a JSON API. Sometimes it is a webpage with the price in a CSS-selectable span. Sometimes it is a table on a regulator’s site, or a CSV download from a fund manager.

What people lack is not the data. It is a structured way to point Wealthfolio at it.

Every “fetch the price for symbol X” operation has roughly the same shape:

1. Construct a URL with the symbol interpolated in.
2. Make an HTTP request.
3. Parse the response.
4. Extract one or more numbers from a specific location in the parsed structure.
5. Return a price keyed to a date.

Once that is written down, the system designs itself. The custom-provider UI is a form with fields for each step of that pipeline:

• URL template with placeholders that get interpolated at fetch time: {SYMBOL}, {CURRENCY}/{currency}, {ISIN}, {MIC}, plus date placeholders {TODAY}, {FROM}, {TO}, and {DATE:format}.
• Format: JSON, HTML, HTML table, or CSV.
• Extraction path: a JSONPath expression for JSON, a CSS selector for HTML, or a table_idx:col_idx pair for tables. Separate paths for price, date, currency, and optionally open, high, low, and volume.
• Optional: headers (for API keys), a factor and an invert toggle for unit and ratio conversion, a locale for number parsing, and a separate “historical” source with the {FROM} and {TO} date placeholders.

You configure it once per security type. Each provider can hold a “latest” source and a “historical” source, and you attach it to as many assets as you want. Wealthfolio handles the request cadence, the rate limiting, and the circuit breaking when a source goes down.

This is not only for scraping HTML. With the JSON format, the URL template, and the headers field, the same feature is a no-code way to wire up a market-data API that Wealthfolio does not bundle. Put your API key in a request header (a JSON object like {"X-API-Key": "..."}) or as a query parameter in the URL itself, point the price and date paths at the response fields, and you have a working provider. Requests are GET only today. POST with a request body is a requested addition that has not shipped.

A saved source is just a declarative config the scraper interprets at fetch time. Placeholders expand, the response gets parsed by format, the extraction paths pull the numbers out, and the result is normalized into the same Quote struct every built-in provider produces:

The same config is what the live preview runs against, so what you see in the test panel is exactly what the sync will pull.

Why JSONPath and CSS selectors

A small Wealthfolio-specific syntax was on the table, something that handled “go to this field, look up this row.” That would have been a mistake, for two reasons.

First, JSONPath and CSS selectors are already standards. Anyone who has inspected a web page knows CSS selectors. Anyone who has queried a JSON payload has used JSONPath. Picking those means people who already know how to pull values out of data do not have to learn a new syntax.

Second, the tooling already exists. The browser’s DevTools writes a CSS selector for you when you right-click an element. Online JSONPath testers validate your expression against a sample payload. Wealthfolio inherits decades of tooling instead of building its own.

The cost is that JSONPath has a few warts (different implementations disagree on edge cases) and CSS selectors require the source page to be server-rendered. The scraper cannot run JavaScript. For data sites that is usually fine. For single-page apps it is not, and you have to find the underlying API instead.

What was actually hard

The extraction was not the hard part. Three other things were.

Live preview. Configuring a provider blind is miserable. You would write a JSONPath expression, save it, attach it to an asset, wait for the next sync, and find out it did not match. So the configuration UI fetches the URL right there and shows you what came back, with the extracted values highlighted. If a value is not what you expected, you adjust the path and re-test in seconds. It feels like talking to the data instead of writing against it.

Circuit breaking. If a custom source goes down for a day, Wealthfolio should not hammer it every minute. The provider registry in crates/market-data wraps every provider, custom ones included, in a circuit breaker. After a handful of consecutive failures (five by default) the circuit opens and that source is skipped for a cooldown window, then a single “half-open” probe checks whether it has recovered. The asset shows as stale rather than retrying forever. The custom scraper also has its own tight rate limit (30 requests per minute, two at a time), so a misconfigured URL cannot turn into a firehose.

Self-hosters and LAN endpoints. A lot of people run their own scrapers: a Python script behind a reverse proxy, a container on a homelab, something hacked together in a Cloudflare Worker. The stance has been that a custom provider is configured by the user, on their own device, so Wealthfolio should not stop them pointing it at their own network. Private IP ranges are not blocked. If the host resolves and responds, the fetcher runs. The SSRF threat model applies to multi-tenant servers, not a single-user desktop app the operator can already reach in fifty other ways.

What people point it at

The requests that drove this, and the configs people have shared since, are mostly mutual funds and regional listings the big providers ignore.

One concrete example came straight from the tracker (#860). Vanguard exposes a workplace fund price API, but only as a historical range:

https://workplace.vanguard.com/.../fundPrice/{SYMBOL}?startDate={DATE:2026-01-01}&endDate={TODAY}

It returns a list of prices in date order, so the latest price is the last element. That request is the reason JSONPath support grew a [-1] last-element selector ($.body.fundPrice.content[-1].price). The feature and the refinement both came from a user’s actual config.

Other sources people have wired up: open-end mutual funds whose fund manager publishes a daily NAV page, exchange listings that Yahoo quotes in the wrong currency, and self-hosted endpoints where someone runs a small scraper of their own and points Wealthfolio at it over their LAN. None of these needed a line of Wealthfolio code to change. That is the whole point.

What I would add given more time

JavaScript execution. Some sites are pure single-page apps where the server-rendered HTML is useless. Headless-browser scraping would unlock them, but it is a serious engineering investment and a real security surface. For now the workaround is to find the underlying API the app calls (the Network tab in DevTools). Sometimes there is not one, and you are stuck.

A community recipe gallery. Right now, if someone works out a clever provider for, say, European mutual funds, the only way to share it is to paste the config into Discord or an issue. A curated, importable library of provider templates would be much better. The hard question there is who hosts and moderates it, not the technical one.

Why this matters beyond market data

The custom-provider feature is the one extensibility surface in Wealthfolio with a security model I am fully confident in. It makes an unauthenticated HTTP GET to a URL the user typed in, parses the response with a no-code expression, and writes the result into a numeric column. The worst case is “the source returns something weird and the parser returns no number,” which is bounded and recoverable.

Custom providers are the easy half of extensibility: declarative configuration instead of imperative code. Almost everything in Wealthfolio that could be a config file probably should be one. The work ahead is finding the next category that fits this pattern and is not market data.

Try it

If you hold something Wealthfolio does not cover, start with the Custom Market Data Providers guide, which has worked examples for the common cases. If you build a config that works for a source other people are likely to want, drop it in an issue or in Discord. The last few improvements to this feature started exactly there.