So everyone is building LLM trading bots now

For the last few months I keep running into the same thing. Someone at a meetup mentions the bot they have running on a small account. A friend sends me a screenshot of a chart that only goes up. My feed fills with threads about letting a language model manage a portfolio. At some point I got curious enough to stop scrolling and actually look into it, and this post is what I found, written down mostly so I stop repeating it over coffee.

I should say up front that I am an engineer and nowhere near a professional trader. Nothing here is financial advice, and if you take one thing away it should probably be that this is an easy place to lose money quickly.

What most of these bots actually are

Once you look past the screenshots, a surprising number of these projects are a loop. Every few minutes the script grabs some recent prices and maybe a few headlines, pastes them into a prompt, asks the model whether to buy or sell and how much, and sends whatever comes back to an exchange. It is a fun weekend project, and I understand why people build it. It also hands the one part of the system that can wipe out an account to the part that is least reliable at numbers.

The more I read, the more it seemed like the useful version of this idea looks quite different.

What the model is genuinely good for

Language models read well. Company filings, earnings call transcripts, central bank statements and the endless stream of news are all text, and turning that text into something a program can use is where these models clearly beat the older rule-based tools. Asking one whether a press release changes a company's outlook, how nervous the language in a call sounds, or which of the stocks you care about a headline is really about all seem like sensible jobs for it.

Arithmetic, probabilities and consistency are where things get shaky. The same prompt can give different answers from one day to the next, the model has no feel for fees or slippage unless you spell them out, and it will cheerfully suggest a position size that makes no sense for the account it is looking at.

So the design I kept coming back to is simple to say. The model reads and describes, and anything that touches money stays in plain code that behaves the same way every time.

How I would put one together

If I were building this properly, the model would sit in the middle of the pipeline and nowhere near the edges. Market data and text sources would come in on one side. The model would turn the text into a small, fixed set of signals, something like a sentiment score, the kind of event and which tickers it affects, and every one of those outputs would be saved with the prompt and the model version that produced it. A strategy written in ordinary code would decide what it wants to hold based on those signals and the prices. A separate risk check, also ordinary code, would look at every order against limits on position size, total exposure and daily losses, and it would be allowed to say no. Only after that would anything reach the broker.

What I like about splitting it up like this is that each piece can be tested on its own. You can replay last week's news through the model and just look at the signals it produced. You can throw ridiculous orders at the risk check and make sure it refuses them. When something goes wrong, you can tell whether the model misread the news or the strategy made a bad call with a correct reading.

I would also want a single switch that stops all trading, checked before every order, and something that flips it automatically when losses or errors or response times drift outside what looks normal.

The part that surprised me most

This was the thing that changed how I look at every impressive chart I have been sent. When you test a normal trading strategy on data from a few years ago, the strategy has no idea what happened afterwards. When you test a language model on data from a few years ago, the model very likely read about what happened afterwards while it was being trained. It has seen the news, the commentary and in a lot of cases the prices themselves.

That means a backtest of an LLM strategy over a period the model was trained on can look brilliant for a reason that has nothing to do with skill. From what I could find, a lot of the published results in this area fall apart once you only test on dates after the model's training cutoff, and plenty of them end up no better than picking at random. That was a sobering thing to read.

The honest way around it seems to be testing only on data from after the cutoff, which for a recent model is not much history, and then running it on paper with live data for a long while before any real money goes in.

What happened when models traded real money

There was a public experiment last autumn that got a lot of attention. Six of the big models were each given ten thousand dollars to trade crypto futures for a couple of weeks. One finished roughly twenty percent up, a couple lost around forty percent, and one was up about fifty percent at one point before giving most of it back.

It made great headlines, and I do not think it says much about which model is the better trader. A couple of weeks of leveraged crypto mostly measures how much risk each one happened to take in a very jumpy market, and a different fortnight could have flipped the order completely. The lesson I took from it was a different one. A model that controls its own position sizes can burn through a large chunk of an account very quickly, which is exactly why I would keep it away from that decision.

I also came across a few more careful setups where several model instances play different roles, a couple of analysts, someone arguing the bullish case and someone arguing the bearish one, with a risk manager sitting in front of the trades. I like that shape a lot. The results people report for it are still short and run into the same training data problem, so I read them as an interesting design and not much more yet.

If you actually run one

The rest is the unglamorous part. Pin the exact model version, because a quiet update on the provider's side can change your signals overnight. Keep a log of every prompt and response next to the orders they led to, since you will want to understand past decisions later. Keep an eye on API costs if you run a model over a lot of news, and accept that anything needing to react in under a second or two is out of reach.

And if the idea ever grows into managing someone else's money, talk to a lawyer early. In Europe that is regulated territory and it is not something you want to discover late.

Where I landed

After all this digging I am more interested in the idea than when I started. A language model that reads everything and hands clean, structured signals to a strict and fairly boring system seems genuinely useful to me, and that is the version I would build if I ever try this myself.

If you are building something like this and want to compare notes, I would be glad to hear from you.