IFM

/

K2-Horizon-7B

Like 266

Follow

Institute of Foundation Models 1.53k

I wish you delivered what you say you delivered
This is hot garbage.

Could you expand with few words on the hot garbage and missing delivery? What are your findings?

Hi, my friends,

I am from the IFM team. Could you let me know what specifics are you looking for?

Btw, our team do find that some of our artifacts are not finished uploading yet (e.g., we mess up on some ckpt uploading due to simple index confusion). Here is our rough follow-up plan:

• code will be ready soon, some cleanup on branches will be needed.
• checkpoints are a bit messy, they are there as git references but we need to provide a table explaining it better
• data are halfway uploaded.

The full paper is probably gonna take longer for us to polish.

Excellent. Good to know! Thank you.
Literal garbage was showing up in the results, and I wasn't the only one. Benchmarks were not even coming close in most areas.
If you can straighten this out to what you have claimed, this will be an amazing piece of work.

this is some hard core benchmaxxing

It is currently very hard to get this model actually running locally. I gave up trying to figure out how to compile the llama.cpp fork etc. and just used another model which benchmarked 2x worse than K2-Horizon-7B while having more parameters. It would be nice if the setup process was elaborated on more because I would love to use this model.

Could you please make a commit with this model architecture into llama.cpp.
I am getting Failed to load model: The installed llama.cpp does not recognise this GGUF's model architecture ('k2-horizon'). The file is valid. If the model is newer than this llama.cpp build, updating llama.cpp may add support for it.

Got it my friends. We have llama.cpp ports but haven't merged them yet. Will follow up. @mvillmow-mbzuai @lmlmcat

@inikishev Can you explain more what your issue is with the llama.cpp fork? We are working with the llama.cpp team to get the code integrated here: https://github.com/ggml-org/llama.cpp/discussions/28308

I have also put up a PR against lemonade-server.ai here: https://github.com/lemonade-sdk/lemonade/pull/3530

It shows how to build the fork and install it with lemonade-server. The commands from the PR are:

git clone --branch model/K2Horizon --single-branch https://github.com/MBZUAI-IFM/llama.cpp.git llama.cpp-ifm
cmake -S llama.cpp-ifm -B llama.cpp-ifm/build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build llama.cpp-ifm/build --config Release --target llama-server -j

If you don't have a Mac, then you will need to set the correct GGML_ flags instead of GGML_METAL=ON from the documentation found here: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#blas-build.

Any status on this model lately? Especially with the llama.cpp fork? Or has this model been forgotten about? It's too bad, because it's losing out on its potential popularity.

Hi Nez! Thank you for using our models!

We are working on this. This has been in the works
https://github.com/ggml-org/llama.cpp/pull/29535

and was just merged!

I usually test models with this prompt:

Build a minimal static-file web server in Rust (an nginx substitute for serving sites). Support multiple virtual hosts, each mapping a domain name to a local root directory.
Load them from TOML config. Match on the HTTP Host header (case-insensitive; strip ports and [ ] IPv6 brackets). Serve exact files efficiently using actix-files' NamedFile. For
directory requests, serve <root>/index.html; otherwise return 404. No directory listings or symlinks needed for v1. Don't forget about security. All requests are routed through a
single catch-all handler (default_service). No need for TLS in v1. Test it using curl.exe with --path-as-is parameter.

And Tiel-Coder, for example, completes the task in 12-15 minutes.

This model is too stupid, I've stopped it's test after an hour. About a month I've waited for llama.cpp to support it, and got this disappointment 😢

Thanks for the heads-up. Interesting, LM Studio was on the ball and pushed their updates today, already. I have successfully loaded K2-Horizon-7B for the first time and have successfully tested it with a few simple prompts like "Hi. Tell me about yourself", "How well can you code", etc. All was fine.
Then I started a new session and asked it to build a one-page aquarium simulator.
it started at 29 t/s and quickly degraded to 10 t/s at 2222 tokens, 7 t/s at 3700 tokens, 6 t/s at 4300 tokens, etc

This makes we wonder it there's a caching issue that's accidentally building up geometrically instead of reusing/appending to the cache. I've seen it before with other new models, so I'm just throwing this clue out there as things start slowing down immediately.
I have 8gb VRAM, using LM Studio, KV unified cache, q8_0 kv cache, 55k context, 2 context checkpoints, eval batch size 1024, Physical Batch Size 512.

This is so unfair to say. 99% of the time the issue is with runtime - just because you’re not component enough to work on getting it running doesn’t mean something is “garbage”.

Running it at q8 equivalent version results in starting out at ~80token/s on M5 max but drops to ~40token/s after 3 turns, same as you described. I also know full well this is related to inference

A caching issue wouldn’t be causing that much of a speed. Its literally up to the community or the team or the IFM team to help with runtime, but 99% of the time, even with models like Qwen and GLM, the speeds come from the community.

IFM, great job. Compared to Spark x2.5 4b, this 7b model excels at document editing and math/logic related to finance. The reasoning efforts seem to be not working and only works at high (fails to close when using med or low) but I do see that you guys don’t recommend using any of the lower reasoning efforts - the model does tend to reason efficiently but I do wish there was a better way to control its reasoning effort

Great intuitive observations, dealigni. I can tell you read all the posts here. Anyway, you are completely wrong on your "It can't be caching". It IS a caching problem 100% at the moment; and up until recently it didnt work very well with llama.cpp. IFM has also had problems with outputting literal garbage characters. IFM has been working on it and I appreciate that, so if you don't mind, sit down until you know what you're talking about.
Now, in regards to spark x2.5 4b, I helped spark discover their caching problem; almost the same exact problem was happening to them. The cache was geometrically growing because of how the cache was incorrectly re-added back on top of the old cache, hence a geometric cache growth progression. It's probably due to the code that was added to llama.cpp to handle this model, but I don't know if that's exactly the problem, but it sure seems like it, and I'll let IFM figure out their own code. It's getting better, but it's still not there yet. I'm hoping for the best.

Hi @NezTheNaughty , thanks for the settings you shared earlier. A bit more info will help us narrow this issue down:

• the exact GGUF file you’re running
• your GPU model and LM Studio version
• the LM Studio developer log from loading the model and running a prompt where the slowdown happens