any fault here: void loadshardlayers(MLA_weights w, Config cfg, const char *bin_path,…
- User
any fault here: void load_shard_layers(MLA_weights *w, Config *cfg, const char *bin_path, const char *meta_path) { FILE *fm = fopen(meta_path, "rb"); if (!fm) { perror(meta_path); exit(1); } fseek(fm, 0, SEEK_END); long msize = ftell(fm); fseek(fm, 0, SEEK_SET); char *mtext = malloc(msize + 1); fread(mtext, 1, msize, fm); mtext[msize] = '\0'; fclose(fm);
cJSON *root = cJSON_Parse(mtext); if (!root) { fprintf(stderr, "JSON parse error\n"); exit(1); } FILE *fb = fopen(bin_path, "rb"); if (!fb) { perror(bin_path); exit(1); } int n_tensors = cJSON_GetArraySize(root); for (int i = 0; i < n_tensors; i++) { cJSON *item = cJSON_GetArrayItem(root, i); const char *name = cJSON_GetObjectItem(item, "name")->valuestring; cJSON *shape_arr = cJSON_GetObjectItem(item, "shape"); if (strstr(name, ".weight_scale_inv")) continue; const char *dtype = cJSON_GetObjectItem(item, "dtype")->valuestring; size_t numel = numel_from_shape(shape_arr); int layer = get_layer_index(name); /* Allocate destination pointer depending on key */ float *dst_ptr = NULL; int offset = (layer >= 0) ? layer * numel : 0; if (strstr(name, "model.embed_tokens.weight")) { dst_ptr = w->embed_table; } else if (strstr(name, "input_layernorm.weight") && layer >= 0) { dst_ptr = w->rm_norm_weights_hs + offset; } else if (strstr(name, "self_attn.q_a_layernorm.weight") && layer >= 0) { dst_ptr = w->rm_norm_weights_q+ offset; } else if (strstr(name, "self_attn.q_a_proj.weight") && layer >= 0) { dst_ptr = w->wq_a + offset; } else if (strstr(name, "self_attn.q_b_proj.weight") && layer >= 0) { dst_ptr = w->wq_b + offset; } else if (strstr(name, "self_attn.kv_a_proj_with_mqa.weight") && layer >= 0) { dst_ptr = w->wkv_a + offset; } else if (strstr(name, "self_attn.kv_a_layernorm.weight") && layer >= 0) { dst_ptr = w->rm_norm_weights_kv; } else if (strstr(name, "self_attn.kv_b_proj.weight") && layer >= 0) { dst_ptr = w->wkv_b + offset; } else if (strstr(name, "self_attn.o_proj.weight") && layer >= 0) { dst_ptr = w->wo + offset; } else { /* Skip anything not a projection we care about */ if (strcmp(dtype, "int8") == 0) { if (i + 1 < n_tensors) { cJSON *next = cJSON_GetArrayItem(root, i + 1); cJSON *next_shape= cJSON_GetObjectItem(next, "shape"); cJSON *dim0 = cJSON_GetArrayItem(next_shape, 0); cJSON *dim1 = cJSON_GetArrayItem(next_shape, 1); size_t rows_size = (size_t)dim0->valuedouble; size_t cols_size = (size_t)dim1->valuedouble; fseek(fb, numel * sizeof(int8_t) + (rows_size * cols_size) * sizeof(float), SEEK_CUR); } } else { fseek(fb, numel * sizeof(float), SEEK_CUR); } continue; } if (strcmp(dtype, "float32") == 0) { fread(dst_ptr, sizeof(float), numel, fb); } else if (strcmp(dtype, "int8") == 0) { cJSON *dim0 = cJSON_GetArrayItem(shape_arr, 0); cJSON *dim1 = cJSON_GetArrayItem(shape_arr, 1); size_t rows_size = (size_t)dim0->valuedouble; size_t cols_size = (size_t)dim1->valuedouble; int rows = (int)rows_size; int cols = (int)cols_size; if (!dst_ptr) { fprintf(stderr, "ERROR: dst_ptr is NULL for tensor %s\n", name); exit(1); } printf("row and col for %s: (%d, %d)\n", name, rows, cols); dequantize_and_load_grouped(dst_ptr, fb, cfg, rows, cols); } else { continue; } } Assistant
This
load_shard_layersfunction is quite comprehensive and handles:- Parsing the JSON metadata,
- Allocating/finding the right destination pointer (
dst_ptr) for each tensor, - Skipping unwanted tensors by
fseek-ing past them, - Reading float32 tensors directly,
- Dequantizing int8 tensors using
dequantize_and_load_grouped.
There are no glaring bugs that would cause crashes immediately, but a few subtle gotchas and improvement opportunities that could explain alignment issues, silent corruption, or mismatches after layer 0:
1.
fseekLogic for Skipping TensorsYour skip logic for
int8tensors assumes:fseek(fb, numel * sizeof(int8_t) + (rows_size * cols_size) * sizeof(float), SEEK_CUR);This assumes every
int8tensor is followed by a scale tensor of the same shape (rows × cols floats).Problem:
- If the next tensor isn't a scale (e.g., a float32 tensor, or another int8 tensor without scales), you'll skip wrong bytes and desync the file pointer.
- If the writer stored scales differently (e.g.,
ceil(rows/GROUP_SIZE) * colsfloats), this calculation is wrong.
Fix:
- Instead of hardcoding
rows_size * cols_sizefor scales, parse the next tensor’s dtype/shape explicitly. If next tensor is float32, skipnumel * 4. If int8, skipnumel * 1 + scales_size * 4(computescales_sizefrom its shape).
Example safer skip:
if (strcmp(dtype, "int8") == 0) { if (i + 1 < n_tensors) { cJSON *next_item = cJSON_GetArrayItem(root, i + 1); const char *next_dtype = cJSON_GetObjectItem(next_item, "dtype")->valuestring; cJSON *next_shape = cJSON_GetObjectItem(next_item, "shape"); size_t next_numel = numel_from_shape(next_shape); if (strcmp(next_dtype, "float32") == 0) { // Next tensor is float32: skip (int8 bytes + float32 bytes) fseek(fb, numel * sizeof(int8_t) + next_numel * sizeof(float), SEEK_CUR); } else if (strcmp(next_dtype, "int8") == 0) { // Next tensor is int8: assume it has scales too (be cautious, this might still be wrong) size_t next_rows = (size_t)cJSON_GetArrayItem(next_shape, 0)->valuedouble; size_t next_cols = (size_t)cJSON_GetArrayItem(next_shape, 1)->valuedouble; size_t next_scales = next_rows * ((next_cols + GROUP_SIZE - 1) / GROUP_SIZE); fseek(fb, numel * sizeof(int8_t) + next_numel * sizeof(int8_t) + next_scales * sizeof(float), SEEK_CUR); } } else { // Last tensor: just skip int8 bytes (no scales after it) fseek(fb, numel * sizeof(int8_t), SEEK_CUR); } }2.
cJSON_GetArrayItem(shape_arr, 0/1)->valuedoubleYou're using
valuedoubleto read dimensions.cJSONstores numbers asdoubleinternally, so this works, but:- It's unsafe if dimensions exceed
INT_MAX(unlikely for ML models, but still). - If the JSON accidentally has non-integer dims (e.g., 128.0 vs 128), you might get subtle rounding issues.
Better: Cast via
valueint(or at least round):size_t rows_size = (size_t)round(cJSON_GetArrayItem(shape_arr, 0)->valuedouble); size_t cols_size = (size_t)round(cJSON_GetArrayItem(shape_arr, 1)->valuedouble);Or, if
cJSONsupports it, usevalueintdirectly:int rows = cJSON_GetArrayItem(shape_arr, 0)->valueint; int cols = cJSON_GetArrayItem(shape_arr, 1)->valueint;3. No
ftellDebuggingAfter each
freadorfseek, printftell(fb)to ensure you're at the expected byte offset. This is the fastest way to catch desyncs:fread(dst_ptr, sizeof(float), numel, fb); printf("After reading %s: file offset = %ld\n", name, ftell(fb));or
fseek(fb, ..., SEEK_CUR); printf("After skipping tensor: file offset = %ld\n", ftell(fb));Compare these offsets to the expected positions calculated from the JSON sizes. If
ftellshows you're 4/8 bytes ahead/behind, you’ve misaligned a read.4.
dequantize_and_load_groupedAssumptionsYour
dequantize_and_load_groupedexpects:int8weights, thenfloatscales,- One scale per
GROUP_SIZEcols (common grouped quantization).
Double-check this matches your Python writer:
- If Python wrote
uint8(notint8), you'll get wrong signs. - If scales are stored differently (e.g., transposed, or packed weirdly), your reader will fail.
5. Error Handling in
freadYou have:
fread(dst_ptr, sizeof(float), numel, fb);Never assume
freadsucceeds. Add:size_t got = fread(dst_ptr, sizeof(float), numel, fb); if (got != numel) { fprintf(stderr, "ERROR: fread failed for tensor '%s': read %zu, expected %zu\n", name, got, numel); exit(1); }Same for
dequantize_and_load_grouped: it already checksfreadinternally, but if it fails, your file pointer is toast.6. JSON Memory Leak
You
malloc(mtext)but neverfree(mtext)after parsing JSON. Add:cJSON_Delete(root); free(mtext); // <--- Add this fclose(fb);Minor, but will leak memory if you load multiple shards.
Action Plan
- Sprinkle
ftell(fb)prints after everyfread/fseek. Verify file offsets match expected tensor sizes. - Tighten
fseeklogic as shown above (parse next tensor’s dtype/shape explicitly). - Validate
dequantize_and_load_groupedassumptions:- Dump first 16 bytes of
quant_weightsandscalesin C. Compare to Python-exported raw files. - Ensure
rows * cols * 1 (int8) + rows * ceil(cols/GROUP_SIZE) * 4 (float)matches bytes skipped.
- Dump first 16 bytes of
- Run under ASan/Valgrind to catch accidental OOB writes in `dst_ptr** buffers.
If you still see issues, share:
- The exact JSON metadata for layers 0–3 (name, shape, dtype),
- The first 32 bytes (as hex) of the bin file at those tensor offsets.
That’ll let me reproduce the desync and pinpoint the
fseekordequantizemismatch.- User
how does cJSON does this: what kind of a C syntax is this cJSON_ArrayForEach, is it a lambda? does C support lambda?
Assistant
cJSON_ArrayForEachis a macro, not a lambda or a C standard library feature. It's a clever trick used by thecJSONlibrary to simplify iterating over JSON arrays. Let's break it down:What is
cJSON_ArrayForEach?#define cJSON_ArrayForEach(element, array) \ for(element = (array != NULL) ? (array)->child : NULL; \ element != NULL; \ element = element->next)This is a C preprocessor macro, defined in
cJSON.h. Here's what it does:- Expands into a
forloop: When you writecJSON_ArrayForEach(my_item, my_array), the preprocessor replaces it with the actualforloop code. - Safely handles NULL arrays: If
arrayis NULL,elementis set to NULL, and the loop does nothing (no crashes). - Iterates over
childandnextpointers:cJSONstores arrays as a linked list:array->childpoints to the first item in the array.- Each item's
->nextpoints to the next item in the array.
How does it work? Let's manually expand the macro for clarity. Suppose you write:
cJSON *root = cJSON_Parse("[1, 2, 3]"); cJSON *item; cJSON_ArrayForEach(item, root) { printf("%d\n", item->valueint); }The preprocessor expands
cJSON_ArrayForEachinto:for(item = (root != NULL) ? (root)->child : NULL; item != NULL; item = item->next) { printf("%d\n", item->valueint); }Which is equivalent to:
if (root != NULL && root->child != NULL) { // safety check item = root->child; // start at first child while (item != NULL) { // loop until end of list printf("%d\n", item->valueint); item = item->next; // move to next sibling } }So, what's happening here?
root->child: Points to the first element (1).item->next: Iterates through2, then3.item = item->nexteventually setsitem = NULL(end of list), exiting the loop.
Does C support lambdas? No, not natively.
- C++11 introduced lambdas (e.g.,
[](int x) { return x * 2; }), but C has no equivalent. - C99/C11/C17 standards don't include closures or anonymous functions.
However, C has three workarounds that mimic lambda-like behavior:
- Function pointers + context structs (manual "capture" of variables):
struct LambdaCtx { int multiplier; }; int lambda_like_fn(void* ctx, int x) { struct LambdaCtx* l = (struct LambdaCtx*)ctx; return x * l->multiplier; } // Usage struct LambdaCtx ctx = {.multiplier = 3}; int result = lambda_like_fn(&ctx, 4); // result = 12 - Macros (like
cJSON_ArrayForEach): Codegen tricks to inline behavior. Not real closures, but compile-time expansion. - Blocks (in GCC/Clang extensions): Non-standard, but supported by some compilers:
(int (^my_lambda)(int) = ^(int x) { return x * 2; }; printf("%d\n", my_lambda(3)); // prints 6^syntax is a GCC/Clang extension, not in ISO C).
Why did
cJSONchoose a macro?- Zero overhead: Expands to plain C code; no function call penalty.
- Readability:
cJSON_ArrayForEach(item, array)is way cleaner than writing theforloop manually every time. - No C closures: Since C doesn't support real lambdas, a macro was the lightest/safest trick.
Other libraries (e.g.,
uthash,list.hin Linux kernel) use similar techniques:HASH_FOREACH(uthash)list_for_each_entry(Linux kernel)
These look like built-in language features but are actually preprocessor magic.
Moral of the story:
- C macros can emulate control structures (loops, conditionals) for cleaner APIs.
- If you ever see
XXX_ForEachin a C library, it's probably a macro expanding to aforloop. - For real closures, use C++11+ or languages like Rust/Python/JavaScript.
Now, if you want to implement your own
foreachfor, say, a linked list:#define MYLIST_FOREACH(item, list) \ for (item = (list) ? (list)->head : NULL; \ item; \ item = item->next) // Usage MYLIST_FOREACH(node, my_linked_list) { printf("%s\n", node->data); }Happy C macro hacking!
Extra tip: If you're curious about
cJSONinternals:- Download the source: https://github.com/DaveGamble/cJSON
- Read
cJSON.c— the entire parser is ~1,500 LOC. Elegant, minimalistic C.
- Expands into a
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.