Skip to content
TheChatArchiveThe Chat Archive
Metallama-4-maverick-03-26-experimental· MetaHugging Face Arena Expert 5K

any fault here: void loadshardlayers(MLA_weights w, Config cfg, const char *bin_path,…

  1. User

    any fault here: void load_shard_layers(MLA_weights *w, Config *cfg, const char *bin_path, const char *meta_path) { FILE *fm = fopen(meta_path, "rb"); if (!fm) { perror(meta_path); exit(1); } fseek(fm, 0, SEEK_END); long msize = ftell(fm); fseek(fm, 0, SEEK_SET); char *mtext = malloc(msize + 1); fread(mtext, 1, msize, fm); mtext[msize] = '\0'; fclose(fm);

    cJSON *root = cJSON_Parse(mtext);
    if (!root) { fprintf(stderr, "JSON parse error\n"); exit(1); }
    
    FILE *fb = fopen(bin_path, "rb");
    if (!fb) { perror(bin_path); exit(1); }
    
    int n_tensors = cJSON_GetArraySize(root);
    for (int i = 0; i < n_tensors; i++) {
        cJSON *item      = cJSON_GetArrayItem(root, i);
        const char *name = cJSON_GetObjectItem(item, "name")->valuestring;
        cJSON *shape_arr = cJSON_GetObjectItem(item, "shape");
        if (strstr(name, ".weight_scale_inv")) continue;
    
        const char *dtype = cJSON_GetObjectItem(item, "dtype")->valuestring;
        size_t numel      = numel_from_shape(shape_arr);
        int layer         = get_layer_index(name);
    
        /* Allocate destination pointer depending on key */
        float *dst_ptr = NULL;
        int offset     = (layer >= 0) ? layer * numel : 0;
    
        if (strstr(name, "model.embed_tokens.weight")) {
            dst_ptr = w->embed_table;
        } else if (strstr(name, "input_layernorm.weight") && layer >= 0) {
            dst_ptr = w->rm_norm_weights_hs + offset;
        } else if (strstr(name, "self_attn.q_a_layernorm.weight") && layer >= 0) {
            dst_ptr = w->rm_norm_weights_q+ offset;
        } else if (strstr(name, "self_attn.q_a_proj.weight") && layer >= 0) {
            dst_ptr = w->wq_a + offset;
        } else if (strstr(name, "self_attn.q_b_proj.weight") && layer >= 0) {
            dst_ptr = w->wq_b + offset;
        } else if (strstr(name, "self_attn.kv_a_proj_with_mqa.weight") && layer >= 0) {
            dst_ptr = w->wkv_a + offset;
        } else if (strstr(name, "self_attn.kv_a_layernorm.weight") && layer >= 0) {
            dst_ptr = w->rm_norm_weights_kv;
        } else if (strstr(name, "self_attn.kv_b_proj.weight") && layer >= 0) {
            dst_ptr = w->wkv_b + offset;
        } else if (strstr(name, "self_attn.o_proj.weight") && layer >= 0) {
            dst_ptr = w->wo + offset;
        } else {
            /* Skip anything not a projection we care about */
            if (strcmp(dtype, "int8") == 0) {
                if (i + 1 < n_tensors) {
                    cJSON *next      = cJSON_GetArrayItem(root, i + 1);
                    cJSON *next_shape= cJSON_GetObjectItem(next, "shape");
                    cJSON *dim0      = cJSON_GetArrayItem(next_shape, 0);
                    cJSON *dim1      = cJSON_GetArrayItem(next_shape, 1);
                    size_t rows_size = (size_t)dim0->valuedouble;
                    size_t cols_size = (size_t)dim1->valuedouble;
                    fseek(fb,
                          numel * sizeof(int8_t) +
                          (rows_size * cols_size) * sizeof(float),
                          SEEK_CUR);
                }
            } else {
                fseek(fb, numel * sizeof(float), SEEK_CUR);
            }
            continue;
        }
    
        if (strcmp(dtype, "float32") == 0) {
            fread(dst_ptr, sizeof(float), numel, fb);
        } else if (strcmp(dtype, "int8") == 0) {
            cJSON *dim0 = cJSON_GetArrayItem(shape_arr, 0);
            cJSON *dim1 = cJSON_GetArrayItem(shape_arr, 1);
            size_t rows_size = (size_t)dim0->valuedouble;
            size_t cols_size = (size_t)dim1->valuedouble;
            int rows = (int)rows_size;
            int cols = (int)cols_size;
    
            if (!dst_ptr) {
                fprintf(stderr, "ERROR: dst_ptr is NULL for tensor %s\n", name);
                exit(1);
            }
            printf("row and col for %s: (%d, %d)\n", name, rows, cols);
            dequantize_and_load_grouped(dst_ptr, fb, cfg, rows, cols);
        } else {
            continue;
        }
    }
    
  2. Meta Assistant

    This load_shard_layers function is quite comprehensive and handles:

    • Parsing the JSON metadata,
    • Allocating/finding the right destination pointer (dst_ptr) for each tensor,
    • Skipping unwanted tensors by fseek-ing past them,
    • Reading float32 tensors directly,
    • Dequantizing int8 tensors using dequantize_and_load_grouped.

    There are no glaring bugs that would cause crashes immediately, but a few subtle gotchas and improvement opportunities that could explain alignment issues, silent corruption, or mismatches after layer 0:

    1. fseek Logic for Skipping Tensors

    Your skip logic for int8 tensors assumes:

    fseek(fb, numel * sizeof(int8_t) + (rows_size * cols_size) * sizeof(float), SEEK_CUR);
    

    This assumes every int8 tensor is followed by a scale tensor of the same shape (rows × cols floats).

    Problem:

    • If the next tensor isn't a scale (e.g., a float32 tensor, or another int8 tensor without scales), you'll skip wrong bytes and desync the file pointer.
    • If the writer stored scales differently (e.g., ceil(rows/GROUP_SIZE) * cols floats), this calculation is wrong.

    Fix:

    • Instead of hardcoding rows_size * cols_size for scales, parse the next tensor’s dtype/shape explicitly. If next tensor is float32, skip numel * 4. If int8, skip numel * 1 + scales_size * 4 (compute scales_size from its shape).

    Example safer skip:

    if (strcmp(dtype, "int8") == 0) {
        if (i + 1 < n_tensors) {
            cJSON *next_item = cJSON_GetArrayItem(root, i + 1);
            const char *next_dtype = cJSON_GetObjectItem(next_item, "dtype")->valuestring;
            cJSON *next_shape = cJSON_GetObjectItem(next_item, "shape");
            size_t next_numel = numel_from_shape(next_shape);
            
            if (strcmp(next_dtype, "float32") == 0) {
                // Next tensor is float32: skip (int8 bytes + float32 bytes)
                fseek(fb, numel * sizeof(int8_t) + next_numel * sizeof(float), SEEK_CUR);
            } else if (strcmp(next_dtype, "int8") == 0) {
                // Next tensor is int8: assume it has scales too (be cautious, this might still be wrong)
                size_t next_rows = (size_t)cJSON_GetArrayItem(next_shape, 0)->valuedouble;
                size_t next_cols = (size_t)cJSON_GetArrayItem(next_shape, 1)->valuedouble;
                size_t next_scales = next_rows * ((next_cols + GROUP_SIZE - 1) / GROUP_SIZE);
                fseek(fb, numel * sizeof(int8_t) + next_numel * sizeof(int8_t) + next_scales * sizeof(float), SEEK_CUR);
            }
        } else {
            // Last tensor: just skip int8 bytes (no scales after it)
            fseek(fb, numel * sizeof(int8_t), SEEK_CUR);
        }
    }
    

    2. cJSON_GetArrayItem(shape_arr, 0/1)->valuedouble

    You're using valuedouble to read dimensions. cJSON stores numbers as double internally, so this works, but:

    • It's unsafe if dimensions exceed INT_MAX (unlikely for ML models, but still).
    • If the JSON accidentally has non-integer dims (e.g., 128.0 vs 128), you might get subtle rounding issues.

    Better: Cast via valueint (or at least round):

    size_t rows_size = (size_t)round(cJSON_GetArrayItem(shape_arr, 0)->valuedouble);
    size_t cols_size = (size_t)round(cJSON_GetArrayItem(shape_arr, 1)->valuedouble);
    

    Or, if cJSON supports it, use valueint directly:

    int rows = cJSON_GetArrayItem(shape_arr, 0)->valueint;
    int cols = cJSON_GetArrayItem(shape_arr, 1)->valueint;
    

    3. No ftell Debugging

    After each fread or fseek, print ftell(fb) to ensure you're at the expected byte offset. This is the fastest way to catch desyncs:

    fread(dst_ptr, sizeof(float), numel, fb);
    printf("After reading %s: file offset = %ld\n", name, ftell(fb));
    

    or

    fseek(fb, ..., SEEK_CUR);
    printf("After skipping tensor: file offset = %ld\n", ftell(fb));
    

    Compare these offsets to the expected positions calculated from the JSON sizes. If ftell shows you're 4/8 bytes ahead/behind, you’ve misaligned a read.

    4. dequantize_and_load_grouped Assumptions

    Your dequantize_and_load_grouped expects:

    • int8 weights, then float scales,
    • One scale per GROUP_SIZE cols (common grouped quantization).

    Double-check this matches your Python writer:

    • If Python wrote uint8 (not int8), you'll get wrong signs.
    • If scales are stored differently (e.g., transposed, or packed weirdly), your reader will fail.

    5. Error Handling in fread

    You have:

    fread(dst_ptr, sizeof(float), numel, fb);
    

    Never assume fread succeeds. Add:

    size_t got = fread(dst_ptr, sizeof(float), numel, fb);
    if (got != numel) {
        fprintf(stderr, "ERROR: fread failed for tensor '%s': read %zu, expected %zu\n", name, got, numel);
        exit(1);
    }
    

    Same for dequantize_and_load_grouped: it already checks fread internally, but if it fails, your file pointer is toast.

    6. JSON Memory Leak

    You malloc(mtext) but never free(mtext) after parsing JSON. Add:

    cJSON_Delete(root);
    free(mtext); // <--- Add this
    fclose(fb);
    

    Minor, but will leak memory if you load multiple shards.

    Action Plan

    1. Sprinkle ftell(fb) prints after every fread/fseek. Verify file offsets match expected tensor sizes.
    2. Tighten fseek logic as shown above (parse next tensor’s dtype/shape explicitly).
    3. Validate dequantize_and_load_grouped assumptions:
      • Dump first 16 bytes of quant_weights and scales in C. Compare to Python-exported raw files.
      • Ensure rows * cols * 1 (int8) + rows * ceil(cols/GROUP_SIZE) * 4 (float) matches bytes skipped.
    4. Run under ASan/Valgrind to catch accidental OOB writes in `dst_ptr** buffers.

    If you still see issues, share:

    • The exact JSON metadata for layers 0–3 (name, shape, dtype),
    • The first 32 bytes (as hex) of the bin file at those tensor offsets.

    That’ll let me reproduce the desync and pinpoint the fseek or dequantize mismatch.

  3. User

    how does cJSON does this: what kind of a C syntax is this cJSON_ArrayForEach, is it a lambda? does C support lambda?

  4. Meta Assistant

    cJSON_ArrayForEach is a macro, not a lambda or a C standard library feature. It's a clever trick used by the cJSON library to simplify iterating over JSON arrays. Let's break it down:

    What is cJSON_ArrayForEach?

    #define cJSON_ArrayForEach(element, array) \
        for(element = (array != NULL) ? (array)->child : NULL; \
            element != NULL; \
            element = element->next)
    

    This is a C preprocessor macro, defined in cJSON.h. Here's what it does:

    1. Expands into a for loop: When you write cJSON_ArrayForEach(my_item, my_array), the preprocessor replaces it with the actual for loop code.
    2. Safely handles NULL arrays: If array is NULL, element is set to NULL, and the loop does nothing (no crashes).
    3. Iterates over child and next pointers: cJSON stores arrays as a linked list:
      • array->child points to the first item in the array.
      • Each item's ->next points to the next item in the array.

    How does it work? Let's manually expand the macro for clarity. Suppose you write:

    cJSON *root = cJSON_Parse("[1, 2, 3]");
    cJSON *item;
    cJSON_ArrayForEach(item, root) {
        printf("%d\n", item->valueint);
    }
    

    The preprocessor expands cJSON_ArrayForEach into:

    for(item = (root != NULL) ? (root)->child : NULL; 
        item != NULL; 
        item = item->next) 
    {
        printf("%d\n", item->valueint);
    }
    

    Which is equivalent to:

    if (root != NULL && root->child != NULL) { // safety check
        item = root->child;                    // start at first child
        while (item != NULL) {                 // loop until end of list
            printf("%d\n", item->valueint);
            item = item->next;                 // move to next sibling
        }
    }
    

    So, what's happening here?

    1. root->child: Points to the first element (1).
    2. item->next: Iterates through 2, then 3.
    3. item = item->next eventually sets item = NULL (end of list), exiting the loop.

    Does C support lambdas? No, not natively.

    • C++11 introduced lambdas (e.g., [](int x) { return x * 2; }), but C has no equivalent.
    • C99/C11/C17 standards don't include closures or anonymous functions.

    However, C has three workarounds that mimic lambda-like behavior:

    1. Function pointers + context structs (manual "capture" of variables):
      struct LambdaCtx { int multiplier; };
      int lambda_like_fn(void* ctx, int x) {
          struct LambdaCtx* l = (struct LambdaCtx*)ctx;
          return x * l->multiplier;
      }
      // Usage
      struct LambdaCtx ctx = {.multiplier = 3};
      int result = lambda_like_fn(&ctx, 4); // result = 12
      
    2. Macros (like cJSON_ArrayForEach): Codegen tricks to inline behavior. Not real closures, but compile-time expansion.
    3. Blocks (in GCC/Clang extensions): Non-standard, but supported by some compilers:
      int (^my_lambda)(int) = ^(int x) { return x * 2; };
      printf("%d\n", my_lambda(3)); // prints 6
      
      (^ syntax is a GCC/Clang extension, not in ISO C).

    Why did cJSON choose a macro?

    1. Zero overhead: Expands to plain C code; no function call penalty.
    2. Readability: cJSON_ArrayForEach(item, array) is way cleaner than writing the for loop manually every time.
    3. No C closures: Since C doesn't support real lambdas, a macro was the lightest/safest trick.

    Other libraries (e.g., uthash, list.h in Linux kernel) use similar techniques:

    • HASH_FOREACH (uthash)
    • list_for_each_entry (Linux kernel)

    These look like built-in language features but are actually preprocessor magic.

    Moral of the story:

    • C macros can emulate control structures (loops, conditionals) for cleaner APIs.
    • If you ever see XXX_ForEach in a C library, it's probably a macro expanding to a for loop.
    • For real closures, use C++11+ or languages like Rust/Python/JavaScript.

    Now, if you want to implement your own foreach for, say, a linked list:

    #define MYLIST_FOREACH(item, list) \
        for (item = (list) ? (list)->head : NULL; \
             item; \
             item = item->next)
    
    // Usage
    MYLIST_FOREACH(node, my_linked_list) {
        printf("%s\n", node->data);
    }
    

    Happy C macro hacking!

    Extra tip: If you're curious about cJSON internals:

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.