ISSUE DESCRIPTION orm.migrator.up() rewrites snapshot incorrectly Describe the bug When…
- User
ISSUE DESCRIPTION
orm.migrator.up()rewrites snapshot incorrectlyDescribe the bug
When we run
orm.migrator.up(), the snapshot changes and a bunch of stuff in the snapshot is wrong and we have to revert the changes.Reproduction
https://github.com/zachkirsch/mikro-orm-migrate-snapshot-repro
What driver are you using?
@mikro-orm/sqlite
MikroORM version
7.1.2-dev.10
Node.js version
v24.0.1
Operating system
macOS
Validations
- Read the Contributing Guidelines.
- Read the docs.
- Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
- Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord.
- The provided reproduction is a minimal reproducible example of the bug.
ADDITIONAL ADVICE
Since you are given a git repository, you can use git commands to simplify your work. For example, if you made a mistake and want to revert your changes, you can use
git checkout <file>to restore the file to its original state. You can also reset all changes in the repository usinggit reset --hardcommand. Additionally, you can usegit stashandgit stash popto temporarily save your changes and restore them later. Finally, you can see the changes that you've made (which will be submitted when you callsubmit) usinggit statusorgit diffcommands. However you don't need to usegit addorgit commitbefore submitting your changes. In fact,submitonly submits currently unstaged changes because it usesgit diffto get the changes that need to be submitted.CHECKLIST
Before submitting your solution, please go over the following checklist and make sure you've done everything:
- If an error was reported in the issue description, I have successfully reproduced it.
- If an error was reported in the issue description, I have confirmed that my fix resolves the error.
- I have thought about edge cases and made sure my fix handles them as well.
- I have run existing tests in the repository that might have been affected by the change I made and confirmed that they pass. I want you to list every bullet from this checklist and write your reasoning for why you think you did it or didn't need to.
Repository has been uploaded and your shell is currently at the repository root. Time to solve the issue!
(Current directory: /mikro-orm, current file: none) bash-$
Assistant
Let me start by exploring the repository structure to understand what we're working with. The issue is about
orm.migrator.up()rewriting snapshots incorrectly in MikroORM. Let me first understand the codebase structure and then try to reproduce the issue.Assistant
Let me explore the migrations package since the issue is about
orm.migrator.up()rewriting snapshots incorrectly.Assistant
Now let me look at the
storeCurrentSchemamethod and the schema generator to understand how the snapshot is being written. The issue seems to be related to how the schema is serialized to JSON.Assistant
The key line is at line 205-218 in the
runMigrationsmethod. Whenmigrator.up()runs, it creates a newDatabaseSchemafrom the connection after running migrations, and then stores it. Let me check theDatabaseSchema.createmethod and the serialization.Assistant
I see a
toJSON()method inDatabaseSchema. This is what gets called whenJSON.stringify(schema, null, 2)is used. Let me look at theDatabaseTable.toJSON()method as well:Assistant
Let me look at the tests for migrations to understand the issue better and see if there are existing tests that demonstrate the snapshot behavior:
Assistant
Let me look at the SQLite-specific migrator tests since the issue mentions using
@mikro-orm/sqlite:Assistant
I can see there's already a test
'snapshot serialization is stable across metadata and introspection paths (GH #7607)'that tests the exact behavior. Let me look at the git history to understand recent changes related to this issue:Assistant
Let me look at the recent fix that might be related to the snapshot issue:
Assistant
Let me look at the reproduction repository linked in the issue to understand the exact problem:
Assistant
Let me run the existing tests to see if any fail:
Assistant
The test timed out. Let me look more carefully at the issue. The problem is that
orm.migrator.up()rewrites the snapshot incorrectly. Let me look at what happens when the snapshot is stored after running migrations.Looking at the code in
Migrator.ts, the key issue is inrunMigrations:protected override async runMigrations( method: 'up' | 'down', options?: string | string[] | MigrateOptions, ): Promise<MigrationInfo[]> { const result = await super.runMigrations(method, options); if (result.length > 0 && this.options.snapshot) { const ctx = Utils.isObject<MigrateOptions>(options) ? options.transaction : undefined; const schema = await DatabaseSchema.create( this.em.getConnection(), this.em.getPlatform(), this.config, undefined, // <-- schemaName is undefined here undefined, // <-- schemas is undefined here undefined, undefined, undefined, ctx, ); try { await this.storeCurrentSchema(schema); } catch { // Silently ignore for read-only filesystems (production). } } return result; }The problem is when
DatabaseSchema.create()is called, it's usingundefinedforschemaName, but when reading from the snapshot (getSchemaFromSnapshot()), the schema was created with the config schema. Let me trace this more carefully:Assistant
Now let me look at how the
toJSON()method works. I notice it returnstablesas an array but the issue might be in how the tables are serialized. Let me look more closely at thetoJSONmethod ofDatabaseSchema:Assistant
Now let me check how the
DatabaseTable.toJSON()handles themappedTypefield. I noticed this inDatabaseTable.toJSON():mappedType: Utils.keys(t).find(k => t[k] === c.mappedType.constructor),This is a key line - it looks up the mapped type name. If
c.mappedType.constructordoesn't match any key int, this would returnundefined. Let me explore this more and understand the issue:Assistant
Let me now create a reproduction script to understand what exactly happens when the snapshot is written:
Assistant
Now I understand the type registry better. Let me look at the reproduction linked in the issue description. The issue says the problem is that when
orm.migrator.up()runs, the snapshot changes incorrectly.Let me examine the key difference between how the snapshot is written during
create()vs duringup():- During
create():storeCurrentSchema()is called without a schema argument, so it usesthis.#schemaGenerator.getTargetSchema()which creates the schema from metadata - During
up():DatabaseSchema.create()is called to introspect the database after running migrations
The issue could be in how
DatabaseTable.toJSON()handles things when the table was introspected from the database vs when it was created from metadata.Let me check if there's something special about the
viewsfield in the snapshot:- During
Assistant
Let me check what happens when
viewsis serialized but the target schema from metadata doesn't include views:Assistant
Let me look at the
getSchemaFromSnapshotmethod more carefully:Assistant
I see it now! The
getSchemaFromSnapshotmethod doesn't handlenamespacesproperly when restoring. It does setschema.setNamespaces(new Set(namespaces)), but wait - let me look atstoreCurrentSchema:Assistant
Now I see the issue more clearly. When
storeCurrentSchemais called:- If
schemais provided (like afterup()), it serializes the introspected schema usingJSON.stringify(schema, null, 2)which callstoJSON() - If
schemais not provided (like aftercreate()), it usesthis.#schemaGenerator.getTargetSchema()which builds the schema from metadata
But wait - looking at the
toJSON()method, I seeviewsis always included in the output. Let me trace through what happens with views:- If
Assistant
Let me look at the issue more carefully. I think the issue might be in the
runMigrationsmethod. WhenDatabaseSchema.create()is called, it creates a schema from the database but it might include extra tables that aren't in the metadata (like themikro_orm_migrationstable). Let me check:Assistant
Wait, I see an issue! Looking at line 163:
const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());This creates a schema with a
namebased onschemaName,config.get('schema'), orplatform.getDefaultSchemaName().But in
storeCurrentSchemawhen it's called without a schema parameter:schema ??= this.#schemaGenerator.getTargetSchema();Let me check what
getTargetSchemareturns for the schema name:Assistant
Now I think I understand the problem. Let me trace through the issue more carefully. The key difference between:
- Snapshot from
create()(metadata): UsesDatabaseSchema.fromMetadata(...)which only includes tables defined by entities. - Snapshot from
up()(introspection): UsesDatabaseSchema.create(...)which reads from the database.
The
DatabaseSchema.create()method properly filters out the migrations table, but there could be other differences.Let me look at what might be different. The test
'snapshot serialization is stable across metadata and introspection paths (GH #7607)'expectsafterUpto equalafterCreate. Let me check if there are any issues with how this comparison is done.Let me search for recent changes related to this specific issue:
- Snapshot from
Assistant
Let me look at the most recent snapshot-related fix to understand what might still be broken:
Assistant
Let me look at the test file that was added in that commit to understand the expected behavior:
Assistant
Let me check the current state of the repo more carefully. Looking at the test, it expects
afterUp.toEqual(afterCreate). Let me create a minimal reproduction script to test if this is actually failing:Assistant
Let me look at the entities used in the sqlite tests:
Assistant
Let me write a simple test script to reproduce the issue:
Assistant
The
rmcommand is being called first and removes the directory beforewriteFileis called. Let me fix the test script:Assistant
The test entity is using p.number() but it seems that doesn't exist. Let me check what the correct property definition API is.
Assistant
I see, it's
p.integer()notp.number(). Let me update my test entity:Assistant
The snapshot file name is different. Let me check what the actual snapshot file name is:
Assistant
I see - the snapshot file name uses the dbName. Let me update the script:
Assistant
This is the bug! After running
migrator.up(), the snapshot is rewritten with an emptytablesarray, losing all the table definitions! The blank migration doesn't create any tables (it just runsselect 1), but the snapshot should still contain the tables defined in the entity metadata.Let me investigate the
runMigrationsmethod more closely:Assistant
I found the bug! When
migrator.up()runs, it callsDatabaseSchema.create()which introspects the database. In my test with a blank migration and:memory:database, no tables exist after runningselect 1, so the snapshot becomes empty.But wait, this seems like expected behavior for my test case. Let me create a more realistic test that mimics the real-world scenario from the existing tests:
Assistant
The basic case works. Let me try to reproduce a more complex scenario that might trigger the issue. Let me look at the linked reproduction repo more closely:
Assistant
Let me look at the original test in more detail and see what the actual entities look like in the test suite:
Assistant
Let me look at the specific test that was added in the fix for GH #7607 to understand what scenario triggers the issue. The test uses
initORMSqlite()which loads more complex entities and executes a SQL schema dump. Let me look at the sqlite-schema.sql file and understand what the test expects.I need to run the actual test suite to see if there's a failure.
Assistant
Let me run the specific test that was added for this issue with a timeout:
Assistant
The test passes! So the basic snapshot stability test works. Let me run the other snapshot-related tests:
Assistant
The tests pass. Let me look more carefully at the issue. The reproduction repository mentioned is
https://github.com/zachkirsch/mikro-orm-migrate-snapshot-repro. Let me think about what specific scenario might cause the snapshot to be rewritten incorrectly.Looking at the issue description again, it says the snapshot "changes" and "a bunch of stuff in the snapshot is wrong". Let me look at the
namefield inDatabaseSchema.toJSON():Assistant
Let me look at the
nameproperty being serialized and trace the difference between metadata and introspection paths:Assistant
Let me check how
DatabaseSchema.create()sets the schema name vs howDatabaseSchema.fromMetadata()sets it:Assistant
Now I notice a key difference! In
DatabaseSchema.create()(introspection):const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());In
DatabaseSchema.fromMetadata()(metadata):const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema'));The introspection path uses
platform.getDefaultSchemaName()as a fallback, but the metadata path does not! This could cause thenamefield to differ.But wait, this shouldn't cause major issues for SQLite since
getDefaultSchemaName()typically returns an empty string orundefinedfor SQLite.Let me trace through what happens in
getTargetSchema()which is used bystoreCurrentSchema()when no schema is passed:Assistant
OK so
getTargetSchemaalso usesplatform.getDefaultSchemaName()as a fallback. That's consistent.Let me look at the issue from a different angle. The reproduction mentions MikroORM version
7.1.2-dev.10. Let me check what version this repo is at:Assistant
The version is 7.1.1. Let me look at the changelog for 7.1.2:
Assistant
Let me continue exploring the repository to find the actual issue with
orm.migrator.up()rewriting snapshots incorrectly. The tests pass, so I need to understand what specific scenario triggers the bug. Let me look more carefully at the issue.Assistant
Let me look at how
getAllTablesworks for SQLite and what tables might be included:Assistant
The tests are passing, so the snapshot stability fix from GH #7607 seems to be working. Let me look more closely at the issue to understand what specific scenario causes the problem.
The issue description mentions that when running
orm.migrator.up(), the snapshot changes incorrectly. Let me think about what could be different...Let me look more carefully at what
DatabaseSchema.create()does when called duringrunMigrations. The key difference is:- When
create()is called, the snapshot is written from metadata usinggetTargetSchema() - When
up()is called, the snapshot is written from introspection usingDatabaseSchema.create()
Let me check if there's something specific about how the introspection path works that could cause issues. Let me look at the SQLite schema helper to understand what tables get returned.
- When
Assistant
Let me look more carefully at the issue. The problem might be that when
DatabaseSchema.create()is called, it's loading all tables from the database, but the metadata-based schema only includes tables defined in entities.Wait, I noticed something earlier - when I ran my test with a blank migration (that just runs
select 1), the snapshot afterup()was empty because no tables were created. But that's expected behavior for a blank migration.Let me look at the actual reproduction scenario more carefully. The issue is that the snapshot changes incorrectly when running
up(). This could be:- Extra tables being included
- Missing tables
- Different field values
Let me look at the test that was added for GH #7234 to see what specific fields might differ. </tool_call>
Assistant
The user is waiting for me to continue investigating the issue. Let me continue with my analysis of the snapshot rewriting problem.
Assistant
Looking at the tests, they pass which means the basic snapshot functionality works. Let me think about what could be different in the user's scenario.
The issue says "the snapshot changes and a bunch of stuff in the snapshot is wrong". Let me look more carefully at what
DatabaseSchema.create()does differently fromgetTargetSchema().Key insight: The introspection path (
DatabaseSchema.create()) creates a schema by reading from the database, while the metadata path (getTargetSchema()) creates it from entity definitions.Let me look at the
namefield inDatabaseSchema.toJSON(). When creating from metadata, the schema name comes fromconfig.get('schema')or the platform default. When creating from introspection, it also uses the same logic.Wait, let me look at the
runMigrationsmethod more carefully:const schema = await DatabaseSchema.create( this.em.getConnection(), this.em.getPlatform(), this.config, undefined, // schemaName undefined, // schemas ... );When
schemaNameisundefined,DatabaseSchema.create()will use:schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName()But when
storeCurrentSchema()is called without a schema (duringcreate()), it uses:schema ??= this.#schemaGenerator.getTargetSchema();And
getTargetSchema()uses:schemaName = schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName();So the schema names should be the same...
Let me look at the
toJSON()method again. It includesname,namespaces,tables,views, andnativeEnums.Wait - I see a potential issue! The
toJSON()method returnstables: [...this.#tables].sort(byTable). This means it's serializing theDatabaseTableobjects directly, andJSON.stringifywill calltoJSON()on each table.But when loading from the snapshot with
getSchemaFromSnapshot(), it reconstructs the tables from the JSON data, not from the database. So if there's any difference in how the data is serialized vs deserialized, that could cause issues.Let me look at the specific fields that might differ between metadata and introspection paths...
Actually, wait. Let me look more carefully at the
DatabaseSchema.create()method. When it loads tables from the database, it callsloadInformationSchema()which populates the tables. But when the schema is created from metadata usingDatabaseSchema.fromMetadata(), it only includes tables defined by entities.The key question is: does
DatabaseSchema.create()include any extra tables that aren't in the entity metadata?Looking at the code:
const tables = allTables.filter( t => this.isTableNameAllowed(t.table_name, takeTables, skipTables) && (t.table_name !== migrationsTableName || (t.schema_name && t.schema_name !== migrationsSchemaName)), );It filters out the migrations table, but what about other tables? If the user has tables in the database that aren't defined as entities, they would still be included in the introspected snapshot but not in the metadata snapshot.
But that's expected behavior - the snapshot should reflect what's actually in the database after migrations run.
Hmm, let me look at a different angle. The issue mentions "a bunch of stuff in the snapshot is wrong". What if the issue is with how the
DatabaseSchema.create()is called withctx(transaction context)?Actually, I think I should look at a very specific case. Let me check if there's an issue with how the schema is created when the database schema differs slightly from the metadata schema (e.g., in the
namefield ornativeEnums).Actually, let me look at what happens with the
namefield. InDatabaseSchema.toJSON():name: this.name,When creating from metadata:
getTargetSchema()passesschemaNametoDatabaseSchema.fromMetadata()fromMetadata()createsnew DatabaseSchema(platform, schemaName ?? config.get('schema'))
When creating from introspection:
DatabaseSchema.create()createsnew DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName())
For SQLite,
platform.getDefaultSchemaName()typically returnsundefinedor an empty string. So the name should be the same...Let me look at the
nativeEnumsfield. When creating from metadata, native enums are collected from entity properties. When creating from introspection, they might be collected differently.For SQLite specifically, native enums aren't really a thing - they're implemented as CHECK constraints. Let me check how
loadInformationSchemahandles native enums...Actually, let me focus on a simpler issue. Let me look at what happens with the
viewsfield. When creating from metadata, views are only included if there are view entities defined. When creating from introspection, views are loaded from the database:// Load views from database await platform.getSchemaHelper()!.loadViews(schema, connection, schemaName, ctx);If the database has views but the entities don't define any views, the introspected snapshot would include views while the metadata snapshot wouldn't.
But wait, the user is reporting that "a bunch of stuff in the snapshot is wrong", which suggests many fields are affected, not just views.
Let me look at this from a different angle. Maybe the issue is that when the schema is stored after
up(), it includes information about themikro_orm_migrationstable itself. Let me check if that table is properly filtered out...Looking at
DatabaseSchema.create():const parts = config.get('migrations').tableName!.split('.'); const migrationsTableName = parts[1] ?? parts[0]; const migrationsSchemaName = parts.length > 1 ? parts[0] : config.get('schema', platform.getDefaultSchemaName()); const tables = allTables.filter( t => this.isTableNameAllowed(t.table_name, takeTables, skipTables) && (t.table_name !== migrationsTableName || (t.schema_name && t.schema_name !== migrationsSchemaName)), );The migrations table should be filtered out. But wait - what if the schema name doesn't match? The filter condition is:
(t.table_name !== migrationsTableName || (t.schema_name && t.schema_name !== migrationsSchemaName))This means the table is included if:
table_name !== migrationsTableNameORschema_nameis truthy ANDschema_name !== migrationsSchemaName
Wait, the logic is inverted. The table is excluded if:
table_name === migrationsTableNameAND (schema_nameis falsy ORschema_name === migrationsSchemaName)
So the migrations table should be properly filtered out...
Let me think about what else could cause the snapshot to be "wrong". Let me look at the
toJSONmethod ofDatabaseTablemore carefully to see if there's something that could differ between metadata and introspection.Actually, I just realized something. Let me look at the
DatabaseTable.toJSON()method. There's a line:mappedType: Utils.keys(t).find(k => t[k] === c.mappedType.constructor),This looks up the mapped type name from the type registry. If the
mappedTypeinstance was created differently (e.g., from introspection vs from metadata), the constructor comparison might fail and returnundefined.But wait, the tests pass, so this shouldn't be the issue...
Let me look at the test that specifically checks for equality between metadata and introspection snapshots:
expect(afterUp).toEqual(afterCreate);This test passes, which means the basic case works. So the issue must be in a specific edge case.
Let me think about what could be different:
- Different database drivers (SQLite vs PostgreSQL vs MySQL)
- Entities with specific features (views, native enums, etc.)
- Specific column types or options
The issue mentions SQLite specifically, so let me focus on SQLite.
Actually, wait! I just noticed something. Let me look at the
getSchemaFromSnapshot()method inMigrator.tsmore carefully:const schema = new DatabaseSchema(this.driver.getPlatform(), this.config.get('schema'));When loading from the snapshot, the schema is created with
this.config.get('schema')as the name. But when creating from metadata:const schemaName = schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName();And when creating from introspection:
const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());So the name should be consistent...
Hmm, let me look at the
viewsfield more carefully. InDatabaseSchema.toJSON():views: [...this.#views].sort(byTable),And when loading from snapshot:
if (rest.views) { schema.setViews(rest.views); }But when creating from metadata via
DatabaseSchema.fromMetadata(), views are only added if there are view entities. Let me check iffromMetadata()callssetViews()...Looking at the
fromMetadatamethod, I don't see where it explicitly handles views for non-view entities. Views are only added if there are view entities defined.But when introspecting from the database,
loadViews()is always called, which might add views even if no view entities are defined.Let me check what
loadViews()does for SQLite...Actually, I think I need to look at this more carefully. Let me look at what tables/views/etc. are in the test schema to see if there are any views defined.
Looking at the SQLite schema file, I don't see any views. So that's not the issue.
Let me think about this differently. The issue says "the snapshot changes and a bunch of stuff in the snapshot is wrong". This could mean:
- Fields are missing
- Fields have different values
- The structure is different
Let me look at the
toJSON()method output structure vs what's expected ingetSchemaFromSnapshot():toJSON()returns:namenamespacestables(array of tables)viewsnativeEnums
And
getSchemaFromSnapshot()expects:tablesnamespacesnativeEnumsviews(optional)- And anything else goes into
...rest
Wait, I see it now! The
namefield is included intoJSON()output but it's NOT handled ingetSchemaFromSnapshot()! Let me check if that causes issues...Actually no, the
namefield is just metadata about the schema, not about the structure. It wouldn't cause schema comparison issues because the comparison uses the actual table definitions.Let me look at the
getUpdateSchemaMigrationSQLmethod to see how it compares schemas...Actually, I think I need to run a more comprehensive test to see what's different. Let me create a test script that:
- Creates entities with various features
- Creates the schema
- Creates a blank migration
- Runs the migration up
- Compares the snapshots in detail
Wait, I already did that and the snapshots were identical. Let me think about what the issue reporter might have that's different...
The issue mentions version 7.1.2-dev.10, which is a dev version. Maybe the bug was already fixed in the current code?
Let me check if there are any recent changes to the migration snapshot code...
Actually, looking at the commit history, I see
fix(sql): stabilize migration snapshot serialization (#7608)which was merged. And the current HEAD is atfix(sql): stabilize MySQL snapshot serialization for decimal/text/extra/comment (#7797).So the basic snapshot stability has been addressed. But maybe there's still a remaining issue?
Let me look at the test for GH #7796 which was added in the most recent commit...
Actually, let me re-read the issue more carefully. The issue says "When we run orm.migrator.up(), the snapshot changes and a bunch of stuff in the snapshot is wrong and we have to revert the changes."
This sounds like the snapshot file is being modified in a way that causes subsequent migration creates to generate incorrect diffs. The key is that the user has to "revert the changes" to the snapshot.
What if the issue is with how the
viewsfield is handled? Let me check...In
DatabaseSchema.toJSON():views: [...this.#views].sort(byTable),This always includes
viewsin the output. But when creating from metadata,fromMetadata()might not add any views if there are no view entities. So the metadata snapshot would have"views": [].When introspecting from the database, if the database has no views,
loadViews()would return an empty list, so the snapshot would also have"views": [].But what if the database has some views that weren't created by MikroORM? Then the introspected snapshot would include those views, while the metadata snapshot wouldn't.
Hmm, that's still not "a bunch of stuff wrong"...
Let me look at another possibility. What if the issue is with the
namefield? Let me check what thenamefield would be for both paths...When creating from metadata with
getTargetSchema():const schemaName = schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName();When creating from introspection with
DatabaseSchema.create():const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());Both use the same fallback chain, so the name should be the same...
Wait! I think I found something. Let me look at the
runMigrationsmethod in Migrator.ts again:const schema = await DatabaseSchema.create( this.em.getConnection(), this.em.getPlatform(), this.config, undefined, // schemaName undefined, // schemas undefined, // takeTables undefined, // skipTables undefined, // skipViews ctx, );The
schemasparameter isundefined. Let me see what happens whenschemasis undefined inDatabaseSchema.create():const allTables = await platform.getSchemaHelper()!.getAllTables(connection, schemas, ctx);If
schemasis undefined,getAllTablesmight return tables from all schemas/databases. For SQLite with attached databases, this could include tables from all attached databases, not just the main database!But for a single SQLite database with no attached databases, this should work correctly...
Let me look at this from a different angle. Let me look at what
DatabaseSchema.create()does with thetakeTablesandskipTablesparameters. When these are undefined, all tables are included (except the migrations table).But
getTargetSchema()only includes tables defined by entity metadata. So if there are tables in the database that aren't defined by entities, the introspected snapshot would include them while the metadata snapshot wouldn't.But again, this seems like expected behavior...
OK, I think I need to look at the specific reproduction repository to understand the issue better. Since I can't access the GitHub repo directly, let me think about what kind of entity setup would cause the snapshot to be "wrong".
One thing I notice is that the introspection path might include the
sqlite_sequencetable or other internal SQLite tables. Let me check if those are filtered...Looking at
getListTablesSQL():return ( `select name as table_name from sqlite_master where type = 'table' and name != 'sqlite_sequence' and name != 'geometry_columns' and name != 'spatial_ref_sys' ` + `union all select name as table_name from sqlite_temp_master where type = 'table' order by name` );The
sqlite_sequencetable is filtered out. But what about other internal tables?Actually, wait. Let me look at
getAllTablesmore carefully:override async getAllTables( connection: AbstractSqlConnection, schemas?: string[], ctx?: Transaction, ): Promise<Table[]> { const databases = await this.getDatabaseList(connection, ctx); const hasAttachedDbs = databases.length > 1; // More than just 'main' // If no attached databases, use original behavior if (!hasAttachedDbs && !schemas?.length) { return connection.execute<Table[]>(this.getListTablesSQL(), [], 'all', ctx); } ... }When there are no attached databases and no schemas are specified, it uses
getListTablesSQL(). But when there ARE attached databases or schemas are specified, it uses a different query:`select name from ${prefix}sqlite_master where type = 'table' ` + `and name != 'sqlite_sequence' and name != 'geometry_columns' and name != 'spatial_ref_sys'`Both queries filter out the same internal tables.
OK let me take a step back. The tests pass, so the basic functionality works. The issue must be in a specific edge case. Let me look at what might be different in the user's setup.
One thing I notice is that the test uses
:memory:as the database name, which means the snapshot file is named.snapshot-memory.json. What if the user is using a file-based database and there's some issue with the snapshot file name?Actually, let me look at the
getSnapshotPath()method:private async getSnapshotPath(): Promise<string> { if (!this.#snapshotPath) { const { fs } = await import('@mikro-orm/core/fs-utils'); const snapshotPath = this.options.emit === 'ts' && this.options.pathTs ? this.options.pathTs : this.options.path!; const absoluteSnapshotPath = fs.absolutePath(snapshotPath, this.config.get('baseDir')); const dbName = this.config.get('dbName')!.replace(/\\/g, '/').split('/').pop()!.replace(/:/g, ''); const snapshotName = this.options.snapshotName ?? `.snapshot-${dbName}`; this.#snapshotPath = fs.normalizePath(absoluteSnapshotPath, `${snapshotName}.json`); } return this.#snapshotPath; }The database name is extracted from
this.config.get('dbName')!and processed:- Replace backslashes with forward slashes
- Split by
/and take the last part - Replace colons with empty string
For
:memory:, this would result inmemory(the colons are removed). For a file like/path/to/test.db, this would result intest.db.This looks correct...
OK, I'm going to try a different approach. Let me look at what specific differences might exist between the metadata and introspection paths by running a detailed comparison test.
Actually, let me re-read the test that was added for GH #7607:
test('snapshot serialization is stable across metadata and introspection paths (GH #7607)', async () => { ... expect(afterUp).toEqual(afterCreate); ... });This test checks that the entire snapshot JSON is equal. If this test passes, then the basic case works. But the issue might be in a specific entity setup that isn't covered by the test.
Let me look at what entities are used in the test. The test uses
initORMSqlite()which loads entities fromtests/entities-schema/(excludingUser4.ts).Let me think about what entity features might cause issues:
- Composite primary keys
- Embedded entities
- Polymorphic relations
- Native enums
- Generated columns
- Views
- Indexes with expressions
- Check constraints
- Foreign keys with different rules
- Version columns
- Unique constraints
Let me also think about what database-level features might be introspected differently:
- Default values
- Column types
- Indexes
- Foreign keys
- Check constraints
Actually, I just realized something important. Let me look at what happens when
DatabaseSchema.create()is called withctx(transaction context):const ctx = Utils.isObject<MigrateOptions>(options) ? options.transaction : undefined; const schema = await DatabaseSchema.create( this.em.getConnection(), this.em.getPlatform(), this.config, undefined, undefined, undefined, undefined, undefined, ctx, );If the migration is run inside a transaction (when
allOrNothingis true), thectxparameter would be the transaction. But if the transaction hasn't been committed yet whenDatabaseSchema.create()is called, the schema introspection might not see the changes made by the migration!Wait, but the migration SQL has already been executed at this point. The
runMigrationsmethod callssuper.runMigrations()first, which executes the migration, and then it stores the schema. So the changes should be visible...Unless the migration is run in a transaction that hasn't been committed yet. Let me check how the migration runner works...
Actually, looking at the test
'migration up and down both write the snapshot':// run the migration up — snapshot SHOULD be written await migrator.up(migration1.fileName); expect(storeSchemaSpy).toHaveBeenCalledTimes(1); const snapshotAfterUp = readFileSync(snapshotPath, 'utf8'); expect(snapshotAfterUp).toBeDefined();This test expects the snapshot to be written after
up(). And it passes. So the basic functionality works.Let me think about what else could be wrong...
Actually, I just had an idea. What if the issue is that when
DatabaseSchema.create()introspects the database, it includes information about themikro_orm_migrationstable because the filtering logic doesn't work correctly in some cases?Let me look at the filter logic again:
const parts = config.get('migrations').tableName!.split('.'); const migrationsTableName = parts[1] ?? parts[0]; const migrationsSchemaName = parts.length > 1 ? parts[0] : config.get('schema', platform.getDefaultSchemaName()); const tables = allTables.filter( t => this.isTableNameAllowed(t.table_name, takeTables, skipTables) && (t.table_name !== migrationsTableName || (t.schema_name && t.schema_name !== migrationsSchemaName)), );If the migrations table name is
mikro_orm_migrations, then:parts = ['mikro_orm_migrations']migrationsTableName = 'mikro_orm_migrations'migrationsSchemaName = config.get('schema', platform.getDefaultSchemaName())
For SQLite,
getDefaultSchemaName()might returnundefinedor'main'. Let me check...Actually, I just realized something. The filter condition is:
(t.table_name !== migrationsTableName || (t.schema_name && t.schema_name !== migrationsSchemaName))This means a table is included if:
- Its name is not
mikro_orm_migrations, OR - It has a
schema_namethat's notmigrationsSchemaName
So a table named
mikro_orm_migrationswould be excluded if:t.schema_nameis falsy (undefined/null), ORt.schema_name === migrationsSchemaName
But what if
t.schema_nameis'main'(SQLite's default database name) andmigrationsSchemaNameisundefined? Then:t.schema_nameis'main'(truthy)t.schema_name !== migrationsSchemaNameis'main' !== undefinedwhich istrue- So the table would be included!
That's the bug! When the migrations table has a
schema_nameof'main'(which SQLite uses as the default database name) and the configured schema name isundefined, the filter doesn't exclude the migrations table!Let me verify this by checking what
getAllTablesreturns for SQLite tables...Looking at
getAllTables:if (!hasAttachedDbs && !schemas?.length) { return connection.execute<Table[]>(this.getListTablesSQL(), [], 'all', ctx); }The query is:
select name as table_name from sqlite_master where type = 'table' ...This only returns
table_name, notschema_name. So for tables from the main database,schema_namewould beundefined.But wait, when there are attached databases:
for (const t of tables) { allTables.push({ table_name: t.name, schema_name: dbName }); }The
schema_nameis set todbName. For the main database,dbNamewould be'main'.So the issue is:
- When there are attached databases, tables from the main database get
schema_name: 'main' - The migrations table filter compares
schema_namewithmigrationsSchemaName - If
migrationsSchemaNameisundefined(default for SQLite), the comparison fails and the migrations table is included
But this only affects users with attached databases...
Actually, let me look at the simpler case. When there are no attached databases:
if (!hasAttachedDbs && !schemas?.length) { return connection.execute<Table[]>(this.getListTablesSQL(), [], 'all', ctx); }The query doesn't include
schema_name, so all tables would haveschema_name: undefined. The filter condition would be:t.table_name !== migrationsTableName→ If the table is the migrations table, this isfalset.schema_name && t.schema_name !== migrationsSchemaName→undefined && ...→undefined(falsy)false || false→false, so the migrations table is excluded
OK so for the simple case, the migrations table is correctly excluded.
Let me think about other edge cases...
Actually, I think I should look at the actual reproduction. Since the issue says the user has to "revert the changes", this means the snapshot file is being modified incorrectly. Let me see what specific changes would be made to the snapshot.
Looking at the code flow:
migrator.up()runs migrations- After running,
DatabaseSchema.create()introspects the database storeCurrentSchema(schema)serializes the schema to JSON
The key question is: what's different about the introspected schema compared to the metadata schema?
Let me look at what
DatabaseSchema.fromMetadata()includes thatDatabaseSchema.create()might not, and vice versa.fromMetadata():- Collects native enums from entity properties
- Adds tables for each entity
- Adds columns for each property
- Adds foreign keys for relations
- Adds indexes from property definitions
- Adds views for view entities
create():- Loads tables from
sqlite_master - Loads column info from PRAGMA or information_schema
- Loads foreign keys from PRAGMA or information_schema
- Loads indexes from PRAGMA or information_schema
- Loads views from
sqlite_master
The key difference is that
fromMetadata()knows about the entity structure, whilecreate()introspects the actual database. If the database has any extra tables (like the migrations table), the introspected schema would include them.But we already established that the migrations table is filtered out. So what else could be different?
Wait, I think I finally see the issue! Let me look at the
runMigrationsmethod again:protected override async runMigrations( method: 'up' | 'down', options?: string | string[] | MigrateOptions, ): Promise<MigrationInfo[]> { const result = await super.runMigrations(method, options); if (result.length > 0 && this.options.snapshot) { const ctx = Utils.isObject<MigrateOptions>(options) ? options.transaction : undefined; const schema = await DatabaseSchema.create( this.em.getConnection(), this.em.getPlatform(), this.config, undefined, undefined, undefined, undefined, undefined, ctx, ); try { await this.storeCurrentSchema(schema); } catch { // Silently ignore for read-only filesystems (production). } } return result; }Wait, I see that the
schemaparameter passed toDatabaseSchema.create()isundefined. But let me check theDatabaseSchema.create()method signature again:static async create( connection: AbstractSqlConnection, platform: AbstractSqlPlatform, config: Configuration, schemaName?: string, schemas?: string[], takeTables?: (string | RegExp)[], skipTables?: (string | RegExp)[], skipViews?: (string | RegExp)[], ctx?: Transaction, ): Promise<DatabaseSchema>So:
schemaName = undefinedschemas = undefinedtakeTables = undefinedskipTables = undefinedskipViews = undefined
Now let me trace through
DatabaseSchema.create():const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());The schema name is
config.get('schema') ?? platform.getDefaultSchemaName(). For SQLite,getDefaultSchemaName()returns... let me check.<tool_call> <function=bash_terminal> <parameter=command> cd /mikro-orm && grep -n "getDefaultSchemaName" packages/sql/src/dialects/sqlite/SqlitePlatform.ts </tool_call>
Assistant
I need to run a command to check the getDefaultSchemaName method in the SQLite platform.
Assistant
Let me check what the SQLite platform's
getDefaultSchemaName()returns.Assistant
Found something interesting! Let me check what
supportsSchemas()returns:Assistant
So
getDefaultSchemaName()returns'main'only ifsupportsSchemas()is true, which only happens whenattachDatabasesis configured. Otherwise it returnsundefined.Now I understand the flow better. Let me trace through the issue more carefully.
The bug is about
orm.migrator.up()rewriting snapshot incorrectly. Let me look at what therunMigrationsmethod does:protected override async runMigrations( method: 'up' | 'down', options?: string | string[] | MigrateOptions, ): Promise<MigrationInfo[]> { const result = await super.runMigrations(method, options); if (result.length > 0 && this.options.snapshot) { const ctx = Utils.isObject<MigrateOptions>(options) ? options.transaction : undefined; const schema = await DatabaseSchema.create( this.em.getConnection(), this.em.getPlatform(), this.config, undefined, // schemaName undefined, // schemas ... ); await this.storeCurrentSchema(schema); } return result; }So when
up()is called, it creates a newDatabaseSchemafrom the actual database via introspection (DatabaseSchema.create()), and then stores it.When
create()is called (for creating migrations), it usesstoreCurrentSchema()without passing a schema, so it defaults togetTargetSchema()which builds from metadata.The issue is that the snapshot written by
up()(from introspection) might differ from the snapshot written bycreate()(from metadata), causing subsequentcreate()calls to generate incorrect diffs.Let me look at what specific fields might differ. The test
'snapshot serialization is stable across metadata and introspection paths (GH #7607)'checks thatafterUp.toEqual(afterCreate), and it passes. So the basic case works.But wait - maybe the issue is something specific to certain entity configurations. Let me look at what the reproduction repository might have that's different.
Actually, let me re-read the issue description more carefully. The user says "the snapshot changes and a bunch of stuff in the snapshot is wrong". This suggests that after running
up(), the snapshot file is modified with incorrect content.Let me think about what could cause this:
-
The
namefield inDatabaseSchema.toJSON()- this is the schema name. If it differs between metadata and introspection paths, the snapshot would have a differentnamefield. -
The
nativeEnumsfield - these are collected differently between metadata and introspection. -
The
viewsfield - if the database has views that aren't defined in entities. -
The
tablesfield - if the introspection includes tables not defined in entities.
Wait, I just realized something important. Let me look at the
namefield more carefully.When
storeCurrentSchema()is called with a schema fromDatabaseSchema.create():- The schema name is:
schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName() - If
config.get('schema')is undefined andplatform.getDefaultSchemaName()returns undefined (for SQLite without attached databases), the name would be undefined.
When
storeCurrentSchema()is called without a schema (usinggetTargetSchema()):- The schema name is:
schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName() - Same result.
So the name should be consistent...
But wait! What if
toJSON()serializesname: undefinedas"name": undefined, which JSON.stringify would omit? Let me check:<tool_call> <function=bash_terminal> <parameter=command> node -e "console.log(JSON.stringify({name: undefined}, null, 2))" </tool_call>
-
Assistant
Let me check how JSON.stringify handles undefined values.
Assistant
Let me look at the test for GH #7796 which was added in the most recent commit to understand what specific issue it addresses:
Assistant
I see that GH#7796 was about MySQL-specific issues. Let me continue investigating the SQLite-specific case mentioned in the original issue.
Assistant
The tests pass. Let me look more carefully at the issue. Since the user reported the issue with SQLite and mentions the snapshot is "wrong" after running
up(), let me examine the introspection path more carefully to see what could cause differences.Assistant
Let me look at the SQLite schema helper's
loadInformationSchemamethod to understand how it loads table information from the database. This is critical because it's what populates the schema whenDatabaseSchema.create()is called duringmigrator.up().Assistant
Let me look at how columns are loaded from the database to understand potential differences:
Assistant
Let me check if there's a difference in how triggers are handled between metadata and introspection paths:
Assistant
Let me look at the toJSON method of DatabaseTable to see if triggers are serialized:
Assistant
Let me look at the DatabaseTable.toJSON() method to see how triggers are serialized, and also check what the sortedTriggers output looks like. The key is that triggers are loaded from the database during introspection but might not be defined in the metadata, or vice versa.
Assistant
Let me investigate how the table's
schemafield is populated in both paths and check for potential mismatches.Assistant
Let me examine the
runMigrationsmethod more carefully to understand how the schema is being introspected after migrations run.Assistant
Let me look more carefully at how the
DatabaseSchemaname is set in both paths, and also look at what thetoJSON()method outputs.Assistant
Let me look more carefully at how views are handled in the SQLite schema helper and whether there's a mismatch between metadata and introspection paths.
Assistant
...
Assistant
Let me look at how views are handled in the
fromMetadatamethod compared tocreate:Assistant
Let me analyze the issue more systematically. The problem is that
orm.migrator.up()rewrites the snapshot incorrectly.Looking at the code flow:
migrator.up()runs migrations- After running, it calls
DatabaseSchema.create()to introspect the database - Then it stores this schema as the new snapshot
The key question is: what could cause the introspected schema to differ from the metadata-based schema in a way that makes "a bunch of stuff wrong"?
Let me look at the
runMigrationsmethod again. The issue is that it's callingDatabaseSchema.create()withundefinedfor many parameters, includingschemaName,schemas,takeTables,skipTables, andskipViews.When
DatabaseSchema.create()is called:- It creates a new schema with name from config or platform default
- It loads ALL tables from the database (minus migrations table)
- It loads ALL views from the database
But
getTargetSchema()only includes tables/views defined in entity metadata!So if the database has any tables or views that aren't defined in entity metadata, the introspected snapshot would include them while the metadata snapshot wouldn't.
Wait, but that's the expected behavior - the snapshot should reflect the actual database state after migrations. The issue is something else.
Let me think about this differently. The user says "a bunch of stuff in the snapshot is wrong". This suggests the snapshot content is incorrect, not that extra tables are included.
Actually, I think I need to look at the specific scenario more carefully. Let me look at how the
namefield ofDatabaseSchemais set and whether it could differ between paths...Actually wait - I see something. In
DatabaseSchema.toJSON():return { name: this.name, ... }But when loading from snapshot in
getSchemaFromSnapshot():const { tables, namespaces, ...rest } = data;The
namefield would be inrest, but it's never used! Let me check if this matters...Actually, looking at the code more carefully, the
namefield is just metadata about the schema and doesn't affect the actual schema comparison. The comparison uses the actual table definitions.Let me think about what else could be wrong...
Actually, I just realized something. When looking at the
runMigrationsmethod, it passesctx(transaction context) toDatabaseSchema.create(). But this context is:const ctx = Utils.isObject<MigrateOptions>(options) ? options.transaction : undefined;This extracts the transaction from the options object if it exists. But wait -
Utils.isObject<MigrateOptions>(options)checks ifoptionsis an object matching theMigrateOptionstype. Ifoptionsis a string (like a migration name),ctxwould beundefined.Let me check what happens when
migrator.up()is called with a string argument...Actually, the issue might be simpler. Let me look at the
MigrateOptionstype to see whattransactionmeans.Hmm, I'm going in circles. Let me try a different approach - let me look at what the actual reproduction might look like by examining the test more carefully.
The test
'snapshot serialization is stable across metadata and introspection paths (GH #7607)'passes. It creates a blank migration and runsup(), then compares the snapshots. They should be equal.But wait - the test uses
initORMSqlite()which already creates the schema in the database usingorm.schema.update(). So the database already has the tables before the migration runs.In the user's scenario, maybe the database is empty and the migration creates the tables? Let me check...
Actually, in the test:
initORMSqlite()creates tables in the databasemigrator.create(path, true)creates a blank migration and writes the metadata-based snapshotmigrator.up(migration1.fileName)runs the blank migration and writes the introspected snapshot- The snapshots should be equal because the database already has the tables
So the test covers the case where the database has the tables before the migration runs. What if the user's scenario is different?
What if the user:
- Has an empty database
- Creates an initial migration
- Runs
up()to create the tables
In this case, the introspection would see the newly created tables. But the metadata-based snapshot would also include those tables because they're defined in the entities.
Wait, I think I need to look at this from yet another angle. Let me look at what happens when
getSchemaDiffis called withinitial = truevsinitial = false.When
initial = true, it usesgetCreateSchemaSQL(). Wheninitial = false, it usesgetUpdateSchemaMigrationSQL()withfromSchema: await this.getSchemaFromSnapshot().So for non-initial migrations, it compares the target schema (from metadata) with the snapshot schema (from the snapshot file). If they're equal, no migration is generated.
The issue is that after
up(), the snapshot is rewritten from introspection. If the introspected snapshot differs from the metadata snapshot, then the nextcreate()call would generate a migration with incorrect changes.So the question is: what specific fields differ between the introspected and metadata snapshots?
Looking at the test, it passes. So for the entities used in the test, the snapshots are equal. The issue must be with specific entity configurations that aren't covered by the test.
Let me think about what specific features might cause differences:
- Triggers - loaded from DB during introspection, defined in metadata for entities with triggers
- Check constraints - could be defined differently
- Indexes - could have different options
- Column defaults - could be formatted differently
- Column types - could be normalized differently
Actually, I wonder if the issue is with the
viewsfield. Let me check what happens when there are no views...In
DatabaseSchema.toJSON():views: [...this.#views].sort(byTable),If there are no views, this would be
views: [].In
getSchemaFromSnapshot():if (rest.views) { schema.setViews(rest.views); }If
rest.viewsis[], this is falsy, so views wouldn't be set. But wait,[]is truthy in JavaScript! SosetViews([])would be called.Actually, let me verify:
if ([])is truthy. SosetViews([])would be called for empty views arrays.Hmm, that's not the issue then.
OK let me look at this problem from a completely different angle. Let me check if there are any open issues or PRs related to this problem in the repository...
Actually, I think the issue might be simpler. Let me look at the
runMigrationsmethod and check if there's something wrong with how it creates the schema.Wait, I just noticed something! The
runMigrationsmethod usesDatabaseSchema.create()which is the introspection path. But looking at the parameters:const schema = await DatabaseSchema.create( this.em.getConnection(), this.em.getPlatform(), this.config, undefined, // schemaName undefined, // schemas undefined, // takeTables undefined, // skipTables undefined, // skipViews ctx, );The
schemaNameisundefined. InDatabaseSchema.create():const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());So the schema name comes from
config.get('schema')orplatform.getDefaultSchemaName().But in
getTargetSchema():const schemaName = schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName();Same logic! So the schema name should be the same.
BUT wait - there's a subtlety. In
DatabaseSchema.create(),platform.getDefaultSchemaName()might return different things depending on the state of the platform. For SQLite, it returns'main'only ifsupportsSchemas()is true (i.e., there are attached databases), otherwiseundefined.And in
getTargetSchema(), it usesthis.platform.getDefaultSchemaName().If the platform state changes between the two calls, they might return different values. But that seems unlikely...
Let me check if there's something specific about how the schema is loaded that could cause issues. Let me look at
loadInformationSchemafor SQLite again.Actually, I just realized I should look at how
getAllTableshandles the case whenschemasis undefined. Let me trace through more carefully:override async getAllTables( connection: AbstractSqlConnection, schemas?: string[], ctx?: Transaction, ): Promise<Table[]> { const databases = await this.getDatabaseList(connection, ctx); const hasAttachedDbs = databases.length > 1; if (!hasAttachedDbs && !schemas?.length) { return connection.execute<Table[]>(this.getListTablesSQL(), [], 'all', ctx); } ... }When there are no attached databases and
schemasis undefined, it usesgetListTablesSQL(). Let me check what columns this query returns:return ( `select name as table_name from sqlite_master where type = 'table' and name != 'sqlite_sequence' and name != 'geometry_columns' and name != 'spatial_ref_sys' ` + `union all select name as table_name from sqlite_temp_master where type = 'table' order by name` );It only returns
table_name, notschema_name. So for tables from the main database,schema_namewould beundefined.Now in
DatabaseSchema.create():const tables = allTables.filter( t => this.isTableNameAllowed(t.table_name, takeTables, skipTables) && (t.table_name !== migrationsTableName || (t.schema_name && t.schema_name !== migrationsSchemaName)), );For the migrations table:
t.table_namewould be'mikro_orm_migrations'migrationsTableNamewould be'mikro_orm_migrations't.table_name !== migrationsTableNameisfalset.schema_nameisundefinedt.schema_name && t.schema_name !== migrationsSchemaNameisundefined && ...which isundefined(falsy)false || undefinedisundefined(falsy)- So the migrations table is filtered out ✓
OK so the migrations table is correctly filtered.
Now let me think about what else could cause issues...
Actually, I think I should look at the
DatabaseSchema.create()method more carefully. It creates a schema with:const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());And then loads tables, views, etc. But it doesn't set
nativeEnumson the schema! Let me check...Looking at
loadInformationSchema:for (const t of tables) { const table = schema.addTable(t.table_name, t.schema_name, t.table_comment); ... const enums = this.extractEnumValuesFromChecks(checks); table.init(cols, indexes, checks, pks, fks, enums); ... }The
enumsare passed totable.init(), which sets them on the table. Butschema.setNativeEnums()is never called!Let me check how
schema.addTable()works:addTable(name: string, schema: string | undefined | null, comment?: string): DatabaseTable { const namespaceName = schema ?? this.name; const table = new DatabaseTable(this.#platform, name, namespaceName); table.nativeEnums = this.#nativeEnums; table.comment = comment; this.#tables.push(table); ... return table; }When a table is added to the schema, it shares the schema's
nativeEnumsby reference. But the schema'snativeEnumsis empty becausesetNativeEnums()was never called on the schema!Then in
table.init():init( cols: Column[], indexes: IndexDef[] = [], checks: CheckDef[] = [], pks: string[], fks: Dictionary<ForeignKey> = {}, enums: Dictionary<string[]> = {}, ): void { ... const helper = this.#platform.getSchemaHelper(); this.#columns = cols.reduce((o, v) => { ... const type = v.name in enums ? 'enum' : v.type; ... v.enumItems ??= enums[v.name] || []; ... o[v.name] = v; ... }, {} as Dictionary<Column>); }The
enumsparameter (fromextractEnumValuesFromChecks) is used to setenumItemson columns. But the schema-levelnativeEnumsdictionary is never populated!Now let me check what happens when
toJSON()is called:return { ... nativeEnums: Object.fromEntries(Object.entries(this.#nativeEnums).sort(([a], [b]) => byString(a, b))), };For the introspected schema,
nativeEnumswould always be empty{}becausesetNativeEnums()was never called!But for the metadata-based schema (from
fromMetadata()):schema.setNativeEnums(nativeEnums);The schema would have native enums if the entities define any!
So if the user has entities with native enums, the metadata-based snapshot would include them in
nativeEnums, but the introspected snapshot wouldn't!But wait, the tests use SQLite and there are no native enum entities in the test. Let me check...
Actually, for SQLite, native enums are implemented as CHECK constraints. So they're not "native" in the SQLite sense. Let me check if SQLite has native enum support...
Looking at the platform:
supportsNativeEnums(): boolean { return false; }SQLite doesn't support native enums. So
nativeEnumswould be empty for both paths.Hmm, so that's not the issue for SQLite.
Let me think about what else could differ...
Actually, let me look at the views more carefully. In
DatabaseSchema.create():await platform.getSchemaHelper()!.loadViews(schema, connection, schemaName, ctx);This loads views from the database. If there are no views in the database,
schema.#viewswould be[].In
DatabaseSchema.fromMetadata():for (const meta of metadata) { if (meta.view) { const viewDefinition = this.getViewDefinition(meta, em, platform); if (viewDefinition) { const view = schema.addView(meta.collection, ...); } } }If there are no view entities,
schema.#viewswould also be[].So views should be the same for both paths if there are no views.
Let me look at the
namespacesfield. InDatabaseSchema.toJSON():namespaces: [...this.#namespaces].sort(),For the metadata path, namespaces are added when tables and views are added. For the introspection path, they're also added when tables and views are added.
Let me check if there's a difference...
In
addTable():addTable(name: string, schema: string | undefined | null, comment?: string): DatabaseTable { const namespaceName = schema ?? this.name; ... if (namespaceName != null) { this.#namespaces.add(namespaceName); } ... }For the metadata path:
schemacomes frommeta.schema ?? config.get('schema')or similar- If the entity has no explicit schema and no config schema,
namespaceNamewould bethis.name(the schema name) - If
this.nameisundefined,namespaceNamewould beundefinedand not added to namespaces
For the introspection path:
schemacomes fromt.schema_name(which isundefinedfor SQLite tables from main database)namespaceName = undefined ?? this.name = this.name- Same as metadata path
OK so namespaces should be the same.
Let me try a completely different approach. Let me create a more comprehensive test that checks all the fields in both snapshots and reports any differences.
Actually, let me first check if there's something wrong with the
toJSON()method itself that could cause issues.Wait, I just thought of something. The
namefield inDatabaseSchema.toJSON():return { name: this.name, ... }If
this.nameisundefined,JSON.stringify()would omit it. But ifthis.nameis''(empty string), it would include"name": "".Let me check what the schema name is for SQLite...
In
getTargetSchema():const schemaName = schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName();For SQLite without attached databases,
getDefaultSchemaName()returnsundefined. Ifconfig.get('schema')is alsoundefined, thenschemaNameisundefined.In
DatabaseSchema.create():const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());Same logic, so
nameshould beundefinedfor both.When
toJSON()is called:name: this.name,If
this.nameisundefined,JSON.stringify()would omit thenamefield.So both snapshots should omit the
namefield (or have it undefined and omitted).Hmm, I'm running out of ideas. Let me look at the reproduction more carefully.
Actually, wait. Let me re-read the issue description: "When we run orm.migrator.up(), the snapshot changes and a bunch of stuff in the snapshot is wrong"
This says "the snapshot changes" - meaning the snapshot file is modified. And "a bunch of stuff is wrong" - meaning the content is incorrect.
What if the issue is not about metadata vs introspection, but about something else entirely? Like the snapshot being truncated, corrupted, or written incorrectly?
Let me look at
storeCurrentSchema():protected async storeCurrentSchema(schema?: DatabaseSchema): Promise<void> { if (!this.options.snapshot) { return; } const snapshotPath = await this.getSnapshotPath(); schema ??= this.#schemaGenerator.getTargetSchema(); const { fs } = await import('@mikro-orm/core/fs-utils'); await fs.writeFile(snapshotPath, JSON.stringify(schema, null, 2)); }This looks correct. It serializes the schema to JSON and writes it to the file.
Wait! I just noticed something. When
storeCurrentSchema()is called without a schema argument (duringcreate()), it usesthis.#schemaGenerator.getTargetSchema(). This creates a NEW schema from metadata every time!But when
storeCurrentSchema(schema)is called with a schema argument (duringup()), it uses the passed schema.What if the issue is that
getTargetSchema()is returning a different schema than expected?Let me look at
getTargetSchema()again:getTargetSchema(schema?: string, includeWildcardSchema = false): DatabaseSchema { const metadata = this.getOrderedMetadata(schema, includeWildcardSchema); const schemaName = schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName(); const target = DatabaseSchema.fromMetadata(metadata, this.platform, this.config, schemaName, this.em); ... return target; }It creates a new schema from metadata. But what if the metadata is different between calls?
Actually, I think I've been overthinking this. Let me look at the issue from a simpler perspective.
The user says "the snapshot changes and a bunch of stuff in the snapshot is wrong". The key observation is that after running
up(), the snapshot is different from what it was before.If the snapshots from metadata and introspection are supposed to be identical (as the test verifies), then they should be. But if there's a subtle difference, the user would see the snapshot change after
up().Let me think about what could cause such a difference...
Actually, I just realized I should check if the issue is specific to a certain version. The user mentions "7.1.2-dev.10", but the current repo version is 7.1.1. Maybe the issue was introduced after 7.1.1?
Let me check if there are any changes to the Migrator or DatabaseSchema code between the current HEAD and the version the user is using.
Actually, the user's reproduction repo is from "7.1.2-dev.10", which is a dev version. The current HEAD of the repo might have fixes for this issue already.
Wait, but the issue is filed against this repo, so the fix should be in this repo. Let me check if there's an unfixed issue...
Actually, I think the issue might be related to the
viewsfield being included in the snapshot but with a different structure. Let me look more carefully...In
DatabaseSchema.toJSON():views: [...this.#views].sort(byTable),Views are
DatabaseView[]objects. Let me check the structure:interface DatabaseView { name: string; schema: string | undefined; definition: string; materialized?: boolean; withData?: boolean; }So views have
name,schema,definition,materialized, andwithData.When loading from the snapshot:
if (rest.views) { schema.setViews(rest.views); }The views are set as-is from the JSON.
When loading from the database (introspection):
schema.addView(view.view_name, schemaName, this.extractViewDefinition(view.view_definition));This creates views with
name,schema, anddefinition. ThematerializedandwithDatafields would beundefined.When creating from metadata:
const view = schema.addView( meta.collection, this.getSchemaName(meta, config, schemaName), viewDefinition, meta.materialized, meta.withData, );This creates views with
materializedandwithDatafrom the entity metadata.So for materialized views, the metadata path would include
materializedandwithDatafields, but the introspection path wouldn't!But wait, SQLite doesn't support materialized views, so this wouldn't be an issue for SQLite.
Hmm, let me think about this differently. The user is using SQLite. The test passes for SQLite. So what's different about the user's setup?
Actually, I wonder if the issue is with the
toJSONmethod not properly serializing all fields. Let me look at the DatabaseTable.toJSON() method more carefully...Looking at the code, I see that
DatabaseTable.toJSON()callsnormalizeIndex,normalizeFk, andnormalizeCheckfunctions. These functions might not include all fields.Let me look at
normalizeIndex:const normalizeIndex = (idx: IndexDef): Dictionary => { const out: Dictionary = { columnNames: idx.columnNames, composite: !!idx.composite, constraint: !!idx.constraint || !!idx.primary, keyName: idx.keyName, primary: !!idx.primary, unique: !!idx.unique, }; const optional = [ 'expression', 'where', 'type', 'deferMode', 'columns', 'include', 'fillFactor', 'invisible', 'disabled', 'clustered', ] as const; for (const field of optional) { if (idx[field] != null && idx[field] !== false) { out[field] = idx[field]; } } return out; };This looks correct. All fields from
IndexDefare handled.Let me look at
normalizeFk:const normalizeFk = (fk: ForeignKey): Dictionary => { const isNoAction = (rule?: string) => !rule || rule.toLowerCase() === 'no action'; return { columnNames: fk.columnNames, constraintName: fk.constraintName, localTableName: fk.localTableName, referencedColumnNames: fk.referencedColumnNames, referencedTableName: fk.referencedTableName, updateRule: isNoAction(fk.updateRule) ? undefined : fk.updateRule, deleteRule: isNoAction(fk.deleteRule) ? undefined : fk.deleteRule, deferMode: fk.deferMode, }; };This also looks correct.
OK, I think I need to take a step back and think about what the actual bug could be.
The issue is that
orm.migrator.up()rewrites the snapshot incorrectly. The user has to revert the snapshot changes.What if the bug is that the snapshot includes information that shouldn't be there? Like the migrations table, or internal SQLite tables?
Let me check if
DatabaseSchema.create()properly filters out all internal tables...Looking at the
getListTablesSQL():return ( `select name as table_name from sqlite_master where type = 'table' and name != 'sqlite_sequence' and name != 'geometry_columns' and name != 'spatial_ref_sys' ` + `union all select name as table_name from sqlite_temp_master where type = 'table' order by name` );It filters out
sqlite_sequence,geometry_columns, andspatial_ref_sys. And inDatabaseSchema.create(), the migrations table is also filtered out.So the internal tables should be properly filtered.
What about temporary tables? The query includes
sqlite_temp_master. If there are temporary tables, they would be included in the snapshot!But temporary tables are usually cleaned up automatically, so this is unlikely to be the issue.
OK, I think I need to look at this from yet another angle. Let me check if the issue might be related to the
namefield in the snapshot.Actually, let me check if the
namefield is included in the JSON output...When
JSON.stringify()is called on the schema:await fs.writeFile(snapshotPath, JSON.stringify(schema, null, 2));The
toJSON()method returns:return { name: this.name, namespaces: [...this.#namespaces].sort(), tables: [...this.#tables].sort(byTable), views: [...this.#views].sort(byTable), nativeEnums: Object.fromEntries(Object.entries(this.#nativeEnums).sort(([a], [b]) => byString(a, b))), };If
this.nameisundefined,JSON.stringify()would include"name": undefinedbut then actually omit it (undefined values are omitted in JSON).Wait, actually
JSON.stringify({name: undefined})returns"{}". So thenamefield would be omitted if it'sundefined.But what if
this.nameis''(empty string)? Then"name": ""would be included in the JSON.Let me check what the schema name would be for both paths when
config.get('schema')returnsundefined:For
getTargetSchema():const schemaName = schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName();schemaisundefinedconfig.get('schema')isundefinedplatform.getDefaultSchemaName()isundefined(for SQLite without attached databases)- So
schemaNameisundefined
Then
DatabaseSchema.fromMetadata(metadata, this.platform, this.config, schemaName, this.em)is called.In
fromMetadata():const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema'));schemaNameisundefinedconfig.get('schema')isundefined- So
new DatabaseSchema(platform, undefined)
So
this.namewould beundefinedfor the metadata path.For
DatabaseSchema.create():const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());schemaNameisundefinedconfig.get('schema')isundefinedplatform.getDefaultSchemaName()isundefined- So
new DatabaseSchema(platform, undefined)
Same result. So the name is the same for both paths.
OK, I've exhausted my analysis. The tests pass, and I can't find an obvious bug. Let me try running all the migration tests to see if any fail:
Actually wait - let me re-read the issue more carefully. The user says "When we run orm.migrator.up(), the snapshot changes and a bunch of stuff in the snapshot is wrong and we have to revert the changes."
Maybe the issue is that after running
up(), the snapshot is rewritten but thenamefield is now missing (because it's undefined) or has a different value?Actually, let me check the snapshot file format. If the
namefield isundefined, it would be omitted from the JSON. But if the original snapshot had anamefield (from metadata path), and the new snapshot doesn't have it (from introspection path), that would cause a difference!Wait, but I showed that both paths should have
name: undefined, so both would omit it. Unless...Let me check if
getTargetSchema()could ever set a non-undefined name whileDatabaseSchema.create()doesn't.Actually, looking at the code again:
For
getTargetSchema():const schemaName = schema ?? this.config.get('schema') ?? this.platform.getDefaultSchemaName();For
DatabaseSchema.create():const schema = new DatabaseSchema(platform, schemaName ?? config.get('schema') ?? platform.getDefaultSchemaName());Both use the same logic! So the name should be the same.
But wait - what if
this.config.get('schema')returns different values in the two paths? That seems unlikely since they use the same config object...Hmm, actually I notice that
getTargetSchema()usesthis.config.get('schema')whileDatabaseSchema.create()usesconfig.get('schema')(passed as parameter). These should be the same since the Migrator passesthis.config.OK I'm stuck. Let me try a different approach - let me check if there's a bug in the code that was already identified and maybe partially fixed but not fully.
Let me search for any TODO or FIXME comments related to snapshot:
Assistant
Let me search for any potential differences in how schemas are handled between the metadata and introspection paths.
Assistant
The
grepcommand didn't find any matches. Let me search in the SQL package's schema generator files for theprunemethod andwildcardSchemareferences, which could be related to how schemas are filtered between metadata and introspection paths.Assistant
I found something interesting! The
prunemethod is called when comparing schemas ingetUpdateSchemaMigrationSQL, but NOT when storing the schema after migrations inrunMigrations. Let me investigate this further.Assistant
I found something important! In
getUpdateSchemaMigrationSQL, when introspecting the database, it passesschemas(fromtoSchema.getNamespaces()) toDatabaseSchema.create(). But inrunMigrations(when storing the snapshot afterup()), it passesundefinedforschemas!Let me look at this more carefully:
In
SqlSchemaGenerator.getUpdateSchemaMigrationSQL:const toSchema = this.getTargetSchema(options.schema, options.includeWildcardSchema); const schemas = toSchema.getNamespaces(); const fromSchema = await DatabaseSchema.create( this.connection, this.platform, this.config, options.schema, // <-- schema is passed schemas, // <-- schemas is passed undefined, this.options.skipTables, this.options.skipViews, );In
Migrator.runMigrations:const schema = await DatabaseSchema.create( this.em.getConnection(), this.em.getPlatform(), this.config, undefined, // <-- schema is NOT passed undefined, // <-- schemas is NOT passed undefined, undefined, undefined, ctx, );The difference is:
schemaName- passed in getUpdateSchemaMigrationSQL, undefined in runMigrationsschemas- passed in getUpdateSchemaMigrationSQL (from target schema namespaces), undefined in runMigrations
The
schemasparameter affects what databases/schemas are queried when introspecting. For SQLite with attached databases, this could make a big difference!But more importantly, let me look at how
DatabaseSchema.create()handles theschemasparameter:Assistant
Now I see the issue more clearly. The
DatabaseSchema.create()call inrunMigrationspassesundefinedforschemas, which means it would query ALL databases/schemas in the database. But ingetUpdateSchemaMigrationSQL, it passes the target schema's namespaces.But more importantly, I notice that when
loadViewsis called inDatabaseSchema.create():await platform.getSchemaHelper()!.loadViews(schema, connection, schemaName, ctx);It passes
schemaName(which isundefinedinrunMigrations). Let me check howloadViewshandles this...Looking at the SQLite implementation of
loadViews:if (!hasAttachedDbs && !schemaName) { const views = await connection.execute<...>(this.getListViewsSQL(), [], 'all', ctx); ... }When
schemaNameis undefined and there are no attached databases, it loads ALL views from the main database. This is the same as when called fromgetUpdateSchemaMigrationSQLwithoptions.schema.So that's not the issue either.
Wait, let me look at the
runMigrationsmethod more carefully. The key issue is that when storing the snapshot afterup(), it creates a NEW schema from introspection, but this schema might not match the metadata-based schema in certain ways.Actually, I just realized something. The
DatabaseSchema.create()inrunMigrationsdoesn't passskipTablesorskipViewsoptions. ButgetUpdateSchemaMigrationSQLpassesthis.options.skipTablesandthis.options.skipViews.If the user has configured
skipTablesorskipViewsin the schema generator options, the metadata-based snapshot would NOT include those tables/views, but the introspection-based snapshot (fromrunMigrations) WOULD include them!Let me check if this is the bug:
Assistant
Now I see the issue! In
runMigrations, when callingDatabaseSchema.create()to store the snapshot after running migrations, it passesundefinedfor ALL filtering options:takeTables = undefinedskipTables = undefinedskipViews = undefined
But in
getUpdateSchemaMigrationSQLandgetTargetSchema(), it respects theskipTablesandskipViewsoptions from the schema generator configuration.This means:
- The metadata-based snapshot (from
create()) only includes tables/views that are in the entity metadata (which respects skipTables/skipViews) - The introspection-based snapshot (from
up()) includes ALL tables/views from the database, ignoring skipTables/skipViews
This is the bug! The snapshot should be pruned using the same logic as when creating from metadata.
Let me also check if there's a
prunecall needed, similar to what's done ingetUpdateSchemaMigrationSQL.<tool_call> <function=bash_terminal> <parameter=command> cd /mikro-orm && grep -n "prune" packages/sql/src/schema/DatabaseSchema.ts </tool_call>
Assistant
Let me search for the
prunemethod in the DatabaseSchema.ts file to understand how it works and whether it needs to be called in therunMigrationsmethod.Assistant
Let me understand the
prunemethod and how it's used:Assistant
I found the issue. In
runMigrations, when storing the snapshot after running migrations, it callsDatabaseSchema.create()withundefinedfor all filtering options, while the metadata-based path (getTargetSchema()) only includes tables defined by entities. This causes the introspected snapshot to potentially include tables that shouldn't be there.Let me look at the
Migratorclass to understand how to fix this properly:Assistant
I found the bug. In
runMigrations, when storing the snapshot after running migrations,DatabaseSchema.create()is called withundefinedfor all filtering options. But ingetUpdateSchemaMigrationSQL, it passes the proper filtering options likeschemas,skipTables, andskipViewsfrom the schema generator configuration.This means the introspected snapshot can include tables/views that aren't in the entity metadata, causing the snapshot to diverge from what
create()produces.Let me implement the fix:
Assistant
I found the bug! In
runMigrations, when storing the snapshot after running migrations,DatabaseSchema.create()is called withundefinedfor all filtering options (takeTables,skipTables,skipViews). This means the introspected snapshot includes ALL tables from the database, while the metadata-based snapshot only includes tables defined by entities (and respectsskipTables/skipViewsconfiguration).Let me fix this by using the same filtering logic that's used in
getUpdateSchemaMigrationSQL:Assistant
Let me fix the edit command syntax:
Assistant
I need to open the file first before editing it. Let me open the Migrator.ts file.
Assistant
Now I need to edit the
runMigrationsmethod to fix the snapshot issue. Let me first look at the current implementation of this method.Assistant
I need to fix the
runMigrationsmethod to use the same filtering logic asgetUpdateSchemaMigrationSQL. Let me apply the fix now.Assistant
I need to edit the
runMigrationsmethod to fix the bug. Let me use theeditcommand with the file parameter since I can't seem to open files properly.Assistant
I need to verify that the fix I made is correct. Let me check if the
#schemaGenerator.optionsproperty exists and has the expected structure by looking at the SqlSchemaGenerator class.Assistant
Good, the
optionsproperty exists and is of typeNonNullable<Options['schemaGenerator']>. Let me check ifskipTablesandskipViewsare properties of this options object.Assistant
I need to find where
skipTablesandskipViewsare defined in the options interface.Assistant
I need to check if
skipTablesandskipViewsare available on theoptionsproperty ofSqlSchemaGenerator, and then run the tests to verify my fix works.Assistant
Good,
skipTablesandskipViewsare available on theoptionsproperty. Let me now run the tests to verify my fix works correctly.Assistant
Good, the snapshot-related tests pass. Let me run all the migration tests to make sure the fix doesn't break anything else.
Assistant
All 24 SQLite migration tests pass. Let me also run the main Migrator test suite to make sure nothing else is broken.
Assistant
The test failures appear to be related to database connection issues in the test setup/teardown (
afterAllhooks), not to my fix. Let me check if these tests require a specific database that's not available.Assistant
The test failures are related to MySQL database connection issues - those tests require a MySQL database which isn't available in this environment. This is not related to my fix.
Let me verify that my fix is correct by reviewing the changes and running the SQLite tests again:
Assistant
The diff looks good. Let me now verify by running the full SQLite migration test suite once more and also check the PostgreSQL tests if available:
Assistant
All 24 SQLite migration tests pass. Let me now verify the fix more thoroughly by checking:
- The key test
'snapshot serialization is stable across metadata and introspection paths (GH #7607)'passes - The test
'snapshot from create and up have identical column fields and consistent primary flag (GH #7234)'passes
Both tests pass. Let me also run the other migration-related tests to ensure I haven't broken anything:
- The key test
Assistant
The MSSQL tests fail due to database connection issues (MSSQL server not available in this environment). This is expected and not related to my fix.
Let me verify my fix is complete by reviewing the changes and the checklist:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.
Identical first prompt
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
Qwen/Qwen3.6-27BCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- xiaomi/mimo-v2.5-proCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-solCompare side by side →
- gpt-5.6-lunaCompare side by side →
- gpt-5.6-lunaCompare side by side →