Table of Contents

Class SqliteNcLuggageStorage

Namespace
Hi.SqliteUtils
Assembly
HiNc.dll

Internal Use Only. The luggage archive of a recorded play result (the NC play result recording's P6): a second SQLite file beside result.db, luggage.db, holding the step cache's force arrays (MillingForceLuggageColumn) and engagements (LayerMillingEngagementColumn) of the result's steps, byte for byte as the cache stores them – the no-cut record of an untouched step as its 8-byte header, the engagement as its WriteBin bytes – so that a review reads them where the cache, reset when the review opened, holds nothing. The removal geometry (SubstractionData) is not archived: the archive's table has no such column. The two briefs are not here either: the result file holds them in its StepBriefChunk.

Archive(string, string, int, int, DateTime, IProgress<IMessage>, NcLuggageArchiveOptions) writes the file at the recording's completion, on the completing thread, before the result file is completed: the step cache is attached to the new file's connection and the rows of the result's step range are copied with INSERT INTO … SELECT, SQLite streaming the bytes inside itself (nothing of a row passes through managed memory). First the disk gate: the free space of the result folder's drive is compared with the cache file's size (the archive is a column subset of it, so it never exceeds it: a free upper bound); when that does not fit, the two columns' bytes over the range are scanned cold (TOTAL(LENGTH(…))) and multiplied by the headroom; when that does not fit either, nothing is written and the result completes without an archive. The page size is chosen from the touched rows' lengths (16 KiB for rows under 4,000 bytes, 4 KiB at or above). The file is in rollback-journal mode from the start and stays so; its Meta writes Finalized 1 as its last statement, after the copy's one transaction committed, so a file whose copy never completed is told apart (Finalized) and never read as the result's luggage. The result file it belongs to is named by that file's creation time (ResultCreatedUtc), which the review compares before it reads.

Amend(string, string, DateTime, IReadOnlyList<(int Begin, int End)>, CancellationToken, NcLuggageArchiveOptions) rewrites the rows of given step ranges from the step cache again (the step-physics recompute replaced their force arrays after the play); OpenRead(string) opens the archive for a review (SqliteNcLuggageReader). The table and its columns carry the step cache's names (StepLuggageTableName, MillingForceLuggageColumn, LayerMillingEngagementColumn), so the diagnostic SQL written for the cache reads the archive too. 

public static class SqliteNcLuggageStorage
Inheritance
SqliteNcLuggageStorage
Inherited Members

Fields

ApplicationId

The file's application_id: ASCII HiNL. A result file is HiNR (ApplicationId).

public const int ApplicationId = 1214860876

Field Value

int

DefaultChunkSteps

The steps one copy statement takes: 65,536.

public const int DefaultChunkSteps = 65536

Field Value

int

DefaultProgressEverySteps

The steps between two progress messages of the copy: 1,048,576.

public const long DefaultProgressEverySteps = 1048576

Field Value

long

DefaultSampleRows

The touched rows the page size is sampled from: 4,096.

public const int DefaultSampleRows = 4096

Field Value

int

LargePageRowBytes

The touched row length (force bytes plus engagement bytes) under which the file takes 16 KiB pages, at or above which 4 KiB: 4,000 bytes.

public const int LargePageRowBytes = 4000

Field Value

int

LargePageSize

The page size of a file whose touched rows are short (or that has none): 16 KiB.

public const int LargePageSize = 16384

Field Value

int

LayerMillingEngagementColumn

The engagement column: the step cache's column (LayerMillingEngagementColumn), its WriteBin bytes.

public const string LayerMillingEngagementColumn = "LayerMillingEngagementData"

Field Value

string

MetaTableName

The Meta table.

public const string MetaTableName = "Meta"

Field Value

string

MillingForceLuggageColumn

The force column: the step cache's column (MillingForceLuggageColumn), in the cache's stored form.

public const string MillingForceLuggageColumn = "MillingForceLuggageData"

Field Value

string

SchemaVersion

The schema version this build writes and reads (user_version).

public const int SchemaVersion = 1

Field Value

int

SmallPageSize

The page size of a file whose touched rows are long: 4 KiB.

public const int SmallPageSize = 4096

Field Value

int

StepLuggageTableName

The one data table, a row a step: the step cache's table name (StepLuggageTableName).

public const string StepLuggageTableName = "StepLuggage"

Field Value

string

Methods

Amend(string, string, DateTime, IReadOnlyList<(int Begin, int End)>, CancellationToken, NcLuggageArchiveOptions)

Rewrites the rows of the given step ranges from the step cache again – the step-physics recompute replaced their force arrays in the cache after the play, and the review is to see what the result file's amended briefs go with – in chunks of ChunkSteps steps, each a transaction of its own (INSERT OR REPLACE … SELECT, the whole row: the engagement column is copied again as it is), so the file's rollback journal is bounded by a chunk's pages; the ranges in the order given, each clipped to the archive's own step range, the token looked at before every chunk. Each chunk that wrote stamps LuggageAmendedUtc with this amend's start and the Meta's five counts as they stand after it, kept by difference in the same transaction (the chunk's rows counted before and after the rewrite: a range may bring rows the copy did not hold, a step the flush had lost and the recompute filled), so a chunk that fails leaves the committed chunks counted; a Meta without the counts has them taken over the whole table at the end. No file at the path, one this build cannot read (not an archive, a 0-byte file the process died before writing), one whose copy never completed, or one that belongs to another result (its ResultCreatedUtc is not resultCreatedUtc: a file put into the folder by hand) writes nothing and answers Found false: the caller drops its marks (the review says what it makes of such a file when it opens).

public static NcLuggageAmendResult Amend(string destinationPath, string sourcePath, DateTime resultCreatedUtc, IReadOnlyList<(int Begin, int End)> ranges, CancellationToken cancellationToken, NcLuggageArchiveOptions options = null)

Parameters

destinationPath string

The archive (luggage.db).

sourcePath string

The step cache file; it must exist.

resultCreatedUtc DateTime

The creation time of the result file the archive must belong to (its Meta.CreatedUtc).

ranges IReadOnlyList<(int First, int Last)>

The step ranges to rewrite, [Begin, End), in the order to take them.

cancellationToken CancellationToken

Looked at before each chunk: a cancelled token ends the amend there (the chunks before it are in; the range it stopped in is not counted as finished).

options NcLuggageArchiveOptions

The chunk (ChunkSteps); null for Default.

Returns

NcLuggageAmendResult

What was rewritten.

Exceptions

ArgumentNullException

A path or the ranges are null.

FileNotFoundException

The step cache file does not exist.

SqliteException

SQLite failed; the chunks committed stay, the one in flight is rolled back.

Archive(string, string, int, int, DateTime, IProgress<IMessage>, NcLuggageArchiveOptions)

Writes the luggage archive of a result: the force and engagement columns of the step cache's rows of [firstStepIndex, lastStepIndex], every row of the range (a row whose two columns are NULL – a step the physics did not run on – included, so that a reader tells “no force for the step” from “no row”), into a new file at destinationPath, byte for byte. The disk gate runs first (the summary of the class): a drive without the room answers Skipped and leaves no file. The copy is one transaction on one connection: the cache is attached, read under one snapshot (a reset of the cache by another service meanwhile cannot reach into it: on Windows the delete fails while the file is held and the reset wipes the tables instead; on Linux the file is unlinked and recreated while this connection reads the old one to its end, and SQLite, finding the path's inode changed when the attached database closes, neither checkpoints it nor deletes the side files at the path), and the new file's pages are appended outside its rollback journal, so the transaction does not grow the journal. Progress is said on progress: NcResult-Record–LuggageArchiving when the copy begins, NcResult-Record–LuggageProgress every ProgressEverySteps steps; the caller says the end. A failure – SQLite (a full disk among others), the file system – throws after what the copy left is deleted, so a failed archive leaves no file for a review to find.

public static NcLuggageArchiveResult Archive(string sourcePath, string destinationPath, int firstStepIndex, int lastStepIndex, DateTime resultCreatedUtc, IProgress<IMessage> progress, NcLuggageArchiveOptions options = null)

Parameters

sourcePath string

The step cache file (DatabasePath); it must exist.

destinationPath string

The archive to write; the file must not exist (a stale side file beside it is deleted first).

firstStepIndex int

The first step index to copy, at least 0.

lastStepIndex int

The last step index to copy (inclusive), at least firstStepIndex.

resultCreatedUtc DateTime

The creation time of the result file the archive belongs to (Meta.CreatedUtc of result.db), written into the archive's Meta for the review to compare.

progress IProgress<IMessage>

Where the copy's progress is said; null for silence.

options NcLuggageArchiveOptions

The gate's free-space probe and reserve, the chunk, the sample, the progress stride and the headroom; null for Default.

Returns

NcLuggageArchiveResult

The outcome with its counts (archived) or the bytes the gate compared (skipped).

Exceptions

ArgumentNullException

sourcePath or destinationPath is null.

ArgumentOutOfRangeException

The step range is negative or empty.

FileNotFoundException

The step cache file does not exist.

IOException

The archive file exists already, or its folder or side files could not be dealt with.

SqliteException

SQLite failed (the disk full, the cache unreadable, an I/O error); what was written is deleted.

OpenRead(string)

Opens a luggage archive read-only, on a connection of its own that is never pooled: the file must exist, carry the ApplicationId, a user_version of 1 to SchemaVersion and a Meta table, else it is refused; its Meta is read now. A file whose copy never completed opens (Finalized false): the caller decides not to read it.

public static SqliteNcLuggageReader OpenRead(string databasePath)

Parameters

databasePath string

The archive's path.

Returns

SqliteNcLuggageReader

The reader.

Exceptions

ArgumentNullException

databasePath is null.

FileNotFoundException

The file does not exist.

InvalidDataException

The file is not a luggage archive this build reads.

SqliteException

SQLite could not open the file.