Fog Project Documentation (FOG Docs)

CSV Import / Export

Properties3
descriptionDescribes FOG 1.6's csv import/export format (headered or positional, associations, name-resolved foreign keys) and why we chose this method
tags1_6-changes, import, export, csv, configuration, management
aliasesCSV Import / Export, Import CSV, Export CSV

11 min read

Permalink

Edit this page · View source

CSV Import / Export

FOG 1.6

This page describes FOG 1.6. CSV import/export itself isn’t new — FOG 1.5 has had it for a long time — but 1.6 reworks the format significantly. See the 1.5 version of this page for what 1.5 does: raw, positional columns with no header row, no associations column, and no foreign-key name resolution.

FOG can mass-import and export most management objects (hosts, images, snapins, groups, printers, users, modules, storage groups and storage nodes) as CSV files from each object’s Import / Export page — see Host Management for where to find the Import Hosts page in the UI.

Two ways to map columns:

  1. Headered — if the file’s first row names the columns (which is what Export now produces), FOG maps columns by name. Columns may be in any order, and you may include only the columns you care about.
  2. Positional (header-less) — if there is no header row, FOG maps columns by position, in the exact order Export produces.

A header row is auto-detected, and there is also a “First row is a header” checkbox on the Import page to force it. Either way, the simplest, safest workflow is Export → edit → Import.

Table of contents

Exporting

Each object type has an Export page (for example Hosts → Export Hosts) with a row of buttons above the table:

  • CSV (All) — the recommended way to get a complete, import-ready file. It exports every item that matches the current search, server-side, as a CSV download — not just the rows visible on screen. The file always begins with a header row of column names, so it re-imports by name with no reordering.
  • Copy, Excel, Print — these act only on the rows currently loaded in the browser (the on-screen page, or what you have scrolled through). They are handy for quick, partial grabs rather than a full export.
  • Column Visibility — show/hide columns in the on-screen table. It does not change the CSV (All) output, which always contains the full canonical column set for the class (the per-class layouts below).

Tip

Type in the table’s search box first to scope the export. CSV (All) honors the active search, so you can export just the matching subset — for example, every host whose name contains lab.

The exported columns — and the optional trailing associations column — are described per class below.

Header row vs. positional

Headered (recommended). If the first row contains the column names, FOG maps each subsequent row by name:

  • Any order — columns may appear in any order.
  • Partial — include only the columns you want to set; omitted columns keep their defaults. (The identity columns are still required: name for every class, plus primac for hosts.)
  • Auto-detected — a first row whose cells are all recognized column names is treated as a header automatically. The “First row is a header” checkbox forces header mode (and reports any unrecognised header names as ignored).
  • Names are matched case-insensitively (ProductKey == productkey).
  • Export emits a header row by default, so an Export file re-imports by name with no editing of column positions.

Valid header names are the column names in the per-class tables below, plus primac (hosts) and associations (where supported).

Example (headered, partial, reordered):

name,primac,associations
PC-Lab-01,00:11:22:33:44:55,groups:Lab A|Lab B;snapins:7zip

Positional (header-less). With no header row, FOG maps columns by position, in the exact order Export produces (the per-class tables below).

General format rules

  • Delimiter: standard comma (,). Use normal CSV quoting (wrap a field in double quotes) if a value itself contains a comma.
  • Column count. In positional mode a row may contain up to the number of columns the class defines (plus the optional trailing associations column). In header mode a data row must not have more columns than the header. Either way, too many columns rejects the import with “Invalid data being parsed.”
  • Order matches Export (positional mode). The per-class tables list it.
  • Pipe (|) is the multi-value separator inside a single field (e.g. a host’s MAC list, and association values). This mirrors how MAC lists have always been delimited.
  • Existing items are skipped. Importing a host whose MAC already exists, or an item whose unique name already exists, fails that row (the rest continue).

A note on sensitive / special fields

FieldClassBehavior
primacHostRequired, first column. Pipe-separated MAC list; the first MAC becomes primary, the rest are added as additional MACs.
productKeyHostAuto-detected: accepts plaintext, base64, or the AES-encrypted form an export produces.
passwordUserStored encrypted on import.
imageID, osID, imageTypeID, …variousNumeric foreign keys. Name-resolved like the associations column: Export emits the referenced object’s name, and Import accepts an id or a (case-insensitive) name. See Foreign-key columns below.

The associations column

Every class that supports relationships accepts one optional final column named associations. It lets a single row carry related objects (a host’s groups/snapins/printers, a group’s member hosts, etc.) alongside the object’s own fields.

Cell format

label:value|value|value;label:value
  • ; separates association types.
  • : separates a type’s label from its values.
  • | separates individual values within a type.

Example (a host):

groups:Lab A|Lab B;snapins:7zip|Chrome;printers:FrontDesk

Escaping delimiters in names

If an object’s name legitimately contains one of the structural characters (;, :, |) — or a backslash — prefix it with a backslash (\) so it is treated as a literal part of the name rather than a separator. Export does this automatically; you only need to write escapes by hand when authoring an import file.

Character in a nameWrite it as
\ (backslash)\\
;\;
:\:
|`\

Example — a group literally named Lab A|Lab B and a printer named Room 3: Floor 2:

groups:Lab A\|Lab B;printers:Room 3\: Floor 2

This parses as the single group name Lab A|Lab B and the single printer name Room 3: Floor 2. A \\ is read as one literal backslash, and only unescaped delimiters split the cell. Labels (groups, printers, …) are fixed keys and are never escaped.

Resolution rules

  • Id or name. Each value is resolved by numeric id first, then by (case-insensitive) name. Names make a file portable between servers whose ids differ — which is the whole point of an export→import migration.
  • Lenient (warn-and-skip). If a value can’t be resolved (e.g. a snapin name that doesn’t exist on this server), the row still imports; only that one association is skipped and a warning is reported. The import result shows “Import Succeeded With Warnings” in that case.
  • Export emits names, not ids, for portability.

Supported labels per class

ClassLabelsResolves against
Hostgroups, snapins, printers, modules, location¹, site²Group, Snapin, Printer, Module, Location, Site — by name
ImagestoragegroupsStorage Group — by name
SnapinstoragegroupsStorage Group — by name
GrouphostsHost — by name (the group’s members)
PrinterhostsHost — by name (hosts the printer is assigned to)
User, Module, Storage Group, Storage Node(none)—

¹ location is provided by the Location plugin and only appears when that plugin is installed. A host has a single location, so only the first value is used.

² site is provided by the Site plugin and only appears when that plugin is installed. A host has a single site, so only the first value is used.

Note: because ;, : and | are structural, an object name that literally contains one of those characters must be escaped with a backslash (see Escaping delimiters in names above). Export escapes them for you; referencing such an object by id also avoids the issue entirely.

Foreign-key columns

A few columns are foreign keys — a numeric id that points at another object:

ClassColumn(s)Points at
HostimageIDImage
ImageosID, imageTypeID, imagePartitionTypeIDOS, Image Type, Image Partition Type
Storage NodestoragegroupIDStorage Group

These now behave like the associations column, so a file moves cleanly between servers whose ids differ:

  • Export emits the referenced object’s name (e.g. the imageID column holds Windows 10, not 4).
  • Import resolves id-first, then (case-insensitive) name. Numeric exports from older versions still re-import unchanged, because the id is tried first.
  • Empty or 0 means “no reference” and is kept as-is — no warning.
  • Lenient. If a name can’t be resolved on this server, the column keeps its default and a warning is reported; the row still imports (“Import Succeeded With Warnings”).

Unlike the associations column, a foreign-key column is an ordinary standalone CSV field, so a name containing ;, : or | needs no backslash escaping — normal CSV quoting (wrapping the field in double quotes) covers commas and the like.

Per-class column layouts

Column order below is generated from the live schema and matches Export output exactly. associations is always the optional final column where supported.

Host

primac is required and first; associations is optional and last.

#Column#Column
0primac (MAC list, |)16printerLevel
1name17kernelArgs
2description18kernel
3ip19kernelDevice
4imageID20init
5building21pending
6createdTime22pub_key
7deployed23sec_tok
8createdBy24sec_time
9useAD25pingstatus
10ADDomain26biosexit
11ADOU27efiexit
12ADUser28enforce
13ADPass29token
14ADPassLegacy30tokenlock
15productKey31associations (optional)

Associations: groups, snapins, printers, modules, location¹, site².

Image

0:name 1:description 2:path 3:createdTime 4:createdBy 5:building 6:size 7:imageTypeID 8:imagePartitionTypeID 9:osID 10:deployed 11:format 12:magnet 13:protected 14:compress 15:isEnabled 16:toReplicate 17:srvsize 18:associations (optional → storagegroups)

Snapin

0:name 1:description 2:file 3:args 4:createdTime 5:createdBy 6:reboot 7:shutdown 8:runWith 9:runWithArgs 10:protected 11:isEnabled 12:toReplicate 13:hide 14:timeout 15:packtype 16:hash 17:size 18:anon3 19:associations (optional → storagegroups)

Group

0:name 1:description 2:createdBy 3:createdTime 4:building 5:kernel 6:kernelArgs 7:kernelDevice 8:init 9:associations (optional → hosts)

Printer

0:name 1:description 2:port 3:file 4:model 5:config 6:configFile 7:ip 8:pAnon2 9:pAnon3 10:pAnon4 11:pAnon5 12:associations (optional → hosts)

User

0:name 1:password (stored encrypted) 2:createdTime 3:createdBy 4:type 5:display 6:api 7:token

(No associations column.)

Module

0:name 1:shortName 2:description 3:isDefault

(No associations column.)

Storage Group

0:name 1:description

(No associations column.)

Storage Node

0:name 1:description 2:isMaster 3:storagegroupID 4:isEnabled 5:isGraphEnabled 6:path 7:ftppath 8:bitrate 9:helloInterval 10:snapinpath 11:sslpath 12:ip 13:maxClients 14:user 15:pass 16:key 17:interface 18:bandwidth 19:webroot 20:graphcolor

(No associations column.)

Worked example (host)

The same host — assigned to two groups, three snapins and one printer — shown both ways.

Positional (full column set, header-less):

00:11:22:33:44:55|00:11:22:33:44:66,PC-Lab-01,Front lab PC,,4,,,,,0,,,,,,,5,,,,,0,,,,1,0,0,0,,,"groups:Lab A|Lab B;snapins:7zip|Chrome|VLC;printers:FrontDesk"

The first MAC is primary, the second is an additional MAC; the trailing quoted field is the associations column.

Headered (only the columns you need, any order):

primac,name,description,associations
00:11:22:33:44:55|00:11:22:33:44:66,PC-Lab-01,Front lab PC,"groups:Lab A|Lab B;snapins:7zip|Chrome|VLC;printers:FrontDesk"

Plugin extensibility

Plugins can register their own association types so they participate in both import and export, without patching core:

  • IMPORT_ASSOCIATIONS — fired while the per-class association config is built. A hook receives childClass and a config array (by reference) and may add an entry:

    $arguments['config']['mylabel'] = [
    'class'     => 'MyClass',    // resolved by id or name
    'namefield' => 'name',
    'get'       => 'myprop',     // item property holding ids (export);
                                 //   or a callable fn($item) returning names
    'apply'     => 'addMyThing', // item method taking an array of ids (import);
                                 //   or a callable fn($item, array $ids)
];

  • EXPORT_ASSOCIATIONS — fired while a row’s associations cell is built; receives the parts array (by reference) for last-mile tweaks. To keep large exports fast this event only fires when a listener is actually registered, so a row with no EXPORT_ASSOCIATIONS listeners never hydrates its object.

  • EXPORT_ASSOCIATIONS_PRIME — fired once per export (not per row), before any cell is built, with childClass and the full list of row ids. Plugins can use it to bulk-resolve their label for the whole result set in a couple of queries and hand the result back via FOGPage::primeAssociationLabel($childClass, $label, $byParentId), where $byParentId maps each parent id to an array of names. Any label that is not primed simply falls back to the per-row get path, so listening is purely an optimization.

The Location plugin (AddLocationImport, in its src/Hooks/) is the reference implementation, registering a single-valued location type for hosts and listening for EXPORT_ASSOCIATIONS_PRIME to batch that label. The retired Site plugin used to follow the same pattern for a host’s site; sites moved into FOG itself in 1.6 and that hook did not move with them, so site is currently not a CSV association label for any class.

Export performance

Building the associations column naively costs roughly five queries per row (a fresh object plus one lazy lookup per label). To avoid that N+1, the exporter bulk-primes every row’s association names once before formatting begins: core host labels (groups, snapins, printers, modules) are loaded with a single IN() query per association class, cached per parent id, and each row’s cell is then built from that cache with no per-row object or query. Plugins opt into the same batching via EXPORT_ASSOCIATIONS_PRIME (above). The output is identical to the per-row path — this only changes how the names are fetched.

See issues #828 (design and history) and #857 (export batching) for details.

Graph View

Backlinks