Examples

Examples

Most examples use literal data in an IOBuffer so the documentation build executes them. A String source is a file path or URL; wrap literal text in IOBuffer. Load DataStrings when you want to name the default text type.

Start here: read, inspect, and write

Install CSV with import Pkg; Pkg.add("CSV"). Install any other package named in an example before using it. File paths and URLs below are placeholders; replace them with your own sources.

using CSV

file = CSV.File(IOBuffer("id,name\n1,Ada\n2,Grace\n"))
names(file)                 # column names
collect(file.name)          # access a column
first(file).id              # access a value in a row

output = IOBuffer()
CSV.write(output, file)
String(take!(output))

"id,name\n1,Ada\n2,Grace\n"

CSV.File returns a table with parsed columns. Text uses DataStrings.DataString by default; pass stringtype=String for ordinary Julia strings. Use CSV.read(source, DataFrame) when you need a DataFrame (see Read into another table package).

Non-UTF-8 character encodings

# assume I have csv text data encoded in ISO-8859-1 encoding
# I load the StringEncodings package, which provides encoding conversion functionality
using CSV, StringEncodings

# I open my `iso8859_encoded_file.csv` with the `enc"ISO-8859-1"` encoding
# and pass the opened IO object to `CSV.File`; the whole input is read into
# memory through the converting stream, then parsed
file = open("iso8859_encoded_file.csv", enc"ISO-8859-1") do io
    CSV.File(io)
end

Concatenate multiple inputs at once

using CSV

# in this case, I have a vector of delimited data inputs that each have
# matching schema (the same column names and types). I'd like to process all
# of the inputs together and vertically concatenate them into one "long" table.
data = [
    "a,b,c\n1,2,3\n4,5,6\n",
    "a,b,c\n7,8,9\n10,11,12\n",
    "a,b,c\n13,14,15\n16,17,18",
]

# I can just pass a `Vector` of inputs, in this case `IOBuffer(::String)`, but it
# could also be a `Vector` of any valid input source, like `AbstractVector{UInt8}`,
# filenames, `IO`, etc. Inputs are parsed on separate tasks and concatenated
# in order. The first input defines the output columns. Later missing columns
# are filled with `missing`, and later extra columns are ignored. To
# send the parsed columns directly to a sink function, we can use `CSV.read`, like
# `df = CSV.read(map(IOBuffer, data), DataFrame)`.
f = CSV.File(map(IOBuffer, data))

6-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = 4, b = 5, c = 6)
 (a = 7, b = 8, c = 9)
 (a = 10, b = 11, c = 12)
 (a = 13, b = 14, c = 15)
 (a = 16, b = 17, c = 18)

Pass source=:origin to add a column that records which input each row came from (paths for file sources, "<source 1>" and so on otherwise), or source=:origin => labels to supply your own labels.

Gzipped input

# assume I have csv text data compressed via gzip
# no additional packages are needed; CSV.jl detects gzip by its magic bytes
using CSV

# pass name of gzipped input file directly; data is decompressed into memory,
# then parsed
file = CSV.File("data.gz")

Delimited data in a string

using CSV

# I have csv data in a string I want to parse
data = """
a,b,c
1,2,3
4,5,6
"""

# Calling `IOBuffer` on a string returns an in-memory IO object
# of the string data, which can be passed to `CSV.File` for parsing
file = CSV.File(IOBuffer(data))

2-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = 4, b = 5, c = 6)

Data from the web/a url

using CSV

# an http(s) URL string is downloaded to a temporary file, read into memory,
# and parsed; the temporary file is removed afterwards
url = "https://example.com/data.csv"
file = CSV.File(url)

# alternatively, fetch the bytes yourself with the HTTP.jl package and pass
# the response body (a `Vector{UInt8}`) directly to `CSV.File`
using HTTP
http_response = HTTP.get(url)
file = CSV.File(http_response.body)

# or download with the Downloads stdlib and pass the temporary file path
using Downloads
file = CSV.File(Downloads.download(url))

Reading from a zip file

using ZipArchives, Mmap, CSV, DataFrames

a = DataFrame(a = 1:3)
CSV.write("a.csv", a)

# write directly into a zip archive
ZipWriter("a.zip") do z
    zip_newfile(z, "a.csv"; compress=true)
    CSV.write(z, a)
end

# read file from zip archive
z = ZipReader(open(mmap, "a.zip"))

# identify the right file in zip
a_copy = CSV.read(zip_readentry(z, "a.csv"), DataFrame)

a == a_copy

Column names on 2nd row

using CSV

data = """
descriptive row with information about the file that we'd like to ignore
a,b,c
1,2,3
4,5,6
"""

# by passing header=2, parsing will ignore the 1st row entirely
# then parse the column names on row 2, then by default, it assumes
# the data starts on the row after the column names (row 3 in this case)
# which is correct for this case
file = CSV.File(IOBuffer(data); header=2)

2-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = 4, b = 5, c = 6)

No column names in data

using CSV

# in this case, our data doesn't have any column names
data = """
1,2,3
4,5,6
"""

# by passing `header=false`, parsing won't worry about looking for column names
# anywhere, but instead just start parsing the data and generate column names
# as needed, like `Column1`, `Column2`, and `Column3` in this case
file = CSV.File(IOBuffer(data); header=false)

2-element CSV.File:
 (Column1 = 1, Column2 = 2, Column3 = 3)
 (Column1 = 4, Column2 = 5, Column3 = 6)

Manually provide column names

using CSV

# in this case, our data doesn't have any column names
data = """
1,2,3
4,5,6
"""

# instead of passing `header=false` and getting auto-generated column names,
# we can instead pass the column names ourselves
file = CSV.File(IOBuffer(data); header=["a", "b", "c"])

# we can also pass the column names as Symbols; a copy of the manually provided
# column names will always be made and then converted to `Vector{Symbol}`
file = CSV.File(IOBuffer(data); header=[:a, :b, :c])

2-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = 4, b = 5, c = 6)

Multi-row column names

using CSV

# in this case, our column names are `col_a`, `col_b`, and `col_c`,
# but split over the first and second rows
data = """
col,col,col
a,b,c
1,2,3
4,5,6
"""

# by passing a collection of integers, parsing will parse each row in the collection
# and concatenate the values for each column, separating rows with `_` character
file = CSV.File(IOBuffer(data); header=[1, 2])

2-element CSV.File:
 (col_a = 1, col_b = 2, col_c = 3)
 (col_a = 4, col_b = 5, col_c = 6)

Normalizing column names

using CSV

# in this case, our data are single letters, with column names of "1", "2", and "3"
# A single digit isn't a valid identifier in Julia, meaning we couldn't do something
# like `1 = 2 + 2`, where `1` would be a variable name
data = """
1,2,3
a,b,c
d,e,f
h,i,j
"""

# in order to have valid identifiers for column names, we can pass
# `normalizenames=true`, which result in our column names becoming "_1", "_2", and "_3"
# note this isn't required, but can be convenient in certain cases
file = CSV.File(IOBuffer(data); normalizenames=true)

# we can access the first column like
file._1

3-element DataStrings.StringVector{DataStrings.DataString}:
 "a"
 "d"
 "h"

# another example where we may want to normalize is column names with spaces in them
data = """
column one,column two, column three
1,2,3
4,5,6
"""

# normalizing will result in column names like "column_one", "column_two" and "column_three"
file = CSV.File(IOBuffer(data); normalizenames=true)

2-element CSV.File:
 (column_one = 1, column_two = 2, column_three = 3)
 (column_one = 4, column_two = 5, column_three = 6)

Skip to specific row where data starts

using CSV

# our data has a first row that we'd like to ignore; our data also doesn't have
# column names, so we'd like them to be auto-generated
data = """
descriptive row that gives information about the data that we'd like to ignore
1,2,3
4,5,6
"""

# with no column names in the data, we first pass `header=false`; by itself,
# this would result in parsing starting on row 1 to parse the actual data;
# but we'd like to ignore the first row, so we pass `skipto=2` to skip over
# the first row; our column names will be generated like `Column1`, `Column2`, `Column3`
file = CSV.File(IOBuffer(data); header=false, skipto=2)

2-element CSV.File:
 (Column1 = 1, Column2 = 2, Column3 = 3)
 (Column1 = 4, Column2 = 5, Column3 = 6)

Skipping trailing useless rows

using CSV

# our data has column names of "a", "b", and "c"
# but at the end of the data, we have 2 rows we'd like to ignore while parsing
# since they're not properly delimited
data = """
a,b,c
1,2,3
4,5,6
7,8,9
totals: 12, 15, 18
grand total: 45
"""

# `footerskip=2` excludes the last two rows from value parsing.
# CSV still scans the input to find the row boundaries.
file = CSV.File(IOBuffer(data); footerskip=2)

3-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = 4, b = 5, c = 6)
 (a = 7, b = 8, c = 9)

Reading transposed data

using CSV

# our data is transposed, meaning our column names are in the first column,
# with the data for column "a" all on the first row, data for column "b"
# all on the second row, and so on.
data = """
a,1,4,7
b,2,5,8
c,3,6,9
"""

# by passing `transpose=true`, parsing will look for column names in the first
# column of data, then parse each row as a separate column
file = CSV.File(IOBuffer(data); transpose=true)

3-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = 4, b = 5, c = 6)
 (a = 7, b = 8, c = 9)

Ignoring commented rows

using CSV

# here, we have several non-data rows that all begin with the "#" string
data = """
# row describing column names
a,b,c
# row describing first row of data
1,2,3
# row describing second row of data
4,5,6
"""

# we want to ignore these "commented" rows
file = CSV.File(IOBuffer(data); comment="#")

2-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = 4, b = 5, c = 6)

Ignoring empty rows

using CSV

# here, we have a "gap" row in between the first and second row of data
# by default, these "empty" rows are ignored, but in our case, this is
# how a row of data is input when all columns have missing/null values
# so we don't want those rows to be ignored so we can know how many
# missing cases there are in our data
data = """
a,b,c
1,2,3

4,5,6
"""

# `ignoreemptyrows=false` keeps the empty row and fills its columns with missing;
# a kept empty row is not a parse problem
file = CSV.File(IOBuffer(data); ignoreemptyrows=false)
file

3-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = missing, b = missing, c = missing)
 (a = 4, b = 5, c = 6)

Including/excluding columns

using CSV

# simple dataset, but we know column "b" isn't needed
# so we'd like to save time by having parsing ignore it completely
data = """
a,b,c
1,2,3
4,5,6
7,8,9
"""

# there are quite a few ways to provide the select/drop arguments
# so we provide an example of each, first for selecting the columns
# "a" and "c" that we want to include or keep from parsing
file = CSV.File(IOBuffer(data); select=[1, 3])
file = CSV.File(IOBuffer(data); select=[:a, :c])
file = CSV.File(IOBuffer(data); select=["a", "c"])
file = CSV.File(IOBuffer(data); select=[true, false, true])
file = CSV.File(IOBuffer(data); select=r"[ac]")
# now examples of dropping, when we'd rather specify the column(s)
# we'd like to drop/exclude from parsing
file = CSV.File(IOBuffer(data); drop=[2])
file = CSV.File(IOBuffer(data); drop=[:b])
file = CSV.File(IOBuffer(data); drop=["b"])
file = CSV.File(IOBuffer(data); drop=[false, true, false])
file = CSV.File(IOBuffer(data); drop=:b)

3-element CSV.File:
 (a = 1, c = 3)
 (a = 4, c = 6)
 (a = 7, c = 9)

select and drop take a list, one name or index, or a Regex; the selected columns keep their file order.

Limiting number of rows from data

using CSV

# here, we have quite a few rows of data (relative to other examples, lol)
# but we know we only need the first 3 for the analysis we need to do
# so instead of spending the time parsing the entire file, we'd like
# to just read the first 3 rows and ignore the rest
data = """
a,b,c
1,2,3
4,5,6
7,8,9
10,11,12
13,14,15
"""

# Only the first three data rows have their values parsed and returned.
# CSV still reads or maps the source and builds its structural index.
# The limit is exact at every thread count.
file = CSV.File(IOBuffer(data); limit=3)

3-element CSV.File:
 (a = 1, b = 2, c = 3)
 (a = 4, b = 5, c = 6)
 (a = 7, b = 8, c = 9)

Specifying custom missing strings

using CSV

# in this data, our first column has "missing" values coded with -999
# but our score column has "NA" instead
# we'd like either of those values to show up as `missing` after we parse the data
data = """
code,age,score
0,21,3.42
1,42,6.55
-999,81,NA
-999,83,NA
"""

# by passing missingstring=["-999", "NA"], parsing will check each cell if it matches
# either string in order to set the value of the cell to `missing`
file = CSV.File(IOBuffer(data); missingstring=["-999", "NA"])

4-element CSV.File:
 (code = 0, age = 21, score = 3.42)
 (code = 1, age = 42, score = 6.55)
 (code = missing, age = 81, score = missing)
 (code = missing, age = 83, score = missing)

An unquoted empty field is always missing; missingstring adds spellings.

String delimiter

using CSV

# our data has two columns, separated by double colon
# characters ("::")
data = """
col1::col2
1::2
3::4
"""

# we can pass a single character or string for delim
file = CSV.File(IOBuffer(data); delim="::")

2-element CSV.File:
 (col1 = 1, col2 = 2)
 (col1 = 3, col2 = 4)

Fixed width files

using CSV

# This is an example of "fixed width" data, where each
# column is the same number of characters away from each
# other on each row. Fields are "padded" with extra
# delimiters (in this case `' '`) so that each column is
# the same number of characters each time
data = """
col1    col2 col3
123431  2    3421
2355    346  7543
"""
# In addition to our `delim`, we can pass
# `ignorerepeated=true`, which tells parsing that
# consecutive delimiters should be treated as a single
# delimiter.
file = CSV.File(IOBuffer(data); delim=' ', ignorerepeated=true)

2-element CSV.File:
 (col1 = 123431, col2 = 2, col3 = 3421)
 (col1 = 2355, col2 = 346, col3 = 7543)

Turning off quoted cell parsing

using CSV

# by default, cells like the 1st column, 2nd row
# will be treated as "quoted" cells, where they start
# and end with the quote character '"'. The quotes will
# be removed from the final parsed value
# we may, however, want the "raw" value and _not_ ignore
# the quote characters in the final value
data = """
a,b,c
"hey",2,3
there,4,5
sailor,6,7
"""

# we can "turn off" the detection of quoted cells
# by passing `quoted=false`
file = CSV.File(IOBuffer(data); quoted=false)

3-element CSV.File:
 (a = "\"hey\"", b = 2, c = 3)
 (a = "there", b = 4, c = 5)
 (a = "sailor", b = 6, c = 7)

Quoted & escaped fields

using CSV

# In this data, we have a few "quoted" fields, which means the field's value starts and ends with `quotechar` (or
# `openquotechar` and `closequotechar`, respectively). Quoted fields allow the field to contain characters that would otherwise
# be significant to parsing, such as delimiters or newline characters. When quoted, parsing will ignore these otherwise
# significant characters until the closing quote character is found. For quoted fields that need to also include the quote
# character itself, an escape character is provided to tell parsing to ignore the next character when looking for a close quote
# character. In the syntax examples, the keyword arguments are passed explicitly, but these also happen to be the default
# values, so just doing `CSV.File(IOBuffer(data))` would result in successful parsing.
data = """
col1,col2
"quoted field with a delimiter , inside","quoted field that contains a
newline and ""inner quotes\"\"\"
unquoted field,unquoted field with "inner quotes"
"""

file = CSV.File(IOBuffer(data); quotechar='"', escapechar='"')

file = CSV.File(IOBuffer(data); openquotechar='"', closequotechar='"', escapechar='"')

2-element CSV.File:
 (col1 = "quoted field with a delimiter , inside", col2 = "quoted field that contains a\nnewline and \"inner quotes\"")
 (col1 = "unquoted field", col2 = "unquoted field with \"inner quotes\"")

A quote inside an unquoted field (with "inner quotes") is content: the structural scan notices it and rebuilds its index under the field-start rule.

DateFormat

using CSV

# In this file, our `date` column has dates that are formatted like `yyyy/mm/dd`. We can pass just such a string to the
# `dateformat` keyword argument to tell parsing to use it when looking for date or date-time columns.
data = """
code,date
0,2019/01/01
1,2019/01/02
"""

file = CSV.File(IOBuffer(data); dateformat="yyyy/mm/dd")

2-element CSV.File:
 (code = 0, date = Dates.Date("2019-01-01"))
 (code = 1, date = Dates.Date("2019-01-02"))

# columns with different formats take a dictionary keyed by column name or index
data = """
code,date,stamp
0,2019/01/01,01-02-2019 10:30
1,2019/01/02,02-02-2019 11:45
"""

file = CSV.File(IOBuffer(data); dateformat=Dict(:date => "yyyy/mm/dd", :stamp => "dd-mm-yyyy HH:MM"))

2-element CSV.File:
 (code = 0, date = Dates.Date("2019-01-01"), stamp = Durations.Timestamp{Dates.Nanosecond}("2019-02-01T10:30:00"))
 (code = 1, date = Dates.Date("2019-01-02"), stamp = Durations.Timestamp{Dates.Nanosecond}("2019-02-02T11:45:00"))

Without a dateformat, ISO dates, ISO date-times (with T or a space), and times are detected. Date-times prefer Durations.Timestamp{Dates.Nanosecond}. Wider dates use Timestamp{Microsecond} if every value fits exactly; otherwise the column stays text. Load Dates and request types=Dict(:stamp => DateTime) when a consumer needs DateTime. Fractions finer than whole milliseconds then produce parse problems instead of being rounded.

Custom decimal separator

using CSV

# In many places in the world, floating point number decimals are separated with a comma instead of a period (`3,14` vs. `3.14`).
# We can correctly parse these numbers by passing in the `decimal=','` keyword argument. If the file has no header or
# delimiter detection is otherwise ambiguous, pass `delim=';'` explicitly so commas are treated only as decimal markers.
data = """
col1;col2;col3
1,01;2,02;3,03
4,04;5,05;6,06
"""

file = CSV.File(IOBuffer(data); delim=';', decimal=',')

2-element CSV.File:
 (col1 = 1.01, col2 = 2.02, col3 = 3.03)
 (col1 = 4.04, col2 = 5.05, col3 = 6.06)

Thousands separator

using CSV

# In many places in the world, digits to the left of the decimal place are broken into
# groups by a thousands separator. We can ignore those separators by passing the `groupmark`
# keyword argument.
data = """
x y
1 2
2 1,729
3 87,539,319
"""

file = CSV.File(IOBuffer(data); groupmark=',')

3-element CSV.File:
 (x = 1, y = 2)
 (x = 2, y = 1729)
 (x = 3, y = 87539319)

Custom groupmarks

using CSV

# In some contexts, separators other than thousands separators group digits in a number.
# `groupmark` supports ignoring them as long as the separator character is ASCII and is not
# itself numeric syntax (a digit, sign, decimal point, or exponent letter).
# It must appear between digits in the integer part of the number.
data = """
name;part number
Ayodele Beren;5538_6111_0574
Trinidad Shiori;3017_9300_0776
Ori Cherokee;4682_5416_0636
"""

# Both spaces and semicolons divide each row consistently. Set the delimiter.
file = CSV.File(IOBuffer(data); delim=';', groupmark='_')
file

3-element CSV.File:
 (name = "Ayodele Beren", var"part number" = 553861110574)
 (name = "Trinidad Shiori", var"part number" = 301793000776)
 (name = "Ori Cherokee", var"part number" = 468254160636)

Custom bool strings

using CSV

# By default, parsing considers `true`, `True`, `TRUE`, `false`, `False`, and `FALSE` as valid `Bool` values.
# To consider alternative values, we can pass a `Vector{String}` to the `truestrings` and `falsestrings` keyword arguments;
# a user list replaces the defaults.
data = """
id,paid,attended
0,T,TRUE
1,F,TRUE
2,T,FALSE
3,F,FALSE
"""

file = CSV.File(IOBuffer(data); truestrings=["T", "TRUE"], falsestrings=["F", "FALSE"])

4-element CSV.File:
 (id = 0, paid = true, attended = true)
 (id = 1, paid = false, attended = true)
 (id = 2, paid = true, attended = false)
 (id = 3, paid = false, attended = false)

Matrix-like Data

using CSV, Tables

# This file contains a 3x3 identity matrix of `Float64`. By default, parsing will detect the delimiter and type, but we can
# also explicitly pass `delim=' '` and `types=Float64`, which tells parsing to explicitly treat each column as `Float64`,
# without having to guess the type on its own.
data = """
1.0 0.0 0.0
0.0 1.0 0.0
0.0 0.0 1.0
"""

file = CSV.File(IOBuffer(data); header=false)
file = CSV.File(IOBuffer(data); header=false, delim=' ', types=Float64)

# to convert the table to a `Matrix`, use `Tables.matrix`
B = Tables.matrix(file)

3×3 Matrix{Float64}:
 1.0  0.0  0.0
 0.0  1.0  0.0
 0.0  0.0  1.0

Providing types

using CSV

# In this file, our 3rd column has an invalid value on the 2nd row `invalid`. Let's imagine we'd still like to treat it as an
# `Int` column, and ignore the `invalid` value. The syntax examples provide several ways we can tell parsing to treat the 3rd
# column as `Int`, by referring to column index `3`, or column name with `Symbol` or `String`. We can also provide an entire
# `Vector` of types for each column (and which needs to match the length of columns in the file). An invalid value becomes
# `missing` and is recorded as a problem; by default one summary warning is printed per read, `on_error=:collect` keeps the
# problems silently for `CSV.problems(file)`, and `on_error=:error` throws a `CSV.ParseError` at the first problem.
data = """
col1,col2,col3
1,2,3
4,5,invalid
6,7,8
"""

file = CSV.File(IOBuffer(data); types=Dict(3 => Int), on_error=:collect)
file = CSV.File(IOBuffer(data); types=Dict(:col3 => Int), on_error=:collect)
file = CSV.File(IOBuffer(data); types=Dict("col3" => Int), on_error=:collect)
file = CSV.File(IOBuffer(data); types=[Int, Int, Int], on_error=:collect)
CSV.problems(file)

1-element Vector{CSV.Problem}:
 CSV.Problem(invalid_value at data row 2, column 3, byte 26: "cannot parse Int64 from \"invalid\"")

# stop at the first invalid value instead
file = CSV.File(IOBuffer(data); types=[Int, Int, Int], on_error=:error)

# In this file we have lots of columns, and would like to specify the same type for all
# columns except one which should have a different type. We can do this by providing the
# exact name for the first column and a Regex to match the rest. Note that an exact
# column name always takes precedence over a regular expression.
data = """
col1,col2,col3,col4,col5,col6,col7
1,2,3,4,5,6,7
0,2,3,4,5,6,7
1,2,3,4,5,6,7
"""
# Numeric Boolean spellings need explicit lists, even with types=Bool.
file = CSV.File(IOBuffer(data); types=Dict(:col1 => Bool, r"^col\d" => Int8),
                truestrings=["1"], falsestrings=["0"])
file

3-element CSV.File:
 (col1 = true, col2 = 2, col3 = 3, col4 = 4, col5 = 5, col6 = 6, col7 = 7)
 (col1 = false, col2 = 2, col3 = 3, col4 = 4, col5 = 5, col6 = 6, col7 = 7)
 (col1 = true, col2 = 2, col3 = 3, col4 = 4, col5 = 5, col6 = 6, col7 = 7)

Typemap

using CSV

# In this file, we have U.S. zipcodes in the first column that we'd rather not treat as `Int`, but parsing will detect it as
# such. In the first syntax example, we pass `typemap=Dict(Int => String)`, which tells parsing to treat any detected `Int`
# columns as text instead, using `stringtype` (DataString by default).
# In the second example, an explicit String type also requests ordinary Julia strings.
data = """
zipcode,score
03494,9.9
12345,6.7
84044,3.4
"""

file = CSV.File(IOBuffer(data); typemap=Dict(Int => String))
file = CSV.File(IOBuffer(data); types=Dict(:zipcode => String))

3-element CSV.File:
 (zipcode = "03494", score = 9.9)
 (zipcode = "12345", score = 6.7)
 (zipcode = "84044", score = 3.4)

Pooled values

using CSV

# In this file, we have an `id` column and a `code` column. There can be advantages with various DataFrame/table operations
# like joining and grouping when `String` values are "pooled", meaning each unique value is mapped to a `UInt32`. Pooling is
# off by default. Via the `pool` keyword argument, we can turn it on: `pool=true` pools every string column,
# `pool=0.4` means that if 40% or less of a column's values are unique, then it will be pooled, and `pool=(0.2, 500)`
# is the ratio-and-cap policy.
data = """
id,code
A18E9,AT
BF392,GC
93EBC,AT
54EE1,AT
8CD2E,GC
"""

file = CSV.File(IOBuffer(data); pool=true)
file = CSV.File(IOBuffer(data); pool=0.4)
file = CSV.File(IOBuffer(data); pool=0.6)

5-element CSV.File:
 (id = "A18E9", code = "AT")
 (id = "BF392", code = "GC")
 (id = "93EBC", code = "AT")
 (id = "54EE1", code = "AT")
 (id = "8CD2E", code = "GC")

Non-string pooled values

using CSV, PooledArrays

# CSV pools text columns only. To pool a numeric category, convert that column
# after reading with PooledArrays.
data = """
category,amount
1,100.01
1,101.10
2,201.10
2,202.40
"""

file = CSV.File(IOBuffer(data))
category = PooledArray(file.category)
category

4-element PooledArrays.PooledVector{Int64, UInt32, Vector{UInt32}}:
 1
 1
 2
 2

Pool with absolute threshold

using CSV

# In this file, we have an `id` column and a `code` column. Via the `pool` keyword argument, we can provide
# greater control: `pool=(0.5, 2)` means that if a column has 2 or fewer unique values _and_ the total number of unique
# values is at most 50% of all values, then it will be pooled.
data = """
id,code
A18E9,AT
BF392,GC
93EBC,AT
54EE1,AT
8CD2E,GC
"""

file = CSV.File(IOBuffer(data); pool=(0.5, 2))

5-element CSV.File:
 (id = "A18E9", code = "AT")
 (id = "BF392", code = "GC")
 (id = "93EBC", code = "AT")
 (id = "54EE1", code = "AT")
 (id = "8CD2E", code = "GC")

Exact decimal columns

using CSV, DataDecimals

# Loading DataDecimals lets an explicitly requested decimal type parse exactly from the field bytes.
# CSV does not infer decimal types; fractional numbers infer as `Float64`.
data = """
item,amount
coffee,3.50
bagel,2.25
"""

file = CSV.File(IOBuffer(data); types=Dict(:amount => Decimal64{2}))
collect(file.amount)

2-element Vector{DataDecimals.Decimal64{2}}:
 3.50
 2.25

With Decimal64{2}, a value such as 1.235 needs rounding and becomes a parse problem. Extra trailing zeros, such as 1.2300, are exact and accepted. Use on_error=:collect to inspect problems or on_error=:error to throw.

Inspect bad values

using CSV

text = "id,amount\n1,10\n2,not-a-number\n"
file = CSV.File(IOBuffer(text); types=Dict(:amount => Int), on_error=:collect)

[(problem.row, problem.col, problem.kind, problem.message)
 for problem in CSV.problems(file)]

1-element Vector{Tuple{Int64, Int64, Symbol, String}}:
 (2, 2, :invalid_value, "cannot parse Int64 from \"not-a-number\"")

Rows that do not match the header

A row with extra fields keeps the header schema: the extra fields are reported, not added as new columns. A row with fewer fields is padded with missing. An unclosed quote is reported and the affected text is kept.

using CSV

text = "id,name\n1,Ada\n2,Grace,extra\n3\n4,\"unclosed\n"
file = CSV.File(IOBuffer(text); on_error=:collect)

(names(file), length(file), [(p.row, p.kind) for p in CSV.problems(file)])

([:id, :name], 4, [(2, :long_row), (3, :short_row), (4, :invalid_quoted_field), (0, :unclosed_quote)])

Keep empty text distinct from missing

using CSV

table = (value=Union{Missing, String}[missing, "", "text"],)
output = IOBuffer()
CSV.write(output, table)
bytes = String(take!(output))
# A missing value in a one-column file occupies an empty row. Keep that row.
roundtrip = CSV.File(IOBuffer(bytes); stringtype=String, ignoreemptyrows=false)

(bytes, collect(roundtrip.value))

("value\n\n\"\"\ntext\n", Union{Missing, String}[missing, "", "text"])

Write to an in-memory buffer

using CSV

table = (id=[1, 2], note=["plain", "comma, inside"])
output = IOBuffer()
CSV.write(output, table; newline="\r\n")
String(take!(output))

"id,note\r\n1,plain\r\n2,\"comma, inside\"\r\n"

An IO sink is written at its current position, like Base.write; a path sink ending in .gz is compressed automatically.

Read into another table package

CSV.read calls any Tables.jl sink. For example, with DataFrames.jl installed:

using CSV, DataFrames

df = CSV.read("input.csv", DataFrame)
CSV.write("output.csv", df)

Process rows or batches

using CSV

rows = CSV.Rows(IOBuffer("id,value\n1,10\n2,20\n"); types=[Int, Int])
total = sum(row[:value] for row in rows)

Use CSV.Chunks when a downstream operation accepts table partitions; every batch is a CSV.File with the same column types:

using CSV

chunks = CSV.Chunks(IOBuffer("id,value\n1,10\n2,20\n3,30\n"); chunkbytes=8)
length(chunks)

CSV.Rows defaults to text unless you supply types. Both readers retain the source bytes; CSV.Rows also retains the whole structural index, while CSV.Chunks indexes one batch at a time. CSV.Chunks checks values across the input to choose one schema before iteration, so its constructor reads every row it will yield — limit excludes the rest from that check — and knows the batch count.

Index first and parse later

using CSV

lazyfile = CSV.lazy(IOBuffer("id,value\n1,10.5\n2,20.0\n"))
first_id = String(lazyfile.id[1])
eager = CSV.File(lazyfile; types=Dict(:value => Float64))

(first_id, collect(eager.value))

("1", [10.5, 20.0])